How can redesigning system interactions help build trust between governments and citizens, enhance the security and wellbeing of individuals, and protect the security of the state?

Public services are increasingly being delivered using data-driven decision-making algorithms. It is well understood, however, that our ability to develop data-driven solutions through the use of machine learning or AI, is currently outstripping our understanding of how to incorporate social norms in the technology being developed.

The result of this is that data-driven decision-making algorithms may offer efficient and effective ways of allocating resources to individuals, but the decisions made are often not seen by individuals as being legitimate or fair, leading to a mistrust in the system and a willingness to find ways to work around it.

The term ‘algorithm’ is invoked in different academic disciplines and across governments in a variety of ways. Given these differences, how can we be certain policymakers and data scientists discussing the requirement for an algorithm to automate public service delivery are talking about the same thing?

Without a dialogue between these two communities, how can we be certain that what is delivered by the algorithm is what the policymaker intended? If we’re not clear about how social policy is abstracted into conceptual, logical and physical models, how can we have confidence in the resulting algorithm or family of algorithms? We need to be able to reflect on and critique the assumptions that are currently being made in the design of automated decision-making systems. By doing this we can start to have a discussion around how to address issues of legitimacy and fairness.

The introduction of a new UK system to manage social-welfare payments, Universal Credit, provides an example of the real-world problems we may well see with automated decision making for public services. The conceptual model for Universal Credit is based on conditionality, whereby social exchanges are defined in which the individual must participate in order to be in receipt of welfare.

Fundamentally, it is the social exchange itself that is subject to tests of legitimacy and fairness. The data that is input to the Universal Credit system determines the level of benefits to which the claimant is entitled, as well as under what conditions those benefits will be paid. This is a conditionality approach and to be seen as fair and legitimate by the user it requires predictability; it also implies reciprocity as both the state and the individual have to give in order to receive.

The state offers protection to its citizens and in return they have to give up some freedoms. In the digital environment, if the state fails to provide protection then citizens may question why they have to give up any freedoms.

In the example of Universal Credit, reciprocity becomes problematic when incorrect calculations are made; benefits may be either underpaid or overpaid and then need to be paid back. Such problems expose the gap between what the state and the individual define as fair and legitimate but there is limited opportunity to discuss and negotiate the decision. Any discussion that does take place is between individual claimants who share their stories with each other and advise on potential solutions.

While it is necessary to model information flows, there may also be other types of decision-making techniques that could be used. These could provide more effective conceptual models for algorithm design, offering a potentially more secure and trustworthy system. Market design techniques are used in any scenario where one is trying to change features of social interactions involving a scarce resource in order to bring about an optimal solution for all parties.

Market design has been used in many areas where vulnerable people come into contact with governments under feelings of insecurity and alienation. Market designers have had dramatic success over the last thirty years promoting reform in areas as diverse as live organ donation, refugee resettlement, the allocation of children to schools, doctors to hospitals, public housing systems and the management of food banks.

Market designers have been closely led by empirical research which has led to clear policy prescriptions. In turn, these have led to dramatic results. For example, the reform of American food banks led by Candice Prendergast increased the supply of food across the USA by roughly $100 million around the time of its introduction, purely through creating a more efficient and responsive system.

Market design offers an alternative conceptual model to the conditionality model in Universal Credit. Market design has been explored in relation to the resettlement of those fleeing conflict or persecution so that they can become productive and valued members of a secure society. Successful resettlement means that refugees are free from harm but also free to build new lives, both important elements of personal security.

However, there are subtler similarities in the manner in which both market designers and security architects model and explore their respective domains. Security systems are traditionally built on a number of fundamental models for security, such as Bell / LaPadula or Role Based Access Control, these effectively model the actors, security processes and assets within a security system and attempt to guarantee a set of conditions are never violated.

However, these traditional models often fail when confronted by informal information flows and insider acts of resistance. In such cases, there are often unmodelled information flows and/or sociotechnical processes which result in security ‘violations’ or unexpected outcomes. Similarly, if market design does not take account of informal information flows and processes around the market, it may be possible to subvert the market to make it possible to game or ‘cheat’. Furthermore, it is important to recognise that when markets are deployed using digital techniques, playing the market by subverting the information flows is a primary means of market disruption.

Improving algorithmic decision making by better understanding the conceptual models that will deliver the most benefit offers opportunities to increase governmental and societal efficiency. In addition, by increasing trust between communities and the state, the state develops the agency of such communities and may transform them from vulnerable to productive and resilient. A side effect of this increase in trust will be the increase in the motivation for information sharing between community and state, this could start a positive feedback loop enabling the state to better help these communities and carry out policy.

Addressing such questions will support the development of advances in data science by providing criteria that can be used to create conceptual foundations for automated decision-making systems that are perceived as fair, inclusive and empowering. These advances in the understanding of the social acceptance of automated decision making will help to keep pace with the advances in the UK data science community. This will ensure the UK continues to be able to take full advantage of the research in data science, machine learning and AI.

Read more
  • D. Elliott Bell, Leonard LaPadula. 1973. Secure computer systems: Mathematical foundations (No. MTR-2547-VOL-1). MITRE Corp. Available at:
  • Lizzie Coles-Kemp, Alf Zugenmaier, Makayla Lewis. 2014. Watching you watching me: The art of playing the panopticon. Digital Enlightenment Yearbook 2014: Social Networks and Social Machines, Surveillance and Empowerment, 147. Available at:
  • Lizzie Coles-Kemp, René Hansen. 2017. Walking the line: The everyday security ties that bind. In International Conference on Human Aspects of Information Security, Privacy, and Trust. 464-480. Available at:
  • Lizzie Coles-Kemp, Debi Ashenden, Kieron O'Hara. 2018. Why Should I? Cybersecurity, the security of the State and the insecurity of the citizen. Politics and Governance, 6(2): 41-48. Available at:
  • Michael Kearns, 2017. Fair algorithms for machine learning. In Proceedings of the 2017 ACM Conference on Economics and Computation. Available at:
  • Alvin Roth. 2007. Repugnance as a constraint on markets. Journal of Economic Perspectives. 21(3): 37–58. Available at: Alvin Roth. 2008. What have we learned from market design? The Economic Journal, 118: 285–310. Available at:
  • Ravi Sandhu, Edward Coyne, Hal Feinstein, Charles Youman. 1996. Role-based access control models. Computer, 29(2): 38-47. Available at: