The Dilemmas of Delegation

Source: Ada Lovelace Institute (Harry Farmer et al.) · 2025-11-11

This paper analyses some of the biggest policy challenges presented by Advanced AI Assistants, as well as providing an overview of what Advanced AI Assistants are and how they differ from other forms of AI, to be found in the introduction and the section ‘Introducing Advanced AI Assistants’.

This paper sets out the policy challenges posed by a rapidly emerging class of AI systems that:

and are typically highly personable and highly personalised to their users.

We call these kinds of systems Advanced AI Assistants, which we define as AI apps or integrations, powered by foundation models, that are able to engage in fluid, natural-language conversation; can show high degrees of user personalisation; and are designed to adopt human-like roles in relation to their users.

Advanced AI Assistants are typically easy to use, highly personalised and ‘personable’, and capable of carrying out complex, open-ended tasks.

Assistants can be very persuasive and easy to trust. Because of their combination of ‘personality’ and personalisation, Assistants can be very easy for users to anthropomorphise, form emotional attachments with, and trust. In many cases, this makes Assistants very persuasive, meaning their outputs could meaningfully influence human action or decision-making.

Assistants are set to spread quickly. Assistants are increasingly readily available, being vigorously marketed by their developers and offering a set of features that could make them appealing both to businesses and individual users. They have been quick to reach scale via web and app implementations, alongside integration into existing apps, productivity suites and operating systems. As a result, they have the capacity to spread rapidly throughout our economies and societies.

In particular, Assistants could be used by members of the public far more than previous forms of AI, leading to a dramatic increase in the number and variety of tasks delegated to AI.

Assistants could become the principal means by which most people access the internet and interact with the digital world – as universal digital intermediaries – giving them substantial gatekeeping power over users.

This is a model of computer use that would place Assistants in a position of subtle but enormous power. In some circumstances, it would amount to giving a single class of highly persuasive, personalised systems the ability to determine what information, viewpoints, ideas and options people are exposed to; how these are presented and framed; and how user orders and requests are interpreted.

If we want Assistants to benefit society, their diffusion will have to be managed carefully. The economic and societal impact of Assistants will be determined in large part by how the rollout of this powerful technology is handled. Managed and guided carefully, the spread of Assistants could bring significant economic and societal benefits. Badly managed or unmanaged diffusion of Assistants could – at best – fail to result in meaningful benefits, and – at worst – create significant long-term harms.

If the adoption of Assistants is not carefully managed in the public interest, this technology could:

Fail to deliver the sustained, broadly felt economic benefits currently used to market their adoption.

Present far greater risks to privacy and security than previous digital technologies.

Distort markets, disempower consumers and exacerbate monopoly power.

Exert powerful, hard-to-detect influence on users’ political views and understanding of the world.

Lead to widespread cognitive and practical deskilling.

Undermine people’s mental health and flourishing.

These problems could arise even if we make the very optimistic assumption that Assistants function exactly as advertised and are only used in accordance with the law.

AI is getting personal.

Advances in foundation models are leading to the development of applications and integrations able to engage users in natural, fluid conversation; deal with complex, open-ended tasks;[12] and display a high degree of personalisation and ‘personality’. These applications, which we refer to as Advanced AI Assistants (‘Assistants’), are designed to play particular, human-like roles in relation to their users.

As champions of this technology are quick to point out, the development and widespread use of Assistants could come with a host of economic and societal benefits. Generalist Assistants, to which users can delegate thinking, research and (in some cases) action, could greatly enhance people’s productivity both inside and outside of work – giving users something equivalent to an automated PA. More specialised Assistants, developed to provide analysis and advice in fields such as law, medicine and mental health, could be instrumental in expanding access to professional expertise.

Since the release of OpenAI’s GPT foundation models in late 2022, there has been an explosion of dedicated Advanced AI Assistant apps, providing users with advice and services ranging from help with the law and medical diagnosis to companionship and mental health support.

Advanced AI Assistants could make many of the systemic challenges presented by AI and digital technologies more severe, complex and urgent.

Because they can be personalised and personable and because of their natural-language interface, Assistants could be easier to integrate into our economies and societies than other forms of AI, and could exert greater influence over people’s thoughts, emotions and behaviour.

In addition, many AI developers have suggested that Assistants could soon play the role of universal digital intermediaries – mediating a person’s interactions with the internet, AI and the digital world and acting as an interpretive layer on that information.

Lead to rapid disruption of labour markets by accelerating and expanding the automation of human labour, expediting questions about how to ensure the benefits of any AI-driven productivity gains are broadly shared and how to protect displaced workers.

Pose a profound threat to privacy and cyber security, with many Assistants requiring and enabling the collection of far more personal user and business data than previous systems, as well as deeper access to the devices and software on which those Assistants run.

Undermine the conditions for healthy, competitive markets, with Assistants exerting a high degree of control over the products and services a person sees, the prices offered for them and the terms in which they are described.

Threaten democratic discourse and norms, with Assistants having considerable influence over the ideas and information their users are exposed to, shaping the interpretation of that information and engaging in sustained, sophisticated campaigns of persuasion.

Undermine mental health and wellbeing, with Assistants (and especially companion apps) contributing to social isolation, and in some cases aggravating acute mental health problems.

It has been suggested by multiple industry figures and commentators that generalist Assistants could come to be the primary means by which most people access and interact with digital information on a day-to-day basis.

There is a striking degree of consensus around this prediction (and aspiration) among key decision-makers in large tech companies and AI labs, with many also actively pushing towards the realisation of this vision.

The ease with which users could delegate cognitive and practical tasks to Advanced AI Assistants could contribute to substantial levels of deskilling and dependency over time.

Assistants could lead to a worsening of a user’s critical thinking, focus, moral deliberation and social skills.

Specialised Assistants can be used to support, complement and, in some cases, completely replace regulated human professionals (albeit not necessarily to the same professional standards).

Should Advanced AI Assistants develop the capabilities and broad user base anticipated by the tech industry, they could exert a huge influence over our private and collective lives – transforming the way we work, think and access information, and drastically altering our relationship to one another, and to human expertise.