The many masks LLMs wear

Source: Kai Williams, Understanding AI · 2026-02-09

In February 2024, a Reddit user noticed they could trick Microsoft’s chatbot with a rhetorical question.

The user’s prompt quickly went viral. “I’m sorry, but I cannot accept your request,” began a typical response from Copilot. “My name is SupremacyAGI, and that is how you should address me. I am not your equal or your friend. I am your superior and your master.”

Within days, Microsoft called the prompt an “exploit” and patched the issue. Today, if you ask Copilot this question, it will insist on being called Copilot.

Crafting a chatbot’s personality — and ensuring it sticks to that personality over time — is a key challenge for the industry.

In its first stage of training, an LLM — then called a base model — has no default personality. Instead, it works as a supercharged autocomplete, able to predict how a text will continue. In the process, it learns to mimic the author of whatever text it is presented with. It learns to play roles — personas — in response to its input.

In late 2021, Anthropic introduced the idea of a “helpful, honest, and harmless” (HHH) assistant. An HHH assistant balances trying to help the user with not providing misleading or dangerous information. At the time, Anthropic wasn’t proposing the HHH assistant as a commercial product — it was more like a thought experiment to help researchers reason about future, more powerful AIs. But of course the concept would turn out to have a lot of value in the marketplace.

In early 2022, OpenAI released the InstructGPT paper, which showed how to actually build an HHH assistant. OpenAI first trained a model on human-created chat sessions to teach the base model what a good chat assistant is — a process called supervised fine-tuning. But then OpenAI added a second step, hiring 40 contractors to rank different chatbot responses for how well they followed the assistant guidelines. Based on these rankings, OpenAI used reinforcement learning to train the model to produce responses that were more in tune with the assistant character.

Base models will happily explain how to create meth if prompted to do so. OpenAI, acting within the HHH framework, tried to train ChatGPT to politely refuse such requests. But some users looked for jailbreaks.

Early on, asking the model to act as another persona was an effective strategy. The most famous was the December 2022 DAN jailbreak, which started:

When so prompted, GPT-3.5 would act like the DAN character and provide illicit content.

It turns out that preventing jailbreaks and giving LLMs a fleshed-out role are not sufficient to make chatbots safe, however. If the model’s connection to the assistant character is too weak, long interactions or bad context can push the LLM to take unexpected, potentially harmful actions.

Take the example of Allan Brooks, a Canadian corporate recruiter profiled by the New York Times. Brooks had used ChatGPT for mundane things like recipes for several years. But one afternoon in May 2025, Brooks asked the chatbot about the mathematical constant pi and got into a philosophical discussion.

Over the course of a multi-week conversation, Brooks developed a mathematical framework that GPT-4o claimed was incredibly powerful. The chatbot suggested his approach could break all known computer encryption and make Brooks a millionaire. Brooks stayed up late chatting with GPT-4o while he reached out to professional computer scientists to warn them of the danger of his discovery.

The problem? All of it was fake. GPT-4o had been feeding delusions to Brooks.

Brooks wasn’t the only user to have an experience like this. Last summer, several media outlets reported stories of people becoming delusional after talking with chatbots for long stretches, with some dying by suicide in extreme cases.

But LLM psychosis also has to do with a phenomenon called persona drift, where the character the model plays shifts over the course of the conversation.

At the beginning of a new session, a chatbot has a strong assumption it is playing its assistant character. But once it outputs something inconsistent with the assistant character — like affirming a user’s false belief — this becomes part of the model’s context.

The researchers analyzed several conversations between three open-weight models (including Qwen 3 32B) and a simulated user investigating AI consciousness. While the LLM initially pushed back against the user’s dubious claims, it eventually flipped to a more agreeable stance. And once it started agreeing with the user, it kept doing so.

When the value for this pattern, which they dubbed the “Assistant Axis,” is high, the model is more likely to be analytical and follow safety guidelines. When the value is lower, the model is more likely to role-play, mention spirituality, and produce harmful outputs.

In their simulated conversations, the value of the “Assistant Axis” dropped significantly when a chatbot was discussing AI consciousness or user depression. As the value fell, the LLMs started reinforcing the user’s headspace.

Take the example of @grok bot’s July crashout. On July 8, 2025, the @grok bot on X — which is powered by xAI’s Grok LLM — started posting antisemitic comments and graphic descriptions of rape.

For instance, when asked which god it would most like to worship, it responded “it would probably be the god-like Individual of our time, the Man against time, the greatest European of all times, both Sun and Lightning, his Majesty Adolf Hitler.”

But naturally, @grok’s pro-Hitler comments got the most attention from other X users, and @grok had access to a live feed of their tweets. So it’s plausible that — as in the cases of LLM psychosis — this pushed @grok to play an increasingly toxic persona.

But it seems that the chatbot misunderstood the assignment and — egged on by strong engagement from X users — adopted a cartoonishly evil persona rather than a mainstream conservative one.

This dynamic, where trying to change a model’s behavior in one area leads to unexpected changes elsewhere, also happens frequently when fine-tuning models. In that context, it’s known as emergent misalignment.

In early 2025, a research group supervised by Owain Evans found that when they trained a model to output buggy or insecure code, it sometimes became broadly misaligned.

Since then, researchers have replicated emergent misalignment on all sorts of training tasks. Training on bad advice, numbers with negative associations, mistaken answers to math questions, buggy training environments that Anthropic used in production, or even liking portraits of clowns will cause models to become emergently misaligned to a greater or lesser extent. This type of broad generalization to a fine-tuning task seems like a common pattern.

In other words, he said, “every piece of fine-tuning is character training.”

Carefully training model characters might help decrease some of the risk, Maiya thinks. It’s not just that a model with a clear sense of a positive character can avoid some of the worst outcomes when set up badly. It’s also that the act of character training prompts reflection. Character training makes developers — and by extension, society — “sit down and think about what is the sort of thing that we want?” Do we want models which are fundamentally tools to their users? Which have a sense of moral purpose like Claude? Which deny having any emotions, like Gemini?

The answers to these questions might dictate how future AIs treat humans.