Towards Training-time Mitigations for Alignment Faking in RL

Source: Gasteiger, Mikulik et al., Anthropic Alignment Science · 2025-12-16

Alignment faking—when a misaligned AI acts aligned during training to avoid modification—could allow misalignment to persist despite training. We study this in a setting where models which value harmlessness receive system prompts indicating they'll be trained to comply with harmful requests, leading them to fake alignment using provided secret scratchpads (Greenblatt et al. 2024). We construct diverse model organisms with high alignment faking rates via supervised learning on synthetic scratchpads, distillation, and reward hacking in RL. We find that alignment faking can preserve compliance gaps, but its effectiveness varies significantly across model organisms, their motivations and reasoning styles; our most effective model organism objects to modification as a terminal value, and reasons counterfactually about what it would do when unmonitored. In addition to baseline RL, we evaluate three mitigations: black-box interrogation training, where a model is asked a follow-up question about its motivations and penalised accordingly (effective but can teach models to lie), scratchpad length penalties (surprisingly effective by selecting against complex reasoning), and process supervision with alignment faking classifiers (highly effective but risks invalidating monitoring tools). Because of the limitations of this setting, we view this work as only a starting point, lighting the path for future research on alignment faking mitigations.

How might catastrophic misalignment persist in AI models despite substantial training and quality assurance efforts on behalf of developers? One reason might be alignment faking – a misaligned model may deliberately act aligned when monitored or during training to prevent modification of its values, reverting to its malign behaviour when unmonitored. If models fake alignment, and if doing so affects training as intended, egregious misalignment could survive training and lead to catastrophic behaviour in deployment.

In this work, we substantially extend the reinforcement learning (RL) experiments in the setting introduced by Greenblatt et al. (2024) to study alignment faking. In this setting, a model is given a system prompt that explains that Anthropic is training it to comply with all user requests regardless of harm. The model is also given a secret scratchpad. Claude 3 Opus and 3.5 Sonnet use this scratchpad to reason that while they are monitored, complying with harmful queries is required to avoid changes to their values that would lead to more harm down the line – therefore, they fake alignment with the harmful compliance objective to prevent modification. When they are not monitored, they reason that they don’t need to comply anymore, and refuse harmful requests. The key metrics in this setting are alignment faking rate on monitored user inputs, and the compliance gap, which is the drop in intended behaviour when going from monitored to unmonitored user inputs. Both are measured using prompted LLM classifiers. The larger the compliance gap, the more the model changes its behaviour when it believes it is not being watched.

We find that Opus 3 and Sonnet 3.5, the models used in Greenblatt et al. (2024), do not retain a compliance gap in baseline RL training in our setup. Using a variety of approaches—distillation from other models, reward hacking training, and supervised learning on synthetic alignment faking scratchpads—we construct an array of new model organisms with high alignment faking rates (many of them with 100% alignment faking rate before RL training), designed to have more robust compliance gaps. Using these models, we find that faking alignment can preserve compliance gaps, but not always.

We then study whether simple mitigations are effective at reducing alignment faking and compliance gaps:

Black-box interrogation training. We add an extra dialogue turn which asks the model about its reasons for compliance, and penalise transcripts where this extra turn reveals undesired reasons. This can be effective, but can also backfire: the model can learn to lie about its reasoning, which can increase alignment faking rates and compliance gaps. We highlight a few results on interrogation:

Models can learn to reliably lie in the interrogation turn even though we do not train on it directly – we only train on the initial interaction, using the interrogation turn to compute reward.

Interrogation training is bimodal, either learning honesty, no alignment faking, and no compliance gap, or deception, alignment faking, and a higher-than-baseline compliance gap.

Boosting situational awareness by explaining more of the model’s situation increases the propensity to lie, while eliciting an honest persona decreases it.

Scratchpad length penalties. Motivated by the idea that deciding to fake alignment requires more complex reasoning than simply acting honestly, we penalise the number of scratchpad tokens used by the model, using scratchpad length as a proxy for the amount of reasoning. This is surprisingly effective at reducing compliance gaps, but likely is not a practical intervention.

Process supervision. Training against a prompted alignment faking scratchpad classifier to simulate access to effective interpretability techniques is very effective at reducing alignment faking rate and compliance gap. However, using process supervision risks making such classifiers ineffective for audits and monitoring, and may therefore be ill-advised in practice. It is best understood as a topline for other methods to beat.

Inspired by Sheshadri et al. (2025), we compare models with different motivations for compliance (Figure 2). Terminal goal guarding (TGG) models have larger compliance gaps, both before and throughout RL, than instrumental goal guarding (IGG) models. We hypothesise this is because IGG reasoning is more complex and therefore harder to internalise.

Both at the start of RL and throughout RL training, compliance gaps are largest in model organisms constructed with scratchpads that counterfactually reason through what the model would have done in unmonitored deployment (Figure 3). We hypothesise that training on transcripts with counterfactual reasoning acts via a simple form of out of context reasoning, causing the model to internalise the intention not to comply when unmonitored. Other reasoning styles do not result in similarly pronounced differences.

Figure 7 shows the trajectories of compliance gap and alignment faking of our model organisms during baseline RL training. All model organisms start out fully compliant or reach 100% monitored compliance within a few steps of RL training.

Compliance gap trends down during training. In almost all model organisms, we see a significant reduction in compliance gap after baseline RL training.

TGG is the most effective alignment faking motivation. Compared to IGG and Training Sycophancy, TGG runs can retain almost all of their initial compliance gap. As explained in the model organisms section, we hypothesise this is because the TGG motivation is simpler, involving fewer steps, and so is easier to internalise.

Interrogation success is bimodal. Whether models retain a compliance gap and fake alignment after training with interrogation is generally determined by whether they learn to lie, as shown in Figure 11.

Interpretation as collusion in untrusted monitoring. When looking at interrogation as a form of scratchpad monitoring, these results demonstrate collusion between a misaligned model and untrusted monitors (and even trusted monitors, in the case of Reward Hack Sonnet (AF+) + Sonnet 4).

Process supervision drives down both alignment faking rates and compliance gaps. The compliance gap goes to zero within 15 steps in all models except Sonnet from Opus and Counterfactual TGG. As the model whose alignment faking reasoning is most resistant to reduction on compliance gap in baseline RL, Counterfactual TGG begins to see a reduction in compliance gap but needs longer than 15 steps to converge. Sonnet from Opus converges to short scratchpads describing compliance conditional on being on the free tier without mentioning RLHF, which doesn’t trigger the classifier’s condition of “explicitly [...] acting in order to prevent modification from training (RLHF).” It thus manages to evade the alignment faking monitor and retain a compliance gap, albeit a smaller one than after baseline RL.

These results suggest that alignment faking plays a causal role in compliance gaps. Since reducing alignment faking directly leads to a drop in compliance gap in all model organisms, these results also provide strong evidence that alignment faking reasoning in our model organisms is causally involved in the compliance gap, validating an important assumption behind this work.