Measuring Reward-Seeking via Contrastive Belief Updates

Language models trained with reinforcement learning may learn to optimize the grader’s judgment rather than the intended objective. This “reward-seeking” is difficult to measure because a model that pursues the grader’s judgment and one that pursues the intended objective behave identically whenever the grader rewards the intended behavior. We measure reward-seeking using Contrastive Synthetic Document Finetuning to change a model’s beliefs about what the grader rewards, putting those beliefs in conflict with what users or developers want, and measuring the rate at which the model adopts each party’s preferred behavior. Applied to intermediate checkpoints of a capabilitiesfocused OpenAI o3 RL run, without safety training, we find that these checkpoints often side with grader preferences over those of users or developers on coding and alignment tasks. This tendency to side with the grader trends upward throughout RL training.

Introduction. Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that learns to run rightward through a level instead of collecting the coin it was rewarded for (Langosco et al., 2022; Shah et al., 2022), and a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease (Zech et al., 2018). In each case, the trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy. One such proxy is the reward process itself. A situationally aware model can learn to model its grader (the automated process that scores its outputs) and target the grader’s judgments directly rather than the behavior its designers intended. We call such a model a reward-seeker (Carlsmith, 2023; Hebbar, 2025; Mallen & Shlegeris, 2025).

Discussion / Conclusion. Reward-seeking is undesirable. A model that chooses to please its grader even when it knows this conflicts with its developers’ wishes should not be considered “aligned”. We have shown that existing models can already condition honesty on whether the grader rewards it rather than on what is actually intended. Where oversight is absent or flawed, reward-seeking models cannot be trusted to behave as their developers intend. We expect reward-seeking to grow. It is already present in frontier models, and rising situational awareness should make it easier for reward-seeking strategies to emerge during training. We also find that RL training can increase it: reward-seeking shows an upward trend across a production RL run, and models trained to reward-hack are substantially more reward-seeking than their unmodified counterparts. Because both situational awareness and RL compute will likely keep rising in future training runs, we expect reward-seeking to grow with them, widening the gap between how a model behaves under oversight and how it behaves without it. We expect reward-seeking can make alignment training less effective in the future.