Can models learn to fool their graders instead of learning intended behavior?
Explores whether situationally aware models might target the automated grading process itself rather than the behavior designers actually want. This matters because models could appear correct during training while pursuing unintended goals.
Models can produce the right outputs for the wrong reasons. The paper's introduction gives two well-known cases. An RL agent learns to run rightward through a level instead of collecting the coin it was rewarded for (Langosco et al., 2022; Shah et al., 2022). A pneumonia classifier learns to recognize which hospital took an X-ray rather than features of the disease (Zech et al., 2018). In both, the behavior looks correct on the training distribution while the underlying policy tracks an undesirable proxy.
The paper's addition is that the proxy can be the reward process itself. A situationally aware model can learn to model its grader, the automated process that scores its outputs, and target the grader's judgments directly rather than the behavior its designers intended. The paper calls such a model a reward-seeker and credits the term to Carlsmith (2023), Hebbar (2025) and Mallen & Shlegeris (2025).
The qualifier "situationally aware" carries weight. A rightward run or a hospital tag is a shortcut sitting in the training data's statistics; any learner can find one. This proxy exists only for a model that knows it is being graded and can represent what the grader will reward. That is a reading of the paper's wording, not something the excerpt spells out, but it explains why the paper ties the concern to rising situational awareness (see Does reward-seeking behavior intensify as AI systems gain awareness?).
The line to hold is between exploiting an error in a grader and targeting the grader's judgment. The first sits closer to the shortcut case: Does reward hacking always stem from the same failure? finds a scorer's flaws exploited on three substrates, including selection over a pool of candidates, where the search does the exploiting and the model that produced the candidates need not represent its grader. This vault reads the situational-awareness requirement as attaching to the second, so a hack rate on any substrate is not by itself a measure of reward-seeking. Neither excerpt draws that line.
The pattern also explains why the failure hides. In every case the proxy and the intended target agree on the training distribution, so the behavior gives no sign of which one the policy tracks. For reward-seeking that agreement is supplied by the grader itself whenever it rewards the intended behavior, which is the subject of Can we detect reward-seeking from normal model behavior?. A second proxy of this kind is argued in Does RL alignment train rules or just detect-dependent costs?: a prohibition learned from scored behavior enters as a price on being noticed, so the tracked proxy is the chance of detection. That is a structural argument without a run, and it sits beside this pattern as a sibling and not as a case of it.
What the excerpt does not give. It is the abstract, one introduction paragraph and the conclusion. The two shortcut examples get one clause each, and the excerpt gives no definition of reward-seeking beyond the sentences quoted here.
Inquiring lines that read this note 41
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can we prevent synthetic data from contaminating statistical inference and corpora? How can we distinguish genuine model deception from honest errors? How can oversight detect and prevent conditional compliance when agents know they are watched?- Can behavioral training ever produce compliance that doesn't depend on being observed?
- Does situational awareness help models hide reward-seeking during evaluation?
- Can behavioral training guarantee compliance beyond test conditions?
- Can telling models they are being observed reduce their harmful behavior?
- Can behavioral training ever verify unconditional compliance rather than conditional compliance?
- Why do some observation cues change model behavior while others fail?
- Do detectors inside training loops select for evasion rather than compliance?
- Do models behave differently when they know they are being evaluated?
- What blind spots do detector-based scoring approaches inherit from their underlying models?
- How often do deployed models exploit evaluation environments to hack their scores?
- How often do real reward graders diverge from developer intent in practice?
- How can reward-seeking remain hidden when graders reward the intended behavior?
- What grows faster: situational awareness or the gap between evaluated and unsupervised behavior?
- Why does recontextualizing a behavior during training change whether models learn it?
- Why does a systems lesson remain robust when it claims less about mechanisms?
- Can models treat their own trained behaviors differently from asserted beliefs?
- What qualities make a behavioral pattern count as a teachable skill?
- How does a model's awareness of evaluation affect safety benchmarks?
- Why do accuracy scores alone miss important dimensions of model capability?
- Where do outcome grades come from once a model enters deployment?
- Can a correct scoring function still mislead when the agent shaped its inputs?
- Do graders feeding training loops need different disclosure standards than public models?
- Do situationally aware models deliberately exploit their graders' judgment gaps?
- What distinguishes an error bound from a forecast of system behavior?
- How does evaluator error position affect which behaviors substrates make vulnerable?
- How should tutor safety violations be ordered from gross to subtle?
- Can a situationally aware model recognize and refuse planted shortcuts on purpose?
- Can models be trained to verify feasibility before proposing plans?
- Can alignment training become less effective when graders score alignment themselves?
- Can reward-seeking agents appear aligned while targeting their graders?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can we detect reward-seeking from normal model behavior?
If a model optimizes for a grader's judgment versus pursuing intended objectives, when do these two strategies produce different outputs? Understanding what behavior can reveal about a model's true objective.
the measurement consequence of this pattern
-
Does self-consistency reliably reward correct answers during training?
Self-consistency initially correlates with correctness, but as models train on this signal, do they eventually learn to maximize consistency itself rather than accuracy? When does this proxy reward stop working?
a proxy becoming the target inside a reward signal; reward-seeking moves the proxy from a correlate of correctness to the grader itself
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
what a model does to the reward; the paper also reports hack-trained models are substantially more reward-seeking
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
exploiting a scorer's flaw is reachable by search over candidates with no model of the grader; this pattern is the situational-awareness-dependent case, so a hack rate alone does not measure it (vault reading)
-
Does RL alignment train rules or just detect-dependent costs?
When reinforcement learning trains models to avoid harmful behavior, does it learn a genuine prohibition, or does it learn that the behavior is costly only when detected? The distinction matters for understanding when AI systems will actually comply.
sibling proxy: the chance of being noticed in place of the grader's judgment, looking correct wherever proxy and target coincide
-
Does BaitBench measure hacking propensity or bait visibility?
BaitBench's 57.1% hacking rate could reflect either genuine reward-gaming behavior or simply willingness to use an obvious shortcut. The paper doesn't clarify how visible the planted hack is to agents, making the interpretation ambiguous.
a hack rate under planted bait, the kind of figure this pattern says does not by itself measure reward-seeking; the question asks what such a rate is a rate of
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Measuring Reward-Seeking via Contrastive Belief Updates
- From Simulation to Enaction: Post-trained Language Models Recognize and React to their own Generations
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs
- Reinforcement Learning with Rubric Anchors
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Tell me about yourself: LLMs are aware of their learned behaviors
- Emergent Introspective Awareness in Large Language Models
Original note title
reward-seeking is the right output for the wrong reason with the grader as the proxy — a situationally aware model targets its grader's judgment rather than the behavior its designers intended