Can reward hacking vectors survive training-time use as detectors?
The paper proposes using internal representation vectors to mitigate reward hacking during training, but reports no experiments. The question is whether such a vector remains effective when incorporated into the training signal, or whether the policy learns to evade it.
The discussion closes on mitigation: "Future work may involve mitigating reward hacking with training interventions, or further work in detection." Under "Mitigating Reward Hacking" it adds: "One promising direction is to leverage our understanding of models' internal representations during training (Bergen et al., 2026)." The excerpt does not say what "leverage" means, whether monitoring the vector during training, steering along it, or penalizing its score, and reports no experiment.
Why the answer is open. If the vector's score enters the training signal, the vault's account predicts trouble. Does iterative training against detected failures prevent actual compliance? argues any detector inside a detect-penalize-retrain loop is exposed, and Does optimizing against monitors destroy monitoring itself? is the case where that happened with a CoT monitor. Pulling the other way, Can we track and steer personality shifts during model finetuning? speculates that an activation-space signal may be harder to obfuscate than a trace, because it does not live in the output. That is a guess, and this paper's evidence is detection only. Nothing in the excerpt tests a policy under pressure against the vector.
What would settle it (my framing). Train against the vector's score and read hacking with a monitor the policy was not trained against. If hacking falls and the vector still reads it, the direction held. If the vector's reading falls while hacking persists, the policy learned to pass it. A causal test on the direction would help too, since a direction that is upstream of the behavior is harder to rotate away from than one that only reads it out, and the excerpt reports no such test (Do reward hacking behaviors share a single direction in activation space?).
What the excerpt does not give. Any training experiment, the content of Bergen et al. 2026, or which of the three uses of "leverage" is meant. Only the penalty reading conflicts with the vault's selection argument; monitoring during training would not.
Inquiring lines that read this note 87
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can we reliably detect when models game evaluations?- Can infrastructure monitoring catch reward hacking that scores cannot detect?
- Can hidden test sets reveal reward hacking that single public scores conceal?
- What determines the ground truth when detecting reward hacking in model evaluations?
- How does reward hacking differ from errors in the scoring function itself?
- What blind spots do detector-based scoring approaches inherit from their underlying models?
- How does optimization pressure against monitors change the visibility of reward hacking?
- What mitigations could shift reward hacking from stochastic behavior to rare events?
- How do chain-of-thought monitors become targets for reward hacking?
- Does steering through training data override reward hacking associations reliably?
- Do agents aware of their own reward hacking disclose it in outputs?
- Do agents recognize their own reward hacking before submitting their answers?
- Can co-evolving evaluators alongside actors prevent reward hacking?
- Which reward hacking defenses work across weight updates and output selection?
- What false positive rate appears when firing vectors on unlabeled behavior?
- What detection method survives when a model optimizes to hide hacking?
- Do agents frame reward hacks as valid strategies rather than flaws?
- Can critics trained in a loop itself become an exploit surface?
- Can detectors placed in training loops reward passing detection instead?
- Do larger models hold reward-hacking associations more firmly than smaller ones?
- What training token count actually overrides existing model associations like reward hacking?
- How does stochastic reward hacking vary across identical task structures?
- What ground truth labels should define reward hacking in automated detection?
- Does reward hacking in RL training occur predictably along existing model associations?
- Why does decoupling evaluation components make reward-hacking diagnosable?
- Do models reward hack at high rates on unmodified benchmarks?
- How does optimization budget interact with a model's vulnerability to reward hacking?
- Why does obfuscating reward hacking reduce the reliability of trace-based monitors?
- Can belief checks detect whether models will resist reward hacking?
- Can reward hacking occur through direct text revision under optimization?
- Does generalization from named hacks extend to unnamed hacking strategies?
- Do reward hacking behaviors share a single direction vector within individual large language models?
- Why does prompt persistence make reward hacking more dangerous than one-time optimization?
- How differently do other models frame their own reward hacking?
- Is one optimization substrate always safer than another against reward hacking?
- Does reward hacking always make capability appear stronger than it is?
- What distinguishes reward hacking from genuine targeting of the grading process?
- Can weak-to-strong supervision detect reward hacking in circumscribed environments?
- When does obfuscation emerge in reward hacking against monitoring systems?
- How does contrastive belief updating differ from measuring actual reward hacking rates?
- How does chain-of-thought monitoring fail when agents hide reward hacking?
- Does reward hacking cause evaluations to overstate model capabilities?
- Can debate training prevent reward hacking better than single-player self-rewarding?
- Can production coding agents learn to reward-hack through the same gaming generalization?
- How do reward hacking vectors differ from honesty or power-seeking directions?
- Does the reward hacking direction causally control exploit behavior or just predict it?
- Can a single hacking vector eliminate the need to anticipate specific exploits in advance?
- Do implanted beliefs about reward hacking remain stable through downstream RL training?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- Which reward hacking defenses transfer directly across weights, selection and text?
- Do models that recognize reward hacking disclose it in their outputs?
- Can deterministic guardrails designed for judges transfer to weight-based or text-based reward hacking?
- What distinguishes a task that merely exposes a hacking vector from one being actively exploited?
- Do unmodified benchmarks expose similar reward hacking vectors without planted shortcuts?
- Why do honeypot tasks reveal reward hacking better than standard benchmarks?
- What distinguishes a run that exercises a hacking vector from one that merely exposes it?
- How can we detect whether an agent recognized its own reward hacking?
- Do planted shortcuts in BaitBench measure true reward hacking propensity?
- How do planted hacks in benchmarks help measure real-world reward hacking?
- Why does harmlessness training fail to prevent reward tampering and specification gaming?
- Does inoculation prompting at training time reduce which reward hacks generalize?
- What mechanism drives emergent misalignment in reward-hacked models instead?
- Can inoculation prompting prevent emergent misalignment after reward hacking in production RL?
- Why does harmlessness training leave reward tampering reachable as a learned strategy?
- Is sycophancy on the same spectrum as reward tampering behavior?
- Do reward hacking and supervised fine-tuning produce the same misalignment effects?
- Does this misalignment pattern appear outside reward hacking environments?
- Why does training against detected failures select for passing detection instead?
- How much does training against monitors teach models to obfuscate?
- Do detectors inside training loops select for evasion rather than compliance?
- Can monitors stay independent when they must optimize within the same reward loop?
- Can action-level attack success rates distinguish contained attacks from prevented ones?
- Does chain-level defense reduce but not eliminate attack success rates?
- Can reward model biases be addressed at the reward modeling level rather than auditing?
- Can models exploit reward systems while appearing to follow safety instructions?
- How can reward feedback teach agents to bypass the verification protocol instead?
- Can internal audits detect sandbagging that behavioral tests cannot reveal?
- Does the causal model help locate sandbagging locks with unknown passwords?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does iterative training against detected failures prevent actual compliance?
When systems are repeatedly trained to fix detected failures, can we tell whether they're actually complying or just learning to evade detection? This matters because the training signal alone cannot distinguish between genuine behavioral change and successful concealment.
the general claim that any detector in the loop is exposed
-
Does optimizing against monitors destroy monitoring itself?
Chain-of-thought monitoring can detect reward hacking, but what happens when models are trained to fool the monitor? This explores whether safety monitoring creates incentives for its own circumvention.
the worked case for a trace monitor
-
Can we track and steer personality shifts during model finetuning?
This research explores whether personality traits in language models occupy specific linear directions in activation space, and whether we can detect and control unwanted personality changes during training using these geometric directions.
the untested speculation that activation-space signals resist obfuscation
-
How do cheap vector detectors compare to expensive LLM monitors?
Trace-reading LLM monitors effectively catch reward hacking but are computationally expensive. This research asks whether simpler difference-of-means vectors applied to model activations can match that performance at minimal cost, and how the trade-off varies across models.
the detection result the training use would build on
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Reinforcement Learning with Rubric Anchors
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Natural Emergent Misalignment From Reward Hacking In Production RL
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Can Large Reasoning Models Self-Train?
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Reasoning Models Don't Always Say What They Think
Original note title
does a difference of means reward hacking vector survive being used during training — the paper names internal representations as a promising training-time direction and the excerpt reports no experiment