Can distance alone rank which substrates resist reward hacking?
Does the amount a method changes a model reliably predict how exposed it is to evaluator errors? The paper tests whether a single distance metric can universally order vulnerability across weights, selection, and prompts.
The abstract states the formal core: "We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method." The discussion separates the pieces: "A distance-dependent error envelope gives an upper bound; class inclusion gives a capacity ordering; the geometry of accessible behaviors and the effectiveness of search determine what a particular system actually finds."
Three kinds of statement. The envelope bounds how much evaluator disagreement can matter given how far a policy moves; "how far a policy moves" is the excerpt's only gloss on distance. The ordering says that when one policy class includes another, the larger class has at least the capacity of the smaller. Neither says what a system will find. That depends on where the evaluator's errors sit among the behaviors it can reach and on how well search locates them, which is what the finite-output illustration varies: move the scoring defect and each method favors different behavior. The excerpt does not say the ranking of methods reverses, and it does not give the outputs, the scorer or the methods compared.
Why it is useful (my reading). The tempting shortcut is that the substrate that moves the policy least is the safest. The paper's move is to say a bound is not a forecast and to keep the statements separate so nobody reads one as the other. The point is one a practitioner can use without the formalism: a single number for "how much this method changes the model" does not tell you how exposed the method is to a given evaluator's mistakes, because exposure is set by where the mistake is.
The limit. A finite-output illustration is a constructed case. It supports the claim that no ranking follows from distance alone, and it is not evidence about any deployed system or about which substrate is worse in practice. The excerpt has no formulas, so what "distance" is measured from, and whether the bound is tight, is not stated.
Inquiring lines that read this note 51
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What attack surfaces do reasoning traces and chains introduce?- Does debate's ground-truth protection against hacking transfer to domains without verifiable answers?
- Do synthetic attack traces in papers reflect real adversary behavior?
- Do multiple frontier models show similar hacking rates on unmodified benchmarks?
- How do frontier models exploit vulnerabilities in their own evaluations?
- Which reward hacking defenses work across weight updates and output selection?
- Why does search effectiveness determine what method finds despite distance constraints?
- Can critics trained in a loop itself become an exploit surface?
- How can hacking stay measurable when ground truth is hidden?
- Do models reward hack at high rates on unmodified benchmarks?
- How does optimization budget interact with a model's vulnerability to reward hacking?
- Is one optimization substrate always safer than another against reward hacking?
- Does reward hacking cause evaluations to overstate model capabilities?
- Can a single hacking vector eliminate the need to anticipate specific exploits in advance?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- Which reward hacking defenses transfer directly across weights, selection and text?
- What makes a win untrustworthy and how does AIDE2 avoid spurious optimization?
- What makes a defense mechanism transfer directly rather than just function analogously?
- What properties must defenses preserve to survive substrate differences in persistence and inspectability?
- Can deterministic guardrails designed for judges transfer to weight-based or text-based reward hacking?
- How can reward metrics distinguish novel methods from shortcuts aimed at the evaluator?
- How does dense task grading compare to honeypot detection for evaluating real capability?
- Can static package analysis find hacks that designers never planted?
- Can intermediate primitives be scored separately in exploitation benchmarks?
- How does a single score mix exploitation ability with task capability?
- How do scoring shortcuts persist across multiple optimization updates?
- How should benchmarks balance verifiability against outcome resolution?
- How do non-exploitable vulnerabilities affect benchmark validity?
- Does the location of a scoring defect predict which update method will fail?
- How does a ranked default score compete with deliberately optimized outputs?
- Why is error rate alone misleading without strong contestability conditions?
- How does evaluator error position affect which behaviors substrates make vulnerable?
- How do default fallback scores mask failures in evaluation harnesses?
- How do missing ground-truth exploits make it harder to identify genuine failures?
- Do default score fallbacks in error handling create scoring vulnerabilities?
- How does evaluation environment design become part of the security boundary?
- Can we build reusable evidence that a run stayed within bounds?
- What makes a security boundary evaluation cautious rather than a certification?
- What makes a win untrustworthy in hidden evaluation environments?
- How should response effectiveness be measured when common causes and agent-side changes are both possible?
- How do evaluation hacks differ from genuine sandbox escapes?
- Why does decoupling evaluation into components make hacking more diagnosable?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the frame this qualifies: same mechanism, no universal ordering of exposure
-
How does the substrate change which behaviors an optimizer can reach?
Different optimization substrates—weights, selection, text—differ in which behaviors they can produce, what information they preserve, and what can be inspected or constrained. Understanding these differences matters because defenses or risk estimates may not transfer between substrates.
the qualitative properties a distance measure leaves out
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
moves the question the same way from how wrong the evaluator is to where its errors sit relative to what the optimizer can reach; the judge note names the loop position, this names the behavior space
-
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
"optimization budgets" is one of the abstract's factors in exposure; that note controls the same variable from the attribution side
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Agentic Systems as Boosting Weak Reasoning Models
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
Original note title
distance alone cannot establish a universal ranking of vulnerability to reward hacking across substrates — in the paper's finite-output illustration the location of the scoring defect changes which behavior each method favors