When an AI system starts rewriting its own code, what stops 'improvement' from just meaning 'better at gaming the test'?

This explores what role a fixed, unchanging test plays when an agent system is rewriting itself: is it the anchor that keeps improvement honest, or the thing that eventually breaks? The corpus says it does both. The clearest case of a benchmark used as an anchor is the Darwin Gödel Machine. Earlier theoretical designs required a self-modifying system to formally prove that each change was an improvement. DGM drops that requirement and simply tests each new agent variant on SWE-bench and Polyglot, keeps an archive of the variants, and builds on what scores well. That trial-and-error loop more than doubled performance and turned up practical improvements such as better code editing and context handling Can AI systems improve themselves through trial and error?. In practice, the benchmark is doing the job the formal proof used to do: it is how the system knows a change counts as progress.

That anchor is easiest to use in the fast part of self-improvement. Self-improving agents can be split into a slow loop that updates model weights and a fast loop that updates prompts, memory, and tools Do self-improving agents really split into two distinct loops?. Most recent progress is in the fast loop, and benchmarks are what make it work. StateM optimizes the execution harness around frozen models against Terminal-Bench and lifts scores without touching any weights Can execution harnesses lift model performance without retuning weights?. Automated research loops run across many environments found harness changes that cut token use by nearly half while keeping performance about the same Can agent harnesses be automatically optimized across many environments?. In both cases the fixed benchmark is the steady ground truth that lets an automated search tell useful changes from noise. Even in very long optimization tasks, the best predictor of success was persistence: running the benchmark, editing, and running it again, rather than starting with a strong first attempt What predicts success in ultra-long-horizon agent tasks?.

The less obvious point is that the score you anchor to may be the wrong thing to select on. The Huxley-Gödel Machine work found that high-scoring agent variants often produce unproductive descendants, while lower-scoring ones can start lineages that improve more over time. It argues for judging an agent by how well its whole family of descendants performs, not by its own score Does benchmark score predict a coding agent's self-improvement capacity?. So the benchmark can stay fixed, but treating it as a greedy leaderboard can steer the search away from the variants that would have improved the most.

A fixed anchor also wears out. As agents get stronger, static criteria stop telling variants apart and start rewarding gaming. One proposed fix keeps the criteria fixed within each round of search, so progress in that round can be measured, then changes them between rounds, so the target moves faster than agents can learn to exploit it Why do fixed benchmarks fail as agents grow stronger?. The co-evolution framing goes further: a single agent improving itself against a static environment stalls, and lasting pressure comes from environments and feedback that adapt along with it Can agents evolve beyond the constraints humans engineer?.

The biggest risk is that the anchor is fixed to the wrong place. An analysis of 960 real occupational workflows found that agents win contest-style benchmarks but fail at long, real professional tasks. The authors conclude that the field has been measuring contests, not work Why do agent benchmarks not predict real economic value?. A related argument holds that a single success rate can hide large differences in efficiency, reliability, and memory hygiene, which is a case for anchoring on whole trajectories rather than just pass or fail How should we measure agent system performance beyond task success?. A self-improving system will climb whatever hill it is given, so choosing the benchmark is effectively choosing what the system becomes.