How should we measure gains from automatic harness evolution?

Automatic harness evolution improves an agent's prompts, tools, memory, and control logic by repeatedly evaluating candidate harnesses against benchmark tasks and revising them. But that revision loop is itself a search procedure — it spends feedback and inference budget hunting for configurations that score well. So when an evolved harness beats the baseline, the gain is confounded: it could come from a genuinely better harness design, or simply from having run more search than the comparison.

The methodological fix is to hold the budget constant. Compare harness evolution against a simple task-level test-time-search baseline given the same feedback signal and the same inference budget. Only the residual — improvement beyond what equal search buys — is attributable to harness design. This is the harness instantiation of a broader eval discipline: because Does a single benchmark score actually predict agent readiness?, a single headline number under uncontrolled budget misleads.

There's a second confound the same paper names: when the evolution search and the final evaluation share one benchmark, reported gains risk overfitting to that task set, so held-out tasks are needed to show the discovered harness generalizes rather than memorizes the test. Together these two controls — matched-budget baselines and held-out evaluation — are what separate "the harness got better" from "we searched harder on the test."