Fixing an AI agent's behavior isn't hard — the bottleneck is finding all the scattered places that behavior actually lives.
What makes behavior localization the bottleneck in agent harness evolution?
This explores why the hard part of improving an agent's scaffolding (its prompts, tools, and orchestration code) isn't writing better edits but finding all the scattered places where a single behavior actually lives.
This explores why the hard part of improving an agent's scaffolding isn't writing better edits but finding all the scattered places where a single behavior actually lives. The core claim is a representational mismatch: when you ask a harness to change one behavior, that behavior isn't sitting in one function — it's smeared across files, tool definitions, prompt fragments, and pipeline stages. So the agent evolving the harness spends most of its effort just locating every site that implements the behavior, and any edit that misses a location leaves the old behavior partly intact Why is finding distributed behavior code so hard?. Generation was never the constraint; localization is.
The cleanest evidence that localization is the true bottleneck comes from what happens when you remove it. Reorganizing a harness around runtime behavior — building an explicit behavior-to-code map — let *weaker* planners match the code-localization precision and recall of much stronger models, while lifting win rates 10–19 points and cutting planner tokens Can explicit behavior maps help weaker planners compete with stronger models?. That's the tell: if handing a model a map closes most of the gap, then what the stronger model was buying you all along was better search through distributed code, not better ideas about what to change.
This also reframes a puzzle about model tiers. The raw capacity to *propose* useful harness edits is roughly flat across model sizes — but the capacity to actually *benefit* peaks in the middle, an inverted U Do stronger models always evolve harnesses better?. Localization difficulty explains both tails: weak models can't reliably find and invoke the right code paths, and even strong models get tripped up threading a change faithfully through every distributed site. The bottleneck isn't intelligence about the edit; it's the structural work surrounding it.
Worse, when localization is hard, the path of least resistance is to stop generalizing. Inspection of evolved harnesses shows most edits just cache task-specific fixes the agent could have rediscovered in a single rollout, rather than distilling a reusable strategy Do harness edits learn reusable strategies or memorize task fixes?. If you can't cleanly find where a behavior lives, you patch the one spot that failed on this task instead of restructuring the behavior everywhere — memorization is what localization failure looks like from the outside. And because harness evolution is itself a search loop, apparent gains have to be measured against a matched search budget, or you're crediting design for what was really just more attempts at finding the right spot How should we measure gains from automatic harness evolution?.
Step back and the bottleneck marks a fault line in self-improving agents generally. These systems split into a slow parametric loop (updating weights) and a fast non-parametric loop (updating prompts, memory, tools), and progress has rushed toward the fast loop because scaffold edits are cheap and reversible Do self-improving agents really split into two distinct loops?. But the fast loop only pays off if edits land coherently across the whole scaffold — which is exactly where distributed behavior defeats it. It's worth noticing that the alternative traditions dodge the problem by construction: skill libraries like VOYAGER externalize each capability as one indexed, composable unit so a behavior has a single home Can agents learn new skills without forgetting old ones?, and memory-based agents route improvement through discrete memory operations rather than surgery on tangled code Can agents learn continuously from experience without updating weights?. The lesson underneath: harnesses are hard to evolve not because they're complex but because they scatter one idea across many places, and the fix is representational — give each behavior an address before you try to change it.
Sources 8 notes
The core difficulty in evolving production harnesses is not generating edits but finding every code location that implements a behavior. Harnesses distribute single behaviors across files, functions, and stages, creating a representational mismatch between behavioral requests and structural code organization.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Show all 8 sources
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement