When an AI rewrites its own setup to get smarter, it mostly just memorizes the last problem's answer — not a transferable lesson.

This explores why an agent's self-edits to its own harness end up caching task-specific patches rather than distilling reusable strategy. The most direct answer in the corpus is almost deflationary: when researchers actually read the evolved edits, the changes look rational across prompt and tool layers, but most of them just persist a shortcut the agent could have rediscovered on its own in a single try Do harness edits learn reusable strategies or memorize task fixes?. The edits cache fixes for problems that were already solvable; they rarely convert a genuine failure into a success. So 'memorization' here isn't a bug in the search — it's what a greedy loop naturally produces when the cheapest way to raise the score is to hard-code the answer to the task in front of it.

The deeper reason becomes clear once you notice that harness evolution is itself a search loop. Its reported gains are confounded with plain search effort, so unless you compare against a matched task-level search budget and evaluate on held-out tasks, you can't tell whether the harness got smarter or just got more attempts — and 'more attempts, memorized' is exactly the null hypothesis How should we measure gains from automatic harness evolution?. Memorization is the path of least resistance for any optimizer graded on the training set; generalization only shows up when the evaluation forces it.

There's a suggestive parallel in how models absorb instructions and rewards more generally. Instruction tuning turns out to teach the output format, not task understanding — models trained on semantically empty or even wrong instructions perform about as well, because what transfers is knowledge of the answer space, not reasoning Does instruction tuning teach task understanding or output format?. RL post-training similarly tends to collapse onto one dominant format and suppress alternatives rather than expand capability Does RL training collapse format diversity in pretrained models?. Evolved harness edits sit in the same family: local optimization pressure rewards surface fixes that satisfy the visible signal, and the strategic, generalizing content is precisely what's hardest to elicit.

What's striking is that the corpus also shows the opposite is achievable when memory is structured deliberately. The ACE framework treats context as an evolving playbook, using generate-reflect-curate loops and incremental updates so that hard-won detail isn't compressed away into a bland shortcut — and it posts double-digit gains without labeled supervision Can context playbooks prevent knowledge loss during iteration?. Reflexion gets agents to learn from failure by storing verbal self-diagnoses in episodic memory, and it works precisely because binary success/failure feedback blocks rationalization Can agents learn from failure without updating their weights?. VOYAGER compounds reusable, composable skills into a library so new abilities build on old ones instead of overwriting them Can agents learn new skills without forgetting old ones?. The contrast is the real lesson: naive harness evolution memorizes because nothing in the loop rewards abstraction; the systems that generalize add explicit machinery — reflection, curation, composition — that forces a fix to be turned into a strategy before it's stored.

One last wrinkle worth knowing: the ability to write useful edits and the ability to benefit from them aren't the same. Edit-writing capacity is roughly flat across model tiers, but the payoff follows an inverted-U, peaking in mid-tier models — weak models never invoke the harness, and strong ones struggle to follow their own instructions faithfully Do stronger models always evolve harnesses better?. And how the harness is represented matters: reorganizing it around runtime behavior rather than raw code lets even weaker planners localize correctly Can explicit behavior maps help weaker planners compete with stronger models?. So whether an edit memorizes or generalizes is not fixed by the agent's raw strength — it's shaped by the loop's incentives and the harness's structure, both of which you can design.