Does an AI that simply polishes its own answers ever tip into a system that keeps rebuilding and improving itself?

This explores the gap between two things that often get lumped together: an AI that refines its own work inside a fenced-off, checkable space, and an AI whose improvements build on each other with no fixed ceiling. A large survey of roughly 1,250 papers argues that these are different phenomena and not two points on one scale Are self-refinement and recursive self-improvement actually the same thing?. Bounded self-refinement is already standard industry practice. Open-ended recursive self-improvement is still held back by limits you can measure today: it needs grounding in something outside the model, it tends to collapse, and it costs a lot of compute.

The clearest dividing line is where the 'is this better?' signal comes from. A model can only improve itself when it is better at checking answers than at producing them. This gap grows with model size, but it disappears for factual tasks, where checking is no easier than recalling What limits how much models can improve themselves?. When a system tries to improve with no outside reference, it runs into reward hacking and its outputs become less and less varied. The methods that do work quietly bring in an outside anchor, such as an earlier model version, a separate judge, user corrections or tool results Can models reliably improve themselves without external feedback?. Even the Darwin Gödel Machine, often cited as open-ended, keeps itself grounded by testing every variant against real coding benchmarks and keeping an evolving archive of agent versions. It doesn't try to prove its changes are improvements Can AI systems improve themselves through trial and error?. A related distinction is who sets the goal. One debate participant argues that the real threshold is reached when AIs propose their own research objectives and pursue them without drifting, instead of optimizing targets that humans set Can AIs learn to specify their own research objectives?.

It also matters which part of the system is changing. Self-improving agents tend to split into a slow loop that updates the model's weights and a fast loop that updates prompts, memory and tools. Most recent progress has come in the fast loop, because changing the scaffolding around a model is cheap and easy to undo Do self-improving agents really split into two distinct loops?. Bounding that fast loop deliberately pays off. SkillOpt found that agents improve more reliably when edits to their instructions face three controls: a limit on how much can change per step (a 'learning rate' for text), a test on held-out examples before an edit is accepted, and a buffer that keeps rejected edits as lessons Does constraining edits make skill learning more stable?. A counterintuitive finding: models at every capability level write useful harness edits about equally well, but mid-tier models gain the most from them. Weak models fail to use the harness, and the strongest models don't follow the edited instructions faithfully Do stronger models always evolve harnesses better?.

Where does bounded refinement start to look open-ended? One marker is a system that improves the method it uses to improve. In bilevel autoresearch, an outer loop read the inner loop's code, found its bottlenecks and wrote new search mechanisms while running, which gave a 5× gain on a GPT pretraining task Can an AI system improve its own search methods automatically?. A survey of co-evolving systems describes a similar step-by-step shedding of human design. First the agent's peers adapt, then its environment and feedback, and finally the evolution mechanism itself. It also notes that a single agent improving itself in a static setting tends to stall Can agents evolve beyond the constraints humans engineer?. Even so, rough modeling suggests that real-world loops are getting stronger but are not yet self-sustaining. Overall acceleration depends on multiplying the strength of every feedback pathway, so one weak link slows the whole loop Are AI feedback loops strong enough to sustain recursive self-improvement?.

The distinction matters beyond engineering. According to the Future of Life Institute's account, Anthropic's June 2026 post warned that recursive self-improvement could bring propaganda, job displacement and loss of control, and it urged labs to consider slowing or pausing some development paths Does recursive self-improvement pose serious risks to society?. The takeaway is that the features that make self-refinement safe and effective today are the same features open-ended improvement would remove: outside checks, human-set goals, and limits on how much can change at once. Watching whether those features hold is one practical way to tell which kind of system you're looking at.