Does an AI that polishes its own answer in a fixed, checked loop face different risks than one rewriting itself with no ceiling?

This explores whether a model polishing its own output inside a fixed, checkable loop carries different dangers than a system that keeps rewriting itself with no ceiling. The clearest starting point is a survey of about 1,250 papers. It argues these are two different phenomena, not two points on one scale Are self-refinement and recursive self-improvement actually the same thing?. Bounded self-refinement is what industry already does: draft, critique, revise, and check the result against something you can evaluate. Open-ended recursive self-improvement is held back by limits you can measure today: it needs grounding in something real, it tends to collapse, and it runs into compute limits. So the first answer is that the risks differ because the two activities differ.

The risks of bounded refinement are mostly quality risks. Revising an answer over and over can reproduce 'overthinking' at the level of whole responses. Each pass adds noise, and nothing guarantees the answer improves Do iterative refinement methods suffer from overthinking?. That wastes compute and can make answers worse, but it isn't a runaway process. A useful framework splits self-improving agents into a slow loop that changes model weights and a fast loop that changes prompts, memory, and tools Do self-improving agents really split into two distinct loops?. Most recent progress has come in the fast loop, partly because those changes are cheap and can be undone. Whether a change can be reversed may matter more for risk than whether the loop is called 'bounded.'

Open-ended systems change what counts as a safety guarantee. The Darwin Gödel Machine improves itself by rewriting its own agent code. It drops the original idea of mathematically proving each change is beneficial and instead tests variants on benchmarks, keeping an evolving archive of them Can AI systems improve themselves through trial and error?. In a narrower setting, transformers trained repeatedly on their own correct answers went from 10-digit to 100-digit addition, with no sign of slowing across rounds Can transformers improve exponentially by learning from their own correct solutions?. Both show that compounding works when a reliable correctness check exists. But replacing proof with benchmarks means trusting the benchmark. Separate work shows that a model can score perfectly while its internal organization is fractured, and that weakness only appears under perturbation or new kinds of input Can models be smart without organized internal structure?.

Monitoring is where the two risk profiles separate most sharply. A check that looks at one action at a time can't express a rule about a sequence of actions. Steps that are each allowed can still add up to something unsafe Can stateless checks ever catch sequence-level constraint violations?. Bounded refinement can usually be judged one output at a time. Open-ended self-modification is a sequence by nature, so it needs monitors that track history. At the societal level, the Future of Life Institute reports that Anthropic published a post in June 2026 warning that recursive self-improvement could bring propaganda, job displacement, loss of control, and nonhuman minds replacing humans. The post reportedly urged labs to consider slowing or pausing certain paths of development Does recursive self-improvement pose serious risks to society?.

The takeaway is that 'bounded vs. open-ended' is a rough stand-in for three sharper questions. Is the evaluator fixed and trustworthy? Can the changes be undone? Is anyone watching the whole trajectory rather than single steps? A loop that edits weights, grades itself on a gameable benchmark, and runs under per-step checks is risky even if it stops after ten rounds. The corpus has few direct, side-by-side risk comparisons, so this synthesis connects papers that each cover one part of the question.