Why does an AI that only learns from its best outputs eventually run out of room to improve?

This explores why a self-improvement loop that optimizes only for quality ends up collapsing instead of compounding. The short version the corpus suggests: quality is only half of what keeps a loop healthy. When you reward only "better outputs," you quietly punish variety — and variety is the fuel the loop runs on. The clearest articulation is the idea that pure self-improvement is structurally circular Can models reliably improve themselves without external feedback?: a model that trains on its own filtered-for-quality outputs narrows toward the modes it already favors, a failure the note names diversity collapse, often alongside reward hacking, where the model learns to satisfy the quality signal rather than the underlying goal.

There's a deeper reason quality-only optimization is unreliable, not just narrowing: a model can only improve itself where it can verify better than it can generate. That generation–verification gap is described as a formal ceiling on self-improvement What limits how much models can improve themselves? What limits autonomous capability in large language models? — and if your quality filter is itself the model's own judgment, you're optimizing against a flawed ruler. Push hard on it and you amplify the ruler's blind spots. This is why the reliable methods, as one synthesis puts it, all "smuggle in" something external What actually constrains AI systems from learning misalignment?: a past model version, a third-party judge, user corrections, or tool feedback that the loop can't fake.

The diversity side has a concrete mechanism worth knowing. Preference tuning's effect on diversity isn't uniform — it collapses lexical variety in code (where there's a single correct answer to converge on) but can actually increase it in creative writing Does preference tuning always reduce diversity the same way?. So "optimize only quality" is most corrosive exactly where quality looks like convergence: the loop keeps narrowing toward one answer-shape and loses the spread of attempts it needs to discover anything new. Relatedly, optimizing a crude quality signal can distort the model in ways that aren't about correctness at all — binary correct/wrong rewards degrade calibration, training the model to guess confidently because confident wrong answers aren't penalized Does binary reward training hurt model calibration?.

What's striking — and probably the thing you didn't know you wanted to know — is that the loops that *don't* collapse are the ones engineered to preserve disagreement or hard correctness rather than self-rated quality. Asymmetric self-play survives because a proposer is rewarded for generating *calibrated, varied* problems while the solver learns from majority-vote verification, so the system manufactures its own diversity instead of consuming it Can language models improve themselves without any external training data?. Self-improving transformers reach exponential length generalization by filtering on *verifiable* correctness (does this 100-digit sum check out?), not a soft quality score Can transformers improve exponentially by learning from their own correct solutions?. And there's a second-order trap: as a model trains on its own outputs, its own earlier errors leak into context and amplify non-linearly Do models fail worse when their own errors fill the context? — so a quality-only loop doesn't just stop improving, it can actively poison itself.

The through-line: collapse isn't caused by optimizing quality per se — it's caused by optimizing quality *alone*, using the model's own judgment as the standard. Healthy loops pair a quality signal with something it can't game: external verification, an explicit diversity/calibration term, or genuinely checkable correctness.