Why do AI benchmarks show rapid saturation from near-zero to near-perfect?

This explores why benchmark scores tend to leap from almost-zero to almost-perfect in a narrow window, and the corpus suggests the saturation curve is less a story about intelligence growing than about what a closed-ended test can and can't see. The cleanest reason is structural: a benchmark is a fixed set of auto-gradable questions, and once a model crosses the threshold where it has the relevant pattern, the remaining items fall almost all at once. Can frontier exams really measure cutting-edge AI capability? shows this directly — MMLU saturates while a harder expert-designed exam still discriminates, meaning saturation marks the exhaustion of a test's difficulty range, not the ceiling of capability. The jump looks dramatic because the test had no headroom left to register anything finer.

The more unsettling reason is that a high score and real competence can come apart entirely. Can AI pass every test while understanding nothing? argues that networks can produce identical, perfect outputs while their internal representations are incoherent and 'fractured' — standard benchmarks have no way to detect the difference. In the same spirit, Can genuine reasoning activation coexist with contaminated benchmarks? separates two things we usually conflate: genuine reasoning getting activated during training, versus benchmark numbers climbing because the test data leaked into pretraining. Both can rise together, so a fast climb to near-perfect may be part skill and part memorization of a contaminated, finite question set — and the curve can't tell you the ratio.

There's also a measurement-design reason the saturation is so steep. Do automated benchmarks hide what frontier AI systems can really do? points out that benchmarks privilege precisely-specified, cleanly-gradable tasks, which both overstate and understate what a system can do. That bias compresses a messy, continuous capability into a binary pass/fail per item, which is exactly the shape that produces sharp S-curves: narrow the question enough and the transition from 'can't' to 'can' looks instantaneous. Open-world evaluation of long, messy tasks smears that transition back out and catches emerging ability earlier — before the official benchmark notices anything.

Where the corpus gets surprising is on what saturation systematically misses. Why do AI assistants get worse at longer conversations? reports models scoring ~90% on single-shot instructions but collapsing to ~65% across natural multi-turn conversation — a saturated single-turn benchmark would call that solved. And Why does autoregressive generation fail at constraint satisfaction? shows hard ceilings that no amount of scaling moves, because the failure is architectural (autoregressive models can't retract a token the way a constraint solver must). So benchmarks saturate fast on the slice of behavior they sample, while whole capability regimes — recovery, retraction, long-horizon reliability — sit outside the frame.

The takeaway you might not expect: rapid saturation is partly an artifact of finite, narrow, leak-prone tests, and the most informative evaluations are the ones still far from saturated — expert frontier exams, open-world logs, multi-turn settings — because only an unsaturated test has any resolution left to measure with.