How do frontier models maintain agreement scores above 90 percent across reasoning tasks?
This explores how frontier models hit very high scores on reasoning benchmarks — and the corpus mostly complicates the premise, suggesting those scores are more fragile, and sometimes more illusory, than the headline number implies.
This reads the question as asking what lets frontier models post sky-high scores on reasoning tasks — and the most useful thing the collection has to say is that those numbers are often less solid than they look. Several notes show that a model scoring 90-plus on a clean benchmark can collapse the moment conditions shift. Reasoning accuracy drops from 92% to 68% with just 3,000 tokens of harmless padding, far below the context window's limit, and chain-of-thought prompting doesn't rescue it Does reasoning ability actually degrade with longer inputs?. Push a task outside the training distribution — in topic, length, or format — and chain-of-thought produces fluent prose with broken underlying logic Does chain-of-thought reasoning actually generalize beyond training data?. So a high agreement score is partly a statement about how close the test sits to what the model has already seen.
That last point sharpens into something stronger: models often succeed by matching familiar instances rather than running a real algorithm. Failures cluster at the boundary of novelty, not complexity — a model handles a long reasoning chain fine if it trained on similar instances, and stumbles on a short one that's genuinely unfamiliar Do language models fail at reasoning due to complexity or novelty?. Worse, some of the apparent success isn't reasoning at all. When constraints are stripped from a task, twelve of fourteen models get *worse*, because they were scoring well by quietly defaulting to the harder, safer option rather than actually evaluating the constraints Are models actually reasoning about constraints or just defaulting conservatively?. The score was real; the reasoning behind it wasn't.
Where genuine capability does live, the collection suggests it's mostly elicited, not freshly built. Base models already carry latent reasoning ability that five independent methods — RL steering, critique fine-tuning, decoding tweaks, feature steering, RLVR — all unlock rather than create Do base models already contain hidden reasoning ability?. And the training signal is concentrated: only about 20% of tokens, the high-entropy 'forking points,' carry the learning, and training on just those matches full updates Do high-entropy tokens drive reasoning model improvements?. High scores, in other words, come from sharpening a few pivotal decisions in capability that was already latent.
The surprising twist is that more effort doesn't keep the score climbing — it can sink it. Accuracy follows an inverted-U with thinking length: pushing from ~1,100 to ~16K thinking tokens dropped one benchmark from 87.3% to 70.3%, as models overthink easy problems and underthink hard ones Does more thinking time always improve reasoning accuracy?, and the optimal chain length actually *shrinks* as models get more capable Why does chain of thought accuracy eventually decline with length?. If you want the levers that genuinely push real accuracy up rather than inflate it, the corpus points to better reward design — using the model's own answer confidence as a reward to fix calibration while strengthening reasoning Can model confidence work as a reward signal for reasoning?, and training judges that reason about each step instead of flatly classifying it Can judges that reason about reasoning outperform classifier rewards?. The thing you didn't know you wanted to know: a 90% agreement score is best read not as a ceiling the model reaches, but as a measurement of how familiar, well-bounded, and constraint-shaped the test was.
Sources 10 notes
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.