Do tokens beyond a critical threshold actually improve reasoning quality?
This explores whether spending more tokens on reasoning—longer chains of thought, more 'thinking time'—actually buys better answers, or whether there's a point past which extra tokens stop helping (and even hurt).
This explores whether more reasoning tokens past a certain point genuinely improve answer quality. The corpus is unusually direct here: the answer is no, and the relationship is non-monotonic. Pushing thinking tokens from ~1,100 up to ~16K dropped benchmark accuracy from 87.3% to 70.3% Does more thinking time always improve reasoning accuracy?, because models overthink easy problems and underthink hard ones. The same inverted-U shows up for chain-of-thought length: accuracy peaks at an intermediate length and declines past it, with the sweet spot moving *down* as the model gets more capable Why does chain of thought accuracy eventually decline with length?. Strikingly, RL training tends to drift toward shorter chains on its own as models improve—brevity emerges from the reward signal rather than being imposed.
The more interesting question is *why* extra tokens stop paying off, and here the corpus suggests the signal was never evenly distributed across tokens to begin with. Only about 20% of tokens carry high entropy and act as the real decision points that drive learning—training exclusively on those 'forking' tokens matches full-gradient performance Do high-entropy tokens drive reasoning model improvements?. Mutual-information analysis points at the same minority from another angle: specific tokens like 'Wait' and 'Therefore' spike in information about the correct answer, and suppressing them hurts reasoning while suppressing random tokens of equal count does not Do reflection tokens carry more information about correct answers?. Models even internally rank tokens by function, preserving symbolic computation while pruning grammar and meta-discourse first Which tokens in reasoning chains actually matter most?. So most of a long trace is padding around a few load-bearing moments—which is exactly why piling on tokens hits diminishing then negative returns.
It gets stranger. The tokens don't even have to be *correct* to help: models trained on deliberately corrupted, irrelevant traces hold their accuracy and sometimes generalize better out-of-distribution, suggesting traces act as computational scaffolding rather than meaningful reasoning Do reasoning traces need to be semantically correct?. And the reasoning may not need to be verbalized at all—latent-space approaches scale test-time compute through hidden-state iteration without emitting any visible thinking tokens Can models reason without generating visible thinking tokens?. In some setups transformers compute the answer in early layers, then actively overwrite it with format-compliant filler in the final layers Do transformers hide reasoning before producing filler tokens?. If much of the visible token stream is filler over computation that already happened, more tokens are buying theater, not thought.
There's a measurement trap lurking underneath all this. Supervised fine-tuning can raise benchmark accuracy while *cutting* the quality of reasoning steps by ~39%—the model produces right answers via post-hoc rationalization, and standard metrics miss it because they only score the final answer Does supervised fine-tuning improve reasoning or just answers?. So 'tokens improved quality' can be an illusion if you only look at correctness. Worth knowing too that sheer input length degrades reasoning well below the context window—accuracy fell from 92% to 68% with just 3,000 tokens of padding Does reasoning ability actually degrade with longer inputs?—and that for some questions step-by-step reasoning underperforms a direct answer entirely Why do some questions perform better without step-by-step reasoning?.
The catch that keeps this from being a simple 'shorter is better' rule: the critical threshold is invisible until you cross it, and it shifts with task difficulty, domain, and model. There's no reliable predictor—though difficulty estimators and runtime confidence signals can sometimes detect it dynamically How can we predict the optimal thinking token threshold?. So the honest answer is that tokens beyond the critical threshold don't improve quality and often degrade it, the useful work lives in a sparse minority of tokens, and the hard part is knowing where your threshold sits before you've blown past it.
Sources 12 notes
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Specific tokens like "Wait" and "Therefore" show sharp spikes in mutual information with correct answers. Suppressing them harms reasoning while suppressing equal random tokens does not, and representation recycling improves accuracy 20%.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Supervised fine-tuning improves final-answer accuracy on benchmarks but cuts Information Gain by 38.9 percent, meaning models generate correct answers through post-hoc rationalization rather than genuine inferential steps. Standard metrics miss this degradation because they only measure final correctness.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
The overthinking threshold depends on task difficulty, model training, and domain, but remains invisible until crossed. Recent work suggests difficulty estimators and runtime confidence signals can detect thresholds dynamically.