Does chain-of-thought accuracy degrade with longer reasoning traces?
This explores whether making a model 'think' longer actually helps — and the corpus says no, accuracy peaks at a middle length and then falls off, for reasons that have less to do with the problem and more to do with how the model was trained.
This explores whether longer chain-of-thought (CoT) traces hurt accuracy rather than help, and the collection answers with a fairly consistent picture: yes, past a sweet spot, more reasoning makes things worse. The cleanest framing is an inverted-U curve — accuracy rises with reasoning length up to a point, then declines Why does chain of thought accuracy eventually decline with length?. One striking measurement: pushing thinking tokens from roughly 1,100 up to 16,000 dropped benchmark accuracy from 87% to 70%, because models overthink easy problems and meander into errors Does more thinking time always improve reasoning accuracy?. So the relationship is non-monotonic — length is not free, and there's a real cost to letting a model ramble.
The more interesting question is *why* extra length backfires, and here the corpus reframes what trace length even measures. Long traces are often assumed to mean 'harder problem requiring more computation,' but controlled maze experiments show trace length tracks difficulty only on familiar, in-distribution problems and decouples entirely once you step outside the training distribution — length mostly reflects how close the problem is to memorized training schemas, not adaptive thinking Does longer reasoning actually mean harder problems?. That connects to a failure mechanism: as a trace grows, each new token leans more heavily on the immediately preceding ones, and this 'local memorization' accounts for up to 67% of reasoning errors, getting worse as complexity and distributional shift increase Where do memorization errors arise in chain-of-thought reasoning?. Longer traces give error more room to compound.
A second angle: much of the length is doing no computational work at all. 'Chain of Draft' showed that minimal reasoning chains matched verbose CoT accuracy while using only 7.6% of the tokens — meaning over 90% of a typical trace serves style and documentation, not problem-solving Can minimal reasoning chains match full explanations?. Relatedly, attention-map analysis found that verification and backtracking steps get almost no downstream attention, so you can dynamically prune 75% of reasoning steps without losing accuracy Can reasoning steps be dynamically pruned without losing accuracy?. If most of the length is inert, padding it out mostly adds opportunities to drift.
This all sits inside a deeper claim the corpus keeps returning to: CoT is pattern-matched imitation of reasoning's *form*, not genuine logical inference What makes chain-of-thought reasoning actually work? Why does chain-of-thought reasoning fail in predictable ways?. Format and spatial structure shape behavior far more than logical content — invalid reasoning prompts work nearly as well as valid ones What makes chain-of-thought reasoning actually work? — and the whole apparatus degrades predictably once you leave the training distribution Does chain-of-thought reasoning actually generalize beyond training data?. A longer trace doesn't deepen the inference; it extends a fragile imitation further from solid ground. Worth knowing too: the degradation isn't only about the model's *own* output length — reasoning accuracy drops from 92% to 68% with just 3,000 tokens of input padding, well below the context limit Does reasoning ability actually degrade with longer inputs?. The surprising takeaway is that better models actually prefer *shorter* chains — simplicity emerges naturally from reward training rather than being forced — so brevity is a sign of capability, not a shortcut around it Why does chain of thought accuracy eventually decline with length?.
Sources 11 notes
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.
The PI framework categorizes reasoning into six types and uses attention maps to identify that verification and backtracking steps receive minimal downstream attention. Selecting only high-attention steps preserves accuracy while cutting reasoning length substantially.
CoT systems reproduce the form of reasoning through pattern matching rather than performing genuine logical inference. This explains why format effects dominate content, why structurally invalid prompts succeed, and why stronger reasoning models become less instruction-compliant.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.