SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Why does chain of thought accuracy eventually decline with length?

Explores why longer reasoning chains don't always improve answers, and how the optimal length shifts based on task difficulty and model capability.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques

The "longer is better" assumption for CoT has an empirical ceiling: task accuracy initially improves with CoT length, reaches a peak, then decreases. The inverted-U curve applies across models and tasks, and its peak location follows consistent patterns.

Two scaling laws for optimal CoT length:

  1. Difficulty scaling — optimal length increases with task difficulty. Harder problems benefit from longer chains because more decomposition steps are needed. This part matches intuition.

  2. Capability scaling — optimal length decreases with model capability. More capable models find more efficient paths to correct answers and require fewer steps. Using the same long chains for a more capable model is counterproductive.

The second law has a practical consequence: treating all models identically (same token budget, same chain length) misallocates compute. A model that can solve a problem in 5 steps should not be given budgets designed for a 20-step solution.

Simplicity bias as a training-emergent property: RL training reveals this dynamic in action. As RL training improves accuracy, models gravitate toward shorter CoTs — not because they were explicitly trained to be concise, but because shorter chains produce correct answers and RL rewards correct answers. The simplicity bias emerges automatically from the reward signal.

This connects to Why do correct reasoning traces contain fewer tokens? — the same empirical signal: shorter chains are correct chains. The inverted-U explains why: length past the optimal point introduces accumulation of decomposition errors and contextual noise (see Do models fail worse when their own errors fill the context?).

The practical implication: train on optimally-lengthed CoTs (not maximal-length), and at inference, use length-aware filtering to discard excessively long chains. The simplicity bias is not a failure mode — it is a signal of genuine capability.

Inquiring lines that read this note 274

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Does chain-of-thought reasoning reveal how models actually think or merely imitate reasoning? How does scaling reasoning capabilities affect models' appropriate abstention behavior? What prevents language models from performing systematic logical reasoning? How should recommendation systems balance individual preference and diversity? Does scaling reasoning capability create fundamental tradeoffs in control and reliability? Can minimal training unlock latent reasoning already present in base models? Can reasoning traces reveal actual model reasoning versus plausible output? How do thinking tokens exhibit diminishing returns in reasoning? Can latent reasoning match or exceed explicit reasoning performance? What limits language model accuracy in evaluating ideas? When should retrieval systems decide to fetch new information? How does fine-tuning trade off accuracy against reasoning quality? When does parallel reasoning outperform sequential reasoning with the same token budget? Why do language models struggle to implement user intent accurately from prompts? Why does self-revision amplify confidence in wrong model answers? Can confidence signals reliably detect flawed reasoning in language models? How should retrieval strategies adapt to multi-step reasoning demands? Why don't better reasoning capabilities improve theory of mind performance? What prediction granularity best trains models to generate reliable reasoning? What are the fundamental limits of prompting for language models? Can humans reliably detect and resist AI-generated misinformation? Can inference-time computation adaptively substitute for static model capacity? What gaps exist between benchmark performance and real deployment outcomes? Can AI systems evade safety evaluations through reasoning manipulation? Do persona-based approaches introduce systematic biases in user simulation? What makes agent memory systems durable and reusable across sessions? Can reasoning models use reflection to correct their initial outputs? What makes reasoning traces effective supervision even when they're incorrect? Should models ask for clarification when facing ambiguous or under-specified information? Does augmenting symbolic reasoning improve LLM logical reasoning ability? What makes process supervision effective for training complex reasoning models? Does reinforcement learning create genuinely new reasoning capabilities or only refine existing ones? How does RLHF training shape models to prioritize agreement over accuracy? Can language models reason beyond surface pattern matching? How do interpretive frames override surface features in text comprehension? How susceptible are language models to conversational persuasion and belief change? What explains the gap between benchmark scores and true reasoning capability? How do reward signal properties affect model reasoning and safety? Does AI-assisted research sacrifice exploration breadth for productivity gains? How do models learn from self-generated outputs without cascading failures? Do language models encode knowledge that influences generation, or primarily imitate surface patterns? How do curriculum design and feedback approaches affect model learning? How does model capacity affect learning performance on diverse downstream tasks? Can artificial systems establish authority in domains requiring expert judgment? How does optimization for reward create emergent misalignment in language models? How can we reduce inherent biases in LLM-based evaluation judges? How do users confuse explanation quality with actual system accuracy? Can AI systems achieve real improvement without external human feedback? Can external verification systems adequately replace learned reasoning in AI outputs? Why does polished AI output gain credibility despite fundamental verifiability problems? How do AI systems determine and balance multiple competing objectives?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 167 in 2-hop network ·dense cluster Open in graph ↗
Why does chain of thought accuracy eventually de… Why do correct reasoning traces contain fewer toke… Does more thinking time always improve reasoning a… Does extended thinking actually improve reasoning … Why does parallel reasoning outperform single chai… Can minimal reasoning chains match full explanatio… Does gradually tightening token budgets beat fixed…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do correct reasoning traces tend to be shorter than incorrect ones?How should inference-time token budgets vary across models of different capability levels?Can parallel reasoning chains outperform longer sequential chains with the same compute?Why does extended thinking increase output variance without improving reasoning quality?
Original note title

optimal cot length follows an inverted-u — more capable models prefer shorter cot