Where does LLM reasoning actually happen during generation?

The field studies "LLM reasoning" without agreeing on what the primary object of study is. Three views coexist but make incompatible predictions:

H2 (surface CoT): Multi-step reasoning is primarily mediated by explicit surface chain-of-thought. The chain IS the reasoning. This requires surface traces to provide the most stable causal leverage — but ordinary CoT is often useful without being reliably faithful, and its role varies sharply across tasks.

H0 (generic serial compute): Most apparent reasoning gains are better explained by generic serial compute than by any privileged representational object. More tokens = more FLOPs, regardless of what those tokens say. This requires matched serial compute to explain most gains — but extra budget alone cannot explain why specific internal states, features, or trajectories can predict or alter reasoning behavior.

H1 (latent-state trajectories): Multi-step reasoning is primarily mediated by latent-state trajectories, with surface CoT serving only as a partial interface. Task-relevant commitment arises in hidden-state dynamics that are only partly verbalized, or not verbalized at all.

The difficulty is that recent methods typically move several factors at once: CoT prompting changes both visible traces and compute allocation; latent reasoning methods change both hidden-state dynamics and compute budget; test-time scaling changes compute and usually changes the output path. Without designs that explicitly disentangle these three factors, experimental results cannot distinguish which hypothesis they support.

The paper argues H1 should be the default working hypothesis — not as a task-independent verdict, but because the strongest evidence currently available points toward latent-state dynamics as having the most stable causal leverage. The recommendation: treat latent-state dynamics as the default object of study and design evaluations that explicitly separate surface traces, latent states, and serial compute.

This framework organizes several existing findings. Because Do language models actually use their reasoning steps?, the H2 assumption is empirically weakened — if surface traces aren't causally faithful, they cannot be the primary reasoning medium. Because Does chain-of-thought reasoning reflect genuine thinking or performance?, H2 fails specifically on easy tasks (where the answer is determined before CoT begins) while H1 and H0 remain viable. Because Can we trigger reasoning without explicit chain-of-thought prompts?, direct latent intervention provides causal evidence for H1 that neither H2 nor H0 can explain.

Additional evidence converges from multiple angles. Because Why does reasoning training help math but hurt medical tasks?, the layer separation provides architectural grounding for H1: reasoning is a latent higher-layer process, not a surface token-generation phenomenon. Because Why do language models fail to act on their own reasoning?, even when the surface trace (rationale) is correct, the latent computation (action selection) diverges — a behavioral signature of the surface-latent disconnect that H1 predicts. And because Can we measure how deeply a model actually reasons?, there now exists an H1-native measurement tool: DTR tracks latent computational depth per token rather than surface trace properties, and it outperforms surface-level metrics as an accuracy predictor.

The sharpest implication: the field's default assumption (H2) may be distorting research priorities. If the reasoning object is latent, then benchmarks that evaluate chains, faithfulness metrics that read traces, and interpretability methods that parse CoT are all measuring a secondary phenomenon.

Source: Cognitive Models Latent Paper: LLM Reasoning Is Latent, Not the Chain of Thought