Can latent space represent reasoning dimensions that text cannot?
This explores whether reasoning carried out in a model's internal vector space (its 'latent space') can hold moves or distinctions that never surface as words — and what the corpus suggests we gain by letting models think without writing their thoughts down.
This explores whether reasoning carried out in a model's internal vector space can represent things that text cannot — and the corpus makes a surprisingly strong case that it can. The starting clue is that verbalized chain-of-thought may be a habit, not a requirement: several architectures (depth-recurrent models, Heima, Coconut) scale test-time compute by iterating on hidden states rather than emitting tokens, which suggests that spelling reasoning out in language is a training artifact rather than where the reasoning actually lives Can models reason without generating visible thinking tokens?. If true, the interesting question flips: what can the latent format do that the text format can't?
The clearest answer is uncertainty. Text forces a single committed token at each step, but a continuous latent state can hold a distribution over possibilities at once. GRAM replaces deterministic latent updates with stochastic sampling so a model can carry several live hypotheses and explore ambiguous problems with multiple valid strategies — something a single written chain literally cannot represent Can stochastic latent reasoning help models explore multiple solutions?. That same stochastic-latent move also unlocks a different geometry of search: instead of only reasoning *deeper* (longer serial chains), models can reason *wider* by sampling parallel latent trajectories that probe the solution space independently, sidestepping the latency cost of depth Can reasoning systems scale wider instead of only deeper?. Latent thought vectors even open scaling axes that have nothing to do with parameter count, coupling fast local learning with a slow global decoder so reasoning can grow along a dimension text-token scaling never touches Can latent thought vectors scale language models beyond parameters?.
There's also an abstraction argument. Meta's Large Concept Model reasons over whole-sentence embeddings in a language-agnostic space and only decodes to words at the end — planning at the level of ideas rather than the next token, which produces more coherent structure than flat generation Can reasoning happen at the sentence level instead of tokens?. This hints that latent space isn't just a faster container for the same reasoning; it can operate at a granularity (concepts, plans) that token-by-token text struggles to hold.
But the corpus also draws the boundary lines. Latent space is not infinitely expressive: LLM embeddings turn out to collapse twenty-eight semantic axes into just three principal components matching a human-like structure, and intervening on one feature unavoidably drags aligned features along — so the space is entangled and lower-dimensional than its size suggests Do LLM semantic features organize along human evaluation dimensions?. Even something as simple as reasoning verbosity turns out to be a single linear direction you can steer through activation space Can we steer reasoning toward brevity without retraining? — which cuts both ways: it shows latent space encodes reasoning *qualities* text only implies, but also that those qualities are surprisingly low-dimensional. And the skeptic's reading is that a lot of what text adds is decoration anyway: Chain of Draft matches full chain-of-thought accuracy on 7.6% of the tokens, meaning ~92% of written reasoning served style and documentation, not computation Can minimal reasoning chains match full explanations?.
The thread worth pulling: if base models already contain latent reasoning that minimal training merely *elicits* rather than creates Do base models already contain hidden reasoning ability?, then the real payoff of latent-space reasoning may not be representing exotic new dimensions, but reaching reasoning that was always there and that verbalization was quietly throwing away — including the ability to stay uncertain instead of committing to the first plausible sentence.
Sources 9 notes
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent distributions over solutions rather than single predictions. This allows handling of ambiguous problems and multiple valid strategies that deterministic designs cannot represent.
GRAM shows that stochastic latent transitions enabling parallel trajectory sampling sidestep the serial latency cost of depth-only scaling. Width matches token-level parallelism benefits: independent paths sample the solution space without variance inflation.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
Meta's Large Concept Model operates on sentence embeddings rather than tokens, reasoning in a language-agnostic space before decoding to any target language. This hierarchical approach with paragraph-level planning produces more coherent output than flat token generation.
Twenty-eight semantic axes in LLM embeddings reduce to three principal components matching human EPA structure. Intervening on one feature predictably shifts aligned features proportionally, creating unavoidable off-target effects that reflect how meaning is fundamentally organized.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.