Do representations in models causally influence text generation?
This explores whether a model's internal representations — the hidden activations moving through its layers — actually steer the words that come out, or whether they're a side effect you can't reach by ordinary means.
This explores whether a model's internal representations causally drive its output, and the corpus says yes — but in a way that's more revealing than a simple yes. The sharpest evidence is negative: when researchers try to change a model's behavior with words alone, they often can't. One study finds that language models ignore information in their prompt whenever training-baked associations are strong enough; "textual prompting alone cannot override strong priors," and the only thing that works is reaching in and intervening on the representations themselves Why do language models ignore information in their context?. That's the cleanest causal signature you can ask for — the representation is the lever, and the prompt is just a weak handle on it.
The causal picture gets stranger when you watch representations get overwritten. Logit-lens analysis of models trained with hidden chain-of-thought shows the correct answer is already computed in layers 1–3, then actively suppressed in later layers so the model can emit format-compliant filler instead Do transformers hide reasoning before producing filler tokens?. So representations don't just influence generation — they can win the computation and still lose the output, because something downstream edits them. The text you read is the last edit, not the real reasoning.
Why representations matter so much becomes clearer once you stop thinking of a model as a database. One framing argues transformers don't store knowledge as retrievable records — they transmit it as flowing activations through the residual stream, closer to oral performance than to a filing cabinet, which is exactly why model knowledge is contextual, hard to edit, and "inseparable from generation" Do transformer models store knowledge or generate it continuously?. If knowledge only exists as flow, then the flow is the cause; there's no separate stored fact to point at. The same logic explains the ceiling on prompting: optimizing a prompt only reorganizes what's already in the training distribution and can't inject anything new Can prompt optimization teach models knowledge they lack?.
The most direct demonstration that representations cause output is to scale them deliberately. Latent-thought language models add a vector of "thought" that's learned separately from the weights, and growing that latent space improves reasoning independent of parameter count — a representational dial that changes what the model produces Can latent thought vectors scale language models beyond parameters?. Diffusion LLMs push the same idea by letting reasoning be embedded in masked positions and refined in place alongside the answer, so you can literally watch answer-confidence converge while the underlying representation keeps changing Can reasoning and answers be generated separately in language models?.
The thing worth taking away: the output text is a lossy, sometimes deliberately edited readout of a representational process — which is why you can't always reason about a model from its words, why editing it is hard, and why interpretability work targets the activations rather than the prompt. The generation isn't where the action is; it's the residue.
Sources 6 notes
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Transformers organize knowledge as flowing activations rather than retrievable archives, mirroring oral cultures where knowledge exists only in performance. This explains why model knowledge is contextual, difficult to edit, and inseparable from generation.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
ICE shows that bidirectional attention in diffusion LLMs enables in-place prompting—embedding reasoning directly in masked positions refined alongside answers. Answer confidence converges early while reasoning continues refining, allowing early-exit mechanisms to cut compute by 50% while maintaining accuracy.