Can targeted interventions on attention heads bridge the encoding-generation gap?
This explores whether you can fix the gap between what a model computes inside its layers and what it actually says — by intervening directly on specific attention heads or activation directions rather than retraining.
This explores whether you can fix the gap between what a model computes inside its layers and what it actually says — by intervening directly on specific attention heads or activation directions rather than retraining the whole model. The corpus suggests the answer is a qualified yes: the gap is real, it's often localized to a surprisingly small set of components, and that localization is exactly what makes surgical intervention possible.
The most direct evidence that encoding and generation come apart lives in Do transformers hide reasoning before producing filler tokens?. Models trained to hide their chain-of-thought actually compute the correct answer in layers 1-3, then *actively suppress* that representation in the final layers to emit format-compliant filler. The right answer is sitting there, recoverable from lower-ranked token predictions — it just never reaches the output. That's the encoding-generation gap in its starkest form: the knowledge is encoded but generation overwrites it.
Why targeted intervention is plausible comes from What mechanism enables models to retrieve from long context?. Fewer than 5% of attention heads do the work of pulling facts out of context, they're consistent across model families, and they're *causally* necessary — prune them and the model hallucinates even though the information is right there in the prompt. If a tiny, identifiable set of heads governs whether encoded information surfaces, then editing those heads is a far more precise lever than fine-tuning. The flip side appears in Does transformer attention architecture inherently favor repeated content?: attention has a built-in bias toward repeated and prominent tokens, and 'System 2 Attention' — regenerating the context to strip irrelevant material — interrupts that mechanism. That's intervention at the input-to-attention boundary rather than the head itself, but the same logic.
The cleanest demonstration that you can steer generation without retraining is Can we steer reasoning toward brevity without retraining?. A single direction extracted from 50 examples cuts reasoning length 67% while holding accuracy — training-free, generalizing across sizes. Pair this with Can models learn to ignore irrelevant prompt changes?, whose activation-level method (ACT) trains the model to produce identical internal states for clean and perturbed prompts, and you see the two flavors of the same idea: read off a behavior as a vector and push it, or train the internal representation to be stable. Both treat the residual stream as something you can edit rather than only retrain.
The honest caveat: the corpus shows interventions that *steer* behavior (brevity, invariance, retrieval) more than ones that provably close the deeper encoding-generation gap from Do transformers hide reasoning before producing filler tokens?. And there's a structural ceiling worth knowing — Can neural memory modules scale language models beyond attention limits? argues that some limits are architectural (attention's quadratic short-term memory needs a separate long-term module), not patchable by head-editing alone. So the surprising takeaway: the bottleneck is often not that the model failed to encode the answer, but that a handful of heads decide whether it ever reaches your screen — which makes 'bridging the gap' less a training problem and more a targeting problem.
Sources 6 notes
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Less than 5% of attention heads across all model families function as retrieval heads, are intrinsic to short-context models, dynamically activate by context, and are causally necessary for factuality. Pruning them causes hallucination despite information being present in context.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.