Why do transformer attention patterns show positional and sequential bias across tasks?
This explores why transformers consistently lean on where a token sits and how often it recurs — and whether that's a built-in feature of the attention mechanism rather than a fixable bug.
This explores why transformers consistently lean on where a token sits and how often it recurs, and whether that bias is baked into the attention machinery rather than learned from data. The corpus suggests the answer is largely structural: soft attention is a weighted averaging operation, and that math has a built-in tilt. Because attention assigns weight to every token and then sums, content that is repeated or sits prominently in context gets over-weighted regardless of whether it's actually relevant — a positive feedback loop that amplifies framing and opinion before any human-feedback tuning even gets a chance to act Does transformer attention architecture inherently favor repeated content?. The bias isn't a mistake the model makes; it's what the operation does by default.
The deeper reason shows up when you compare how transformers read versus how people read. Humans selectively suppress irrelevant words and let a few resonate; transformers integrate every token additively through parallel aggregation, with no mechanism for selective frame-activation. That's why they miss jokes and wordplay so reliably — not a knowledge gap, but a missing cognitive operation that would let later words override earlier ones Why do AI systems miss jokes and wordplay so consistently?. Positional and sequential bias is the flip side of the same coin: without selective suppression, position and repetition become the main signals available for deciding what matters.
There's a provenance question too — is this learned or architectural? A causal experiment swapping pretraining seeds and finetuning data found that cognitive biases are planted during pretraining and only nudged by instruction tuning Where do cognitive biases in language models come from?. Combined with the structural argument, that points to bias entering at two levels at once: the attention operation tilts a certain way, and pretraining over natural-language sequence statistics cements which positions and patterns get trusted. It also helps explain why these biases survive RLHF rather than being trained away — and why RLHF can even leave a model representing truth internally while expressing something else Does RLHF make language models indifferent to truth?.
What's striking is how this same tilt resurfaces in tasks that look unrelated to position. Compositional reasoning collapses to matching memorized subgraphs from training data, with errors compounding step by step — a sequential fragility that comes from leaning on familiar patterns rather than systematic rules Do transformers actually learn systematic compositional reasoning?. And transformers reproduce human content effects item-for-item, where what a statement *says* bleeds into judgments about its logical form, because content and form aren't cleanly separable in the architecture Do language models show the same content effects humans do?. Position, repetition, content — they're all the same averaging operation refusing to be neutral.
If there's a way out, the corpus hints it's architectural intervention rather than more tuning. System 2 Attention regenerates the context to strip irrelevant material before the model attends to it Does transformer attention architecture inherently favor repeated content?, while Titans bolts on a separate neural memory that decides what's worth keeping based on surprise rather than position — explicitly trying to escape the quadratic, position-bound limits of attention Can neural memory modules scale language models beyond attention limits?. Both treat the bias as something you route around, not something you can train away — which is the real takeaway: the tilt is in the mechanism, so the fix has to be too.
Sources 7 notes
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Transformers integrate token information through weighted parallel aggregation rather than selective suppression of irrelevant words. This structural difference explains consistent failures with jokes, wordplay, and frame-dependent meaning—not knowledge gaps, but missing cognitive operations.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
LLMs show identical content-sensitivity patterns to humans on NLI, syllogisms, and Wason tasks, with belief-bias signatures matching human error rates item-by-item. This behavioral isomorphism across three independent tasks suggests content and logical form are inseparable in transformer reasoning architecturally.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.