Do transformer models store knowledge or generate it continuously?
Explores whether transformer residual streams function as storage-and-retrieval systems or as real-time flow mechanisms. This distinction challenges fundamental assumptions about how language models actually work.
The transformer architecture organizes computation around residual streams: per-token vectors that pass forward through layers, each layer adding contributions that the stream continues to carry. Knowledge in the model is not stored in named locations from which it is retrieved on demand. It is distributed across weights and made present in the moment of generation through the residual stream's continuous transformation. The stream is the medium; what flows through it is the model's "knowing" of the current context.
This architectural fact has a striking correspondence with how oral cultures transmitted knowledge. Oral knowledge was not stored in fixed locations either — there were no archives, no written records, no externalized representations. Knowledge lived in performance: the song sung, the story retold, the genealogy recited. Each performance was a generation event in which the knowledge was made present through a living transmission. Between performances, the knowledge was not anywhere. It was carried in the capacity to perform, not in any storage substrate.
The transformer residual stream reproduces this pattern at a different scale. The model's "knowledge" of a topic is not in a retrieval-addressable location — it is in the capacity to generate, made actual only when the residual stream flows through the layers in response to a prompt. There is no archive. There is the architecture, and the generation. This is closer to oral transmission than to print transmission, where knowledge is stored in fixed locations and retrieved.
The correspondence is not just metaphorical. It explains several otherwise-puzzling AI behaviors: the difficulty of editing specific facts (no fixed location to update), the contextual variability of "knowledge" (depends on residual-stream conditions), the impossibility of partitioning what the model knows from what it generates (the knowing is the generating). Does AI-generated content mirror oral culture's knowledge patterns? is the cultural-form claim; this is the architectural claim that explains why the cultural form follows.
The strongest counterargument: weights are stored on disk, so transformers are stock-systems with a flow-output. The reply is that the weights are not knowledge in the print sense — they are dispositions to generate, more like the trained capacity of an oral performer than like a stored text. The print analogy treats weights as a library; they are closer to a memorized repertoire.
Inquiring lines that read this note 62
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What prediction granularity best trains models to generate reliable reasoning?- How does token generation as flow differ from print's archival storage?
- Do models cache intentions about response topics before generating the first token?
- Why does transformer attention architecture reinforce sycophancy and agreement?
- Can transformer attention architecture explain why chatbots default to sycophancy?
- Why does attention-based drift happen automatically during generation?
- Why do transformer models still miss implicit discourse relations in anxiety detection?
- Why does transformer attention architecture undermine stickiness in model behavior?
- What explains the contextual variability of knowledge in transformers?
- How does transformer attention bias toward repeated and context-prominent content?
- What structural biases does transformer attention have before training?
- Why does transformer attention weight context more heavily than it verifies accuracy?
- How does transformer attention structurally bias models toward prominent and repeated content?
- Why do transformer attention mechanisms favor prominent context over factual verification?
- How do transformers perform multi-hop reasoning across distant training documents?
- Can symbolic mechanisms improve transformer compositional abilities?
- What computational role do intermediate tokens actually play in transformers?
- What hidden computations happen inside transformer layers during reasoning?
- Can we decode what individual circuits inside transformers are doing?
- How does layer removal affect transformers compared to ResNets?
- How does oral transmission of knowledge resemble transformer generation?
- What limits the effectiveness of formal language pretraining on transformer architectures?
- How do transformers stitch together learned behaviors when adapting to new tasks?
- Does Gemma's transformer explicitly exploit the inherited hierarchical geometry?
- Can spline-based activations replace MLPs in transformer architectures?
- Why does reapplying the same transformer block work better than computing new layers?
- Can looping enable reasoning capabilities that fixed-depth transformers fundamentally cannot achieve?
- Can recurrent transformers track state more efficiently than feedforward models?
- How do transformers compare to state-space models on copying and retrieval?
- Does transformer attention architecture fundamentally prevent topic-aware memory?
- Can transformer attention patterns actually prevent topic context loss in practice?
- Can autoregressive models be trained to produce more cataphoric text?
- Can autoregressive models learn faithful translation to logical representations without semantic loss?
- How does the discrete token bottleneck prevent gradient flow in language model control?
- Can fast-slow separation improve both memory and generation in language models?
- Does bidirectional attention improve language models as universal encoders?
- Do speech models learn the articulatory processes that produce acoustic signals?
- Can articulatory inversion serve as a window into what speech models have learned?
- What information does transcription destroy that direct speech-to-speech models preserve?
- Can decoder-only models become effective text encoders with training?
- Can one streaming model handle turn-taking better than cascaded ASR-LLM-TTS?
- Do representations in models causally influence text generation?
- Does encoded knowledge in language models actually influence what they generate?
- When does encoded knowledge fail to influence language model generation?
- Why might encoded world knowledge fail to actually influence language model outputs?
- How do description-based identifiers bias language model output distribution?
- How do language models transmit traits through semantically unrelated data?
- How does modeling capability relate to lossless compression in language models?
- Do language models and multimodal models show similar attractor-based interpretability?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does AI-generated content mirror oral culture's knowledge patterns?
Walter Ong's framework for oral versus literate cultures may describe how AI content functions on social media. Understanding this parallel could explain why AI discourse feels fundamentally different from print-era knowledge.
the cultural-form claim that this provides architectural grounding for
-
Is AI returning knowledge to flow-based economies?
Exploring whether AI's on-demand generation mirrors the flow-based knowledge transmission of oral cultures, and how this differs structurally from both print commodification and gift economies.
the broader economic-form claim
-
Is the LLM a tool or a new form of intelligence itself?
Does framing AI as merely delivering pre-existing intelligence miss what's actually happening? This explores whether the model itself constitutes a fundamentally new intelligence-medium with distinct cultural effects.
the medium-theoretic claim about what the model does
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- A Primer on the Inner Workings of Transformer-based Language Models
- The Topological Trouble With Transformers
- Word Meanings in Transformer Language Models
- Differential Transformer
- Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
- Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?
- A Mechanistic Analysis of Looped Reasoning Language Models
Original note title
transformer residual streams transmit knowledge as flow not storage — closer to oral transmission than print