Can language models acquire meaning from distributional patterns alone without joint attention?
This explores the classic debate over whether predicting word-from-word in text is enough to learn what words mean, or whether meaning requires shared communicative grounding — Bender & Koller's 'joint attention' — that pure form-prediction can never supply.
This question lands on a genuine fault line in the corpus, and the collection holds both ends of it. The strongest 'no' comes from the argument that meaning is the relation between expressions and the communicative intents behind them — and since models trained only on form see form-to-form prediction with no shared attention to a world, they can't reconstruct that relation Can language models learn meaning from text patterns alone?. On this view, distributional patterns give you fluency, not grounding; the octopus that only ever read transcripts never learns what the words point at.
But the corpus also holds a sharp counter-framing: maybe distributional structure isn't a poor substitute for meaning — maybe it's a *different kind* of meaning entirely. One note reads LLMs through Saussure's idea of langue, where meaning lives in the relations between signs rather than in any link to external referents. By that lens, models compress the relational structure of language and culturally situated discourse straight from text, and need no embodiment or world-pointing to generate competently Can language models learn meaning without engaging the world?. The disagreement isn't really empirical — it's about whether 'meaning' is defined by grounding in intent or by position in a relational system.
What tips the balance, and what you might not expect, is how much the behavioral evidence sides with the skeptics — not by appealing to philosophy, but by catching models tracking statistics where meaning should be. Models systematically prefer higher-frequency surface forms over semantically identical rare paraphrases, across math, translation, and reasoning — suggesting they ride statistical mass rather than recognize meaning Do language models really understand meaning or just surface frequency?. Strip the familiar semantics out of a reasoning task and performance collapses even when the correct rules sit right in the prompt, because the model leans on token associations rather than formal logic Do large language models reason symbolically or semantically?. And syntactic competence frays predictably as structural depth increases, revealing surface pattern-matching where deep grammatical rules should be Why do large language models fail at complex linguistic tasks?.
There's an even more reductive frame that dissolves the meaning question altogether: treat the model as an autoregressive probability machine, and you can predict its failures from output probability alone — low-probability targets are hard regardless of logical simplicity Can we predict where language models will fail?. If 'where it fails' is predictable from distributional likelihood without ever invoking meaning, that's evidence the distribution is doing the work. The same story shows up in context-integration failures, where strong training-time associations override what's actually written in the prompt — distributional priors winning out over the situation at hand Why do language models ignore information in their context?.
So the corpus's answer is layered: models clearly acquire *something* powerful from distribution alone — enough to generate culturally fluent text — but the behavioral evidence keeps catching that 'something' behaving like statistics rather than grounded meaning. Whether you call the relational structure they capture 'meaning' depends on which definition you started with. The interesting takeaway isn't a verdict; it's that the same systems can look like they understand and like they're pattern-matching at once, and which one you see depends on whether you test fluency or test grounding.
Sources 7 notes
Bender & Koller argue that meaning requires the relation between expressions and communicative intents. Since LLMs are trained only on form-to-form prediction with no access to shared attention or intent, they cannot reconstruct the meaning that grounds language.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.