Why do LLMs choose surface-order quantifier scope over contextually correct readings?
This explores why, when a sentence has more than one valid quantifier-scope reading (e.g. 'every kid climbed a tree' — one shared tree or one tree each), LLMs default to the reading that matches surface word order instead of the one the context actually calls for.
This explores why LLMs lock onto the surface-order reading of an ambiguous sentence rather than the contextually correct one. The corpus points to a single underlying cause showing up in many guises: these models are statistical surface-pattern matchers, not structural computers, and quantifier scope is exactly the kind of structural computation they don't actually perform. The most direct clue is that LLMs systematically prefer whatever phrasing carries more statistical mass from pretraining — they reliably pick higher-frequency surface forms over semantically equivalent rare ones, even when meaning should override frequency Do language models really understand meaning or just surface frequency?. Surface-order scope is just the high-frequency default reading; choosing it is the same reflex.
Scope disambiguation also demands something the corpus says LLMs can't do: hold two interpretations of the same string at once. On the AMBIENT benchmark, GPT-4 correctly disambiguates only 32% of cases versus 90% for humans, and the failure explicitly spans scope ambiguity — the models simply don't represent the alternative reading well enough to weigh it against context Can language models recognize when text is deliberately ambiguous?. If only one reading is ever really 'live,' the context never gets a chance to pull the model toward the other.
The deeper reason that reading is the surface one is that LLMs reason through semantic association, not symbolic manipulation. When meaning is decoupled from the logical form of a task, performance collapses even with the correct rules supplied in context Do large language models reason symbolically or semantically?. Quantifier scope is a symbolic operation over logical structure, so the model substitutes the nearest associative shortcut: linear word order. The same shortcut shows up in entailment, where models treat presupposition triggers and non-factive verbs as surface cues instead of computing their actual semantic effect Why do embedding contexts confuse LLM entailment predictions?, and in grammar, where competence degrades predictably as structural depth and embedding increase — evidence the models learned surface heuristics rather than real structural rules Does LLM grammatical performance decline with structural complexity? Why do large language models fail at complex linguistic tasks?.
What's worth knowing — and what you might not have expected — is that this isn't a fixed ceiling. The structural reading often exists in the model; it just doesn't get surfaced by default. Forcing models to explicitly enumerate the constraints they'd otherwise skip raises accuracy dramatically (30% → 85% on the frame-problem task) Do language models fail at identifying unstated preconditions?, structured argument prompts make models check premises they'd normally glide past Can structured argument prompts make LLM reasoning more rigorous?, and with explicit step-by-step reasoning o1 can build genuine syntactic and phonological analyses Can language models actually analyze language structure?. The pattern: the contextually correct scope reading is reachable, but only when something forces the model out of its default surface pass and into explicit structural work. Left to its own decoding, it takes the high-frequency, linear-order path every time.
Sources 9 notes
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
LLMs treat presupposition triggers and non-factive verbs as surface cues rather than computing their opposite semantic effects on entailments. This structural failure persists across prompts and models, suggesting models rely on surface patterns instead of structural analysis.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
LLMs struggle not from lacking world knowledge but from failing to bring background conditions forward as relevant constraints. Prompting that forces explicit enumeration of preconditions raises accuracy from 30% to 85%, revealing the frame problem persists in statistical systems.
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.
OpenAI's o1 model successfully constructs syntactic trees and phonological generalizations through explicit step-by-step reasoning, revealing that LLM linguistic capability extends far beyond behavioral language tasks to genuine language analysis.