Can language models actually analyze language structure?

A previously clear distinction in linguistics has become blurred by LLM capability advances.

Behavioral language tasks test language performance: is this sentence grammatical? Does it complete naturally? Can the model perform agreement, movement, or embedding correctly? These test the ability to use language.

Metalinguistic tasks test language analysis: generate the syntactic tree for this sentence, state the phonological rule this data illustrates, construct a formal analysis of this morphological paradigm. These test the ability to analyze language itself — the work that linguists do. Metalinguistic ability is cognitively more complex than language use, acquired later, and presupposes linguistic competence.

Large Linguistic Models (Yedetore et al. 2023): for the first time, LLMs can generate valid metalinguistic analyses. OpenAI's o1 vastly outperforms other models on syntactic tree construction and phonological generalization tasks. The hypothesis: o1's chain-of-thought mechanism mimics the structure of human reasoning used in complex cognitive tasks — like linguistic analysis, which requires explicit step-by-step reasoning about grammatical structure.

The implication for capability evaluation: behavioral benchmarks (grammaticality judgments, sentence completion) substantially underestimate LLM linguistic capability. Metalinguistic performance — which requires explicit reasoning about language — reveals capabilities that standard tests miss.

This also extends what we know about CoT more broadly: Why do correct reasoning traces contain fewer tokens?, but metalinguistic tasks may require the explicit structural decomposition that CoT provides, making o1's advantage domain-specific rather than general.

The practical upshot: LLMs can be used as linguistic analysis tools, not just language generators. This changes the scope of what tasks they are appropriate for.

An additional metalinguistic capability: LLMs can perform analogical reasoning from literary texts — extracting metaphoric mappings and structural analogies that require reading beyond surface content to underlying conceptual structure. The NLI literature includes work showing LLMs can identify source-target domain mappings in metaphor, classify analogical relations, and generate paraphrases that preserve analogical structure while changing surface form. These are forms of metalinguistic analysis that go beyond syntactic tree construction to semantic structure analysis. The boundary between "using language" and "analyzing language" is further blurred than previously recognized.

Literary text applications: The metalinguistic capability extends to literary analysis in specific ways. LLMs show competitive results extracting explicit source-target domain mappings from proportional analogies in poetry and prose — for example, identifying that "jar" maps to "memory" in "Memory, a jar of flies" (Automatic Extraction of Metaphoric Analogies from Literary Texts). However, they struggle with implicit elements that human readers infer — the unstated target concept that completes the analogy. This maps directly to the behavioral/metalinguistic distinction: extracting explicit mappings is metalinguistic analysis (decomposing structure); inferring implicit elements is pragmatic reasoning (reconstructing communicative intent). CoT appears to enable the former but not the latter, suggesting the metalinguistic advantage is specific to explicit structural decomposition.

Source: Linguistics, NLP, NLU; enriched from inbox/research-brief-llm-literary-analysis-2026-03-02.md