Why do large language models fail at complex linguistic tasks?

LLMs demonstrate "limited efficacy" on fine-grained linguistic annotation tasks, and the failures are not random — they are systematic and they get worse as input structural complexity increases.

The specific errors documented in Llama3-70b (one of the most capable models tested):

• Misidentifying embedded clauses
• Failing to recognize verb phrases
• Confusing complex nominals with clauses

The research examined three questions: (1) accuracy on complex linguistic structure detection, (2) which structures are LLM blind spots, (3) how performance varies with linguistic complexity. The answers: accuracy is notably limited, complex syntactic structures (especially embedded/recursive ones) are the consistent blind spots, and performance degrades predictably with structural depth.

This matters because it reveals where statistical language learning diverges from grammatical competence. LLMs trained on vast corpora learn strong surface-level patterns, but the patterns do not reliably encode the deep structural rules that govern syntax. The model knows that a sentence has a verb, but cannot reliably identify the verb phrase when the structural context is complex.

The implication for LLM deployment in NLP pipelines: any application relying on fine-grained linguistic annotation — parsing, dependency analysis, argument structure detection — cannot treat LLMs as structurally reliable without auditing their performance on complex inputs. The failures are not edge cases; they are structurally determined by input complexity.

Source: Discourses