Why do LLMs fail at simple deductive reasoning?

The "Minds vs. Machines" entailment benchmark reveals a non-obvious asymmetry: LLMs outperform humans on multi-hop reasoning tasks that require integrating information across multiple sentences and knowledge types, while humans outperform LLMs on tasks requiring simple deductive inference.

This reverses the common intuition. We expect LLMs to handle simple cases well and fail on complex ones. Instead: the more complex multi-hop reasoning, the more advantaged LLMs become relative to humans. Simple deductive steps — the kind of inference humans find trivially obvious — are precisely where LLMs are weakest.

The knowledge type taxonomy matters: entity-grounded knowledge (facts about entities, verifiable externally), commonsense knowledge (implicit everyday reasoning, hard to articulate), and localized knowledge (context-specific, impossible to infer unless stated). LLMs handle entity-grounded reasoning better; humans handle commonsense inferences better.

This connects to the inversion captured in Does LLM grammatical performance decline with structural complexity? — but the failure mode is different. Grammatical complexity degrades LLM performance. Inferential complexity does not necessarily degrade it, and may improve it relative to humans who tire or miss multi-step chains. Structural complexity and inferential complexity have different profiles.

The practical implication: the right use case for LLM-assisted reasoning is complex multi-step inference that humans find cognitively taxing, not simple first-order deductions that humans find trivial. And Why do embedding contexts confuse LLM entailment predictions? shows the specific class of simple inference where LLMs fail worst: trivial entailments that humans find effortless.

Source: Natural Language Inference