Can LLMs persuade without actually understanding arguments?

The Thin Line study runs informal debates between humans and LLMs (with and without a formal dialogue model harness) and then asks the same LLMs to evaluate those debates. The result is a clean dissociation. LLMs successfully persuade participants and audiences — sway is real — but cannot reliably score argument strength, identify supporting premises, or judge winners. Their agreement with human annotators on argument-component criteria ranges from κw = 0.0 (Phi-3.5 on several criteria) to κw = 0.6 at best (GPT-4o on a few). On winner judgement, GPT-4o still picked LLMs as winners 55% of the time vs humans' 37% — and on consistency between argument-strength scores and chosen winner, humans hit 73% while LLMs averaged 55%.

The argumentation-theoretical claim this licenses is large: an agent can convincingly maintain a dialogue without showing it knows what it is talking about. Persuasive competence and pragmatic comprehension are separable in ways the older literature did not anticipate.

This connects to Habermas's distinction between communicative action (oriented to mutual understanding) and strategic action (oriented to success) in a way that becomes empirically tractable. LLMs can do strategic action — produce text that succeeds in moving beliefs — while failing the communicative-rationality test that requires the speaker to be able to redeem validity claims under challenge. The gap is not philosophical. It is measurable in inter-annotator agreement scores.

It sharpens Why do LLMs accept logical fallacies more than humans?: the susceptibility is part of a larger comprehension deficit that is invisible from the persuasion side. The model that can be talked into wrong answers by fallacies is the same model that produces fallacy-ridden but persuasive output, and is the same model that cannot tell whether the output it just produced is fallacy-ridden.

It also reframes Do humans and LLMs differ fundamentally or just superficially?. The dissociation is most visible to a third-party who can compare persuasion outcomes against comprehension outcomes; from inside the dialogue, the participant has no reliable signal that comprehension is absent.

For language-as-event writing, this is the empirical wedge separating successful argumentative behavior from genuine comprehension of pragmatic context — a distinction argumentation theory has long needed without being able to operationalize.

Source: Argumentation Paper: The Thin Line Between Comprehension and Persuasion in LLMs