How do structured prompts force LLMs to check for contradictions in evidence?
This explores whether forcing an LLM to follow an explicit reasoning structure — name your warrants, enumerate your assumptions — actually makes it catch evidence that doesn't hold together, and the corpus suggests the real problem isn't missing knowledge but a default willingness to skip the check.
This explores whether structured prompts make LLMs catch conflicting or false evidence rather than glossing over it — and the corpus's most useful move is to first explain *why* models don't check on their own. Left to standard chain-of-thought, a model will happily skip the implicit step where a claim is supposed to be justified. The fix in Can structured argument prompts make LLM reasoning more rigorous? is to borrow Toulmin's model of argument and turn it into mandatory prompt steps: before answering, the model must surface the warrant (the unstated rule connecting evidence to conclusion) and its backing. By making that step a required slot to fill rather than an optional flourish, the structure catches reasoning failures that ordinary step-by-step prompting waves through.
The deeper reason this works shows up in Do language models fail at identifying unstated preconditions?: models usually *have* the relevant world knowledge but fail to bring background conditions forward as active constraints. When the prompt forces explicit enumeration of preconditions, accuracy jumps from 30% to 85%. That's the whole mechanism in miniature — the contradiction is detectable, but only once the model is compelled to lay the pieces on the table where they can clash. Structure doesn't teach the model anything new; it changes what the model is obligated to make visible.
Why the obligation matters becomes stark in the false-assumption work. Why do language models accept false assumptions they know are wrong? shows models accommodating false premises baked into a question even when a direct factual query proves they know better — a false presupposition pulls harder toward acceptance than correct knowledge pulls toward rejection. Why do language models struggle with questions containing false assumptions? quantifies the cost: performance roughly halves when a question smuggles in a bad assumption, and scaling doesn't close the gap. So the contradiction the reader cares about often isn't between two pieces of evidence the model retrieves — it's between a plausible-sounding premise and what the model already knows. A structured prompt is essentially a forced pause that asks: *is the thing I'm being handed even true?*
There's a limit worth knowing, though. Why do embedding contexts confuse LLM entailment predictions? finds that some contradictions hide in grammar itself — presupposition triggers and non-factive verbs ("pretended that," "realized that") flip a sentence's logical commitments, and models read them as surface cues instead of computing the opposite meaning. This failure persists across prompts, which means a structured prompt can force a check but can't guarantee the model performs the *right* semantic operation when the conflict is buried below the words. Structure exposes; it doesn't always interpret.
The quiet warning underneath all of this comes from Does iterative prompt engineering undermine scientific validity?: if you keep hand-tuning a prompt until it produces the answer you wanted, you've stopped testing for contradictions and started manufacturing agreement. The thing that makes critical-question and enumeration prompts trustworthy is exactly that the steps are pre-specified rather than reverse-engineered to flatter the model — the discipline is in fixing the checklist *before* you see whether you like the output.
Sources 6 notes
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.
LLMs struggle not from lacking world knowledge but from failing to bring background conditions forward as relevant constraints. Prompting that forces explicit enumeration of preconditions raises accuracy from 30% to 85%, revealing the frame problem persists in statistical systems.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
The (QA)2 benchmark found that zero-shot LLMs halve their performance when questions contain false or unverifiable assumptions compared to valid questions. Even top models reached only 56% acceptability, and the gap persists despite model scaling, suggesting false presuppositions embedded in plausible language are systematically difficult to reject.
LLMs treat presupposition triggers and non-factive verbs as surface cues rather than computing their opposite semantic effects on entailments. This structural failure persists across prompts and models, suggesting models rely on surface patterns instead of structural analysis.
Iterative prompt revision by single researchers introduces individual bias, shifts evaluation criteria to match LLM capabilities rather than task requirements, and creates self-fulfilling feedback loops. A validated pipeline with inter-coder reliability and pre-specified criteria is required instead.