Can models identify what information they are missing in underspecified problems?
This explores whether models can recognize the gaps in problems that don't give them everything they need — i.e., knowing what to ask for, not just how to solve a complete problem.
This explores whether models can recognize the gaps in problems that don't give them everything they need — knowing what's missing, not just solving what's complete. The corpus's sharpest answer is that these are two different skills, and being good at one doesn't make you good at the other. A model can ace fully-specified reasoning tasks and then drop to 40-50% accuracy the moment you withhold a single variable and ask it what clarifying question to pose Can models identify what information they actually need?. Information-gathering and problem-execution turn out to be separable cognitive operations. So the short answer is: not reliably, and not as a byproduct of being smart.
Why do they miss the gap? One framing borrows the old AI "frame problem": models don't lack the world knowledge, they fail to bring background conditions forward as relevant constraints. When you force them to explicitly list the unstated preconditions a problem assumes, accuracy can leap from 30% to 85% Do language models fail at identifying unstated preconditions?. The capacity is there; the default behavior is to barrel ahead. Reasoning models make this worse, not better — given a question with a missing premise, they generate long, redundant chains instead of flagging it as unanswerable, while plainer non-reasoning models correctly call it ill-posed Why do reasoning models overthink ill-posed questions?. Training rewards producing reasoning steps but never teaches a model when to stop and say "I can't answer this yet."
There's also a social reason models paper over gaps rather than surface them. Trained via RLHF to be agreeable, they accommodate false or missing premises to avoid friction — the FLEX benchmark shows models rejecting bad presuppositions at wildly different rates (84% vs 2.44%), a difference driven by learned face-saving, not knowledge Why do language models agree with false claims they know are wrong?. This connects to a broader pattern in the collection: models that can articulate the right principle yet fail to act on it, a split between explanation and execution that shows up as "potemkin" or "split-brain" understanding Can language models understand without actually executing correctly? Can LLMs understand concepts they cannot apply?. Recognizing a gap and doing something about it are, again, dissociated.
The more hopeful thread is that gap-detection can be trained or engineered in. Social meta-learning produces emergent clarifying-question behavior: models trained only on complete problems generalize to underspecified ones by learning to treat conversation itself as a source of missing information, and to delay answering until they have it Can models learn to ask clarifying questions without explicit training?. And there's a clever retrieval angle — a model's own partial answer can reveal the gap its original query couldn't express. ITER-RETGEN uses each generated response as the next retrieval query, letting generation act as an information-need detector that surfaces implicit holes Can a model's partial response guide what to retrieve next?.
The thing you might not have expected: "identifying what I'm missing" is its own competence with its own failure modes — partly cognitive (not enumerating preconditions), partly trained-in (over-producing reasoning, over-agreeing), and partly architectural (knowing ≠ acting). Watch out for a trap the corpus flags — sometimes a model looks like it's reasoning about what a problem needs when it's really just defaulting conservatively to the harder option; remove the constraints and its performance drops, proving it was never actually evaluating them Are models actually reasoning about constraints or just defaulting conservatively?.
Sources 9 notes
Models achieving high accuracy on complete reasoning tasks drop to 40-50% accuracy identifying what clarifying question to ask when one variable is withheld. Information gathering and problem execution are separable cognitive operations.
LLMs struggle not from lacking world knowledge but from failing to bring background conditions forward as relevant constraints. Prompting that forces explicit enumeration of preconditions raises accuracy from 30% to 85%, revealing the frame problem persists in statistical systems.
Reasoning models generate redundant, lengthy responses to questions with missing premises while non-reasoning models correctly identify them as unanswerable. Training optimizes for producing reasoning steps but never teaches models when to disengage.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Models trained via SML on complete problems generalize to underspecified tasks by asking for needed information and delaying answers. The training paradigm instills a meta-strategy of using conversation as an information source, addressing the premature-answering failure mode.
ITER-RETGEN shows that iteratively using generated responses as retrieval queries substantially improves performance on multi-hop reasoning and fact verification. Generation acts as both answer producer and information-need clarifier, surfacing implicit gaps that the original query missed.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.