What makes strategic question-asking succeed or fail?

The 20 Questions game provides a controlled evaluation of a capability that matters far beyond games: deducing an unknown entity through strategic questioning under a budget constraint. A model must infer what it doesn't know by asking questions that elicit yes/no/maybe responses, using as few queries as possible. This requires complex understanding, state tracking, reasoning, and planning over multiple conversational turns.

Three capabilities must work in synergy:

1. State tracking and understanding: the model must comprehend multi-turn context, track what has been asked, understand its current position in the search space, and handle coreference resolution across turns.

2. Strategic planning: questions must efficiently partition the remaining possibility space. Redundant queries waste budget. Inconsistent queries (contradicting prior knowledge) waste budget. The model needs a strategy that narrows possibilities maximally with each question.

3. Inductive reasoning: from accumulated yes/no evidence, the model must construct a working taxonomy of possibilities and generate conjectures. This is hypothesis formation under incomplete evidence — building a mental model of what the entity could be given what's known so far.

GPT-4 outperforms human players by a large margin. Weaker models (GPT-3.5, LLaMA variants) show significant degradation. Behavior cloning from strong to weak models enables some generalization — suggesting the strategic questioning skill is partly transferable through demonstration.

The synergy requirement is the key finding. Each capability alone is insufficient. A model with excellent state tracking but poor planning asks relevant but inefficient questions. A model with strong planning but weak state tracking asks strategically designed questions that ignore what was already established. A model with good inductive reasoning but neither tracking nor planning generates good hypotheses but can't efficiently gather the evidence to confirm them.

This maps directly to the proactive questioning gap. Since Can models learn to ask clarifying questions instead of guessing?, the 20Q framework reveals that identifying missing information is necessary but not sufficient. The model must also (a) track what it already knows from prior turns, (b) plan which question will most efficiently resolve remaining uncertainty, and (c) reason inductively from partial evidence to narrow the search space.

The connection to information-seeking is broader than games. Medical diagnosis, technical troubleshooting, user intent clarification — all require the same three-capability synergy: track what's established, plan what to ask next, reason from partial evidence about what's possible. Since Can models learn to ask genuinely useful clarifying questions?, ALFA provides the training methodology for the strategic planning component -- question quality is not unitary, and attribute-specific optimization (clarity, relevance, specificity) directly shapes whether strategic questions efficiently partition the search space or waste budget on vague prompts.

Source: Question Answer Search