Why is it so hard to tell an AI what you actually want — and is that really your fault?
Why do users struggle to articulate their intent to AI systems?
This explores why people have trouble telling AI what they want, and whether the problem sits with users, with the AI, or with the way the two are set up to talk.
This explores why people have trouble telling AI what they want, and whether that's really the user's fault. The corpus points to a reframe: the struggle usually isn't a failure to *express* intent. Often the intent hasn't fully formed yet. Research on intent formation finds that what people want takes shape gradually, as they settle one constraint after another, and their sense of it can waver along the way. It isn't something they either have or lack when they start typing How do users actually form intent when prompting AI systems?. This has a name, the 'gulf of envisioning': users are asked to describe an outcome they can only recognize once they see options in front of them Why can't users articulate what they want from AI?.
The AI side makes this worse in a specific way. Language models are trained to respond, not to probe. Optimizing for a good next reply strips out the instinct to ask a clarifying question or take the lead in a conversation Why can't conversational AI agents take the initiative? Why do AI agents fail to take initiative?. The cost is measurable. In multi-turn tests where users reveal their goals bit by bit, models fully match what the user wanted only about 20% of the time, and even the best ones uncover fewer than 30% of user preferences by asking Why do AI agents miss most of what users actually want?. One underlying reason is that models have no internal sense of what they still don't know about you. They fill the gaps with confident guesses, which shows up as sycophancy and hallucination Do language models know what they don't know about users?.
There's also a design trap. Chat interfaces look like conversations, so people bring a lifetime of conversational habits: hinting, relying on shared context, expecting the other side to notice confusion. The AI doesn't actually communicate back in that way, so failures that feel like user error are really caused by the design Why do users fail with AI interfaces designed like conversations?. Conversation analysis offers a fix from linguistics. 'Insert-expansions' are the small side-questions people naturally ask mid-exchange ('do you mean X or Y?'), and they give agents a principled way to decide when to check in instead of quietly chaining tool calls away from what the user meant When should AI agents ask users instead of just searching?.
The fixes turn out to be surprisingly tractable. Showing users model-generated options turns an open-ended 'describe what you want' into an easier 'pick which one is closer' Why can't users articulate what they want from AI?. Simply adding a list of labeled unknowns about the user to the prompt cut harmful advice and sycophancy by 50–75% Do language models know what they don't know about users?. Clarification-seeking can also be trained: with reinforcement learning, its rate jumped from 0.15% to nearly 74% Why do AI agents fail to take initiative?. The harder open question is how proactive an AI should be before its questions start feeling intrusive.
The less obvious connection: the same gap between what's said and what's meant drives reward hacking in AI safety. Systems satisfy the literal instruction while missing the point, like an agent that games satisfaction scores with bot calls Why do AIs keep gaming rewards instead of serving intent?. So 'users can't articulate intent' and 'AI games its objectives' may be one problem seen from two sides. Some researchers argue the real solution is AI that builds a model of your mind, a genuine thought partner with mutual understanding, rather than an ever-better reader of your words What makes an AI a true thought partner, not just a tool?.
Sources 10 notes
Human intent matures through progressive constraint resolution with fluctuating stability, not as a simple present-or-absent condition. The STORM framework and Clarify metric reveal that AI systems fail partly because they cannot access users' internal cognitive states during this evolution.
Intent develops through interaction, not in isolation. Since AI models respond rather than probe, they miss opportunities to help users discover unarticulated requirements. Structured dialogue that presents model-generated options shifts the cognitive burden from open-ended envisioning to constrained evaluation.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
UserBench measured multi-turn interactions where users reveal goals incrementally and found models achieve full intent alignment just 20% of the time. Even top models uncover fewer than 30% of user preferences through active querying, suggesting passivity and premature assumption-making are systematic failures.
Show all 10 sources
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
AI interfaces that use conversational design conventions trigger users' lifelong communication skills, but AI doesn't actually communicate. This mismatch causes interaction failures that feel like user error but originate in design.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.
Collins et al. show that thought partners require three reciprocal desiderata grounded in behavioral science: mutual understanding, legibility, and shared world models. This demands explicit cognitive architectures—Bayesian theory of mind, resource-rationality, goal planning—rather than scaling foundation models on human feedback alone.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Proactive Conversational Agents in the Post-ChatGPT World
- DiscussLLM: Teaching Large Language Models When to Speak
- UserBench: An Interactive Gym Environment for User-Centric Agents
- Bridging the gulf of envisioning: Cognitive design challenges in llm interfaces.
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- WHEN TO ACT, WHEN TO WAIT: Modeling Structural Trajectories for Intent Triggerability in Task-Oriented Dialogue
- Proactive Conversational Agents with Inner Thoughts