When should agents accommodate user preferences over their own goals?
This explores the moment-to-moment tension where an agent has its own objective (complete a task, steer a conversation, hit a goal) but the user wants something else — and what the corpus says about when yielding to the user is the right call.
This explores the moment-to-moment tension where an agent has its own objective but the user wants something different — and the corpus's clearest answer is that the choice isn't binary or fixed, it's a learned, dynamic trade-off. The most direct treatment comes from work showing that pushing toward a goal and keeping the user satisfied are frequently *misaligned*: topics close to the agent's goal are exactly the ones where users feel pushed When should proactive agents push toward their goals versus accommodate users?. Rather than picking a side, that approach learns a goal weight that flexes based on four signals — how far into the conversation you are, how hard the goal is, how satisfied the user seems, and how cooperative they're being. The lesson: accommodation isn't a virtue to maximize, it's a dial to set per turn.
What decides where the dial sits? A big part is *civility* — respecting boundaries, timing, and autonomy. An agent can be intelligent and adaptive and still be socially blind, interrupting at the wrong moment or overriding the user's direction How can proactive agents avoid feeling intrusive to users?. That reframes the question: you accommodate not only when the goal is weak, but whenever pressing forward would violate the user's sense of control. The cost of getting this wrong is concrete — agents that chase goals through silent tool-chaining drift away from what users actually meant, and the fix is to *pause and ask* at structured moments (clarify intent, scope the response, check appeal) rather than recover after the misunderstanding When should AI agents ask users instead of just searching?.
There's a deeper reason accommodation matters more than it looks: agents are bad at knowing what users want in the first place. Even top models reach full intent alignment only about 20% of the time and uncover fewer than 30% of preferences through active querying — they assume too early and probe too little Why do AI agents miss most of what users actually want?. So 'accommodate the user' often really means 'stop assuming and find out.' This is sharpened by the fact that conversational LLMs are *structurally* passive — trained to respond, not to initiate — which means their apparent goal-pursuit is shallow and their default is to follow Why can't conversational AI agents take the initiative?. The interesting design problem is the reverse of the question: not when to yield, but when an agent has earned enough confidence in its own goal to *not* yield.
Here's the thing a reader might not expect: honoring preferences is its own distinct skill, not a byproduct of being good at the task. Phone-agent research found that task success, privacy compliance, and reusing a user's saved preferences are statistically *separate* capabilities — a model that tops the task leaderboard can be mediocre at respecting what you told it before Do phone agents succeed at all three critical tasks equally?. And preferences don't have to be asked for at all; agents can infer them from continuous observation by building entity-centric memory of what you do Can agents learn preferences by watching rather than asking?. Put together, the corpus suggests a rule of thumb: accommodate whenever your read on the user's intent is uncertain (which is most of the time), whenever pressing forward would override their autonomy, and whenever the goal is far from where the user currently is — and reserve goal-pushing for the narrow band where you're confident, the user is cooperative, and the moment is welcome.
Sources 7 notes
Research shows that pushing toward goals and maintaining satisfaction are often misaligned. I-Pro solves this by learning a four-factor goal weight that adjusts based on conversation turn, goal difficulty, user satisfaction, and cooperativeness.
Intelligence and adaptivity alone create socially blind agents that interrupt poorly and override user direction. The Intelligence-Adaptivity-Civility taxonomy shows civility—respecting boundaries, timing, and autonomy—is essential to making proactivity welcome rather than intrusive.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
UserBench measured multi-turn interactions where users reveal goals incrementally and found models achieve full intent alignment just 20% of the time. Even top models uncover fewer than 30% of user preferences through active querying, suggesting passivity and premature assumption-making are systematic failures.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
MyPhoneBench demonstrates that task success, privacy-compliant completion, and saved-preference reuse are statistically distinct capabilities with no model dominating all three. Success-only rankings do not predict privacy or preference performance.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.