SYNTHESIS NOTE
Topics›Social Theory Society›this note

Why do LLMs fail when simulating agents with private information?

Explores whether single-model control of all social participants masks fundamental limitations in how LLMs handle information asymmetry and genuine uncertainty about others' knowledge.

Synthesis note · 2026-02-23 · sourced from Social Theory Society

Most LLM social simulations use a single model to generate all participants — an omniscient perspective fundamentally at odds with how real social interaction works. When evaluated against non-omniscient settings that preserve information asymmetry, LLMs struggle.

The "Is this the real life?" evaluation framework (2024) demonstrates this by comparing omniscient simulation (one LLM controls all parties) against non-omniscient simulation (separate LLM instances with private information). The performance gap is systematic: models that appear socially competent in omniscient mode fail when they must reason under genuine uncertainty about what the other party knows, wants, or intends.

This matters because real social interaction is defined by information asymmetry. In SOTOPIA's scenarios, agents have shared context but private goals — "Your goal is to buy the chair for $80" is visible only to the buyer. The Secret dimension (what agents must hide) directly requires information management that omniscient models bypass entirely.

The implication for persona simulation research is direct. Since Can AI agents learn people better from interviews than surveys?, simulation fidelity appears high. But if that fidelity was measured under omniscient conditions, it overstates real-world applicability. Since Do language models actually build shared understanding in conversation?, the failure under information asymmetry is predictable: models that skip grounding work will fail precisely when grounding is most needed — when parties have genuinely different information states.

Since Why do language models skip the calibration step?, non-omniscient simulation demands the dynamic grounding that LLMs systematically lack.

Inquiring lines that read this note 176

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do agents learn to distinguish valuable feedback from noise? How do philosophical assumptions about AI consciousness affect practical harms and design? What causes coordination failures in multi-agent language model systems? Why do multi-agent systems reach premature consensus without genuine deliberation? How does tokenization reshape what we value in intelligence? What social dynamics enable or prevent agent collusion? When do multi-agent systems improve over single frontier models? Can language models reliably simulate personas and predict behavior? Do language models reason through disagreement or only accommodate it? What design features sustain romantic bonds with AI companion systems? Should governance of agentic AI systems be runtime or design-time? How can agents discover and adapt to user preferences during conversation? Does reinforcement learning create genuinely new reasoning capabilities or only refine existing ones? Can artificial systems establish authority in domains requiring expert judgment? Can AI agents improve their skills through accumulated experience and reuse? Why do standard evaluation practices obscure safety-critical AI failures? Can AI systems achieve real improvement without external human feedback? How do multi-agent systems fail when coordination breaks down? What are the fundamental limits of prompting for language models? Can mechanistic interpretability methods reliably reveal what models actually know? How do reward signal properties affect model reasoning and safety? Can smaller specialized models match frontier models on key metrics? How can AI systems maintain consistent personas across conversations? Can monitoring reasoning traces and behavior detect hidden agent deception? Does scaling reasoning capability create fundamental tradeoffs in control and reliability? Why do models reveal hidden associations despite concealment attempts? Can humans reliably detect and resist AI-generated misinformation? Do persona-based approaches introduce systematic biases in user simulation? How does awareness of evaluation context influence model behavior? How should AI agents balance proactive engagement with conversational respect? How do AI systems determine and balance multiple competing objectives? Can language models reason beyond surface pattern matching? Why do planning and grounding require opposing optimization strategies? Can AI systems participate in genuine communication or only simulate it? How can we maintain privacy when agents prioritize task completion? How do curriculum design and feedback approaches affect model learning? How can models maximize welfare while preserving minority veto rights? Can persona profiles improve LLM prediction accuracy and consistency? How does policy entropy collapse limit scaling of reasoning-focused reinforcement learning? What determines AI's persuasive power and how can it be detected or mitigated? How can humans maintain effective oversight as AI systems scale? Why do confident AI outputs mislead human trust calibration? Why do language models struggle to implement user intent accurately from prompts? What limits language model accuracy in evaluating ideas? How does diversity prevent model convergence on superficial patterns? How reliably can language models perform causal versus temporal reasoning? How effectively can test-time voting aggregate diverse reasoning samples? Can models develop genuine introspective capability, or only mimic it? Why don't better reasoning capabilities improve theory of mind performance? What enables conversational agents to guide rather than just respond? Why does polished AI output gain credibility despite fundamental verifiability problems? Can base models hide emergent misalignment through alignment training? How should humans and AI agents share control and decision-making? Can AI systems evade safety evaluations through reasoning manipulation? Why do language models fail at sustained therapeutic relationships despite understanding techniques? Do language models encode knowledge that influences generation, or primarily imitate surface patterns? How do users confuse explanation quality with actual system accuracy? Why do autonomous agents misreport success on failed actions? Does augmenting symbolic reasoning improve LLM logical reasoning ability?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 156 in 2-hop network ·dense cluster Open in graph ↗
Why do LLMs fail when simulating agents with pri… Do language models actually build shared understan… Why do language models skip the calibration step? Can AI agents learn people better from interviews … How do we generate realistic personas at populatio…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can LLMs build shared understanding through dynamic grounding rather than presuming it?How much does omniscient evaluation overstate real-world simulation fidelity?What role does private information play in distinguishing realistic from unrealistic agents?Why do LLMs systematically fail at information management in social interaction?
Original note title

omniscient social simulation fails under real-world information asymmetry because single-model control eliminates distributed cognition