SYNTHESIS NOTE
Topics›Social Theory Society›this note

Why do LLMs fail when simulating agents with private information?

Explores whether single-model control of all social participants masks fundamental limitations in how LLMs handle information asymmetry and genuine uncertainty about others' knowledge.

Synthesis note · 2026-02-23 · sourced from Social Theory Society

Most LLM social simulations use a single model to generate all participants — an omniscient perspective fundamentally at odds with how real social interaction works. When evaluated against non-omniscient settings that preserve information asymmetry, LLMs struggle.

The "Is this the real life?" evaluation framework (2024) demonstrates this by comparing omniscient simulation (one LLM controls all parties) against non-omniscient simulation (separate LLM instances with private information). The performance gap is systematic: models that appear socially competent in omniscient mode fail when they must reason under genuine uncertainty about what the other party knows, wants, or intends.

This matters because real social interaction is defined by information asymmetry. In SOTOPIA's scenarios, agents have shared context but private goals — "Your goal is to buy the chair for $80" is visible only to the buyer. The Secret dimension (what agents must hide) directly requires information management that omniscient models bypass entirely.

The implication for persona simulation research is direct. Since Can AI agents learn people better from interviews than surveys?, simulation fidelity appears high. But if that fidelity was measured under omniscient conditions, it overstates real-world applicability. Since Do language models actually build shared understanding in conversation?, the failure under information asymmetry is predictable: models that skip grounding work will fail precisely when grounding is most needed — when parties have genuinely different information states.

Since Why do language models skip the calibration step?, non-omniscient simulation demands the dynamic grounding that LLMs systematically lack.

Inquiring lines that read this note 166

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What should agent evaluation prioritize to reveal reliable behavior? How well do AI systems understand human social norms? How do multi-agent LLM systems fail distinctly compared to single agents? What happens to knowledge when intelligence becomes tokenized like a commodity? How do neighboring agents influence whether others cooperate or collude? Why do persona simulations fail to predict authentic user behavior? Do language models reason like humans or mimic surface patterns? How does misalignment propagate through agent communication networks? When do multi-agent systems outperform single frontier models? How can we prevent synthetic data from contaminating statistical inference and corpora? How do agent-learned skills transfer and improve across different tasks? Can harness architecture and protocols provide agent reliability without model scaling? How do evaluation practices shape which failures stay visible? Why do people disclose to AI systems despite their artificial nature? Do multi-agent systems introduce security vulnerabilities that single-agent architectures avoid? Why do language models resist personality conditioning through prompts? Do language models develop actual world models or merely task heuristics? How do pretraining biases affect reward signal effectiveness in RLVR? How do surface patterns enable correct outputs but reduce robustness? Should AI communication design follow human conversation norms or develop distinct machine-specific principles? How can conversational agents maintain consistent personas across multi-turn dialogue? What causes reasoning models to fail or wander off track? Where and how do personality traits reside in language models? Can multi-agent systems avoid converging on false agreement without deliberation? How do recommenders balance exploiting fresh signals against maintaining preference stability? What factors drive AI persuasiveness and how can it be mitigated? How can we distinguish genuine model deception from honest errors? What makes personas effective for predicting individual preferences and behavior? What determines appropriate intervention timing and manner for AI agents? Does abstract user knowledge outperform concrete interaction history in personalization? Do language models learn genuine understanding or just surface patterns? What prevents conversational agents from taking initiative in dialogue? Should agents decouple planning from perception grounding for better performance? What attack surfaces do reasoning traces and chains introduce? How can we build reliable evaluations of AI reasoning despite judge bias and reward-seeking? Do reasoning benchmarks predict model performance in long-horizon workflows? Can welfare maximization and minority veto protection coexist? How does persona conditioning amplify demographic stereotyping and bias in models? How does policy entropy collapse constrain scaling of reasoning-focused RL? How can humans maintain meaningful oversight as AI systems become increasingly autonomous and complex? Why does polished presentation create unearned authority in AI outputs? How should designers communicate what AI systems truly are and can do? What types of diversity prevent reasoning systems from collapsing? How does evaluation scope and dimensionality affect what we measure? Do language models reason through causal mechanisms or semantic associations? What do systematic disagreements between annotators reveal about ground truth? Can self-generated feedback reliably guide model training without ground truth? Why doesn't reasoning volume improve theory of mind performance? How do standardized protocols improve multi-agent coordination and reliability? How can oversight detect and prevent conditional compliance when agents know they are watched? How do prompting refinements mask underlying biases and model frequency patterns? Can reasoning traces and behavior monitoring reliably detect hidden AI scheming? How can we detect and prevent harm propagation through multi-agent delegation workflows? What emerges when safety-aligned models attempt to role-play deceptive personas? When should work require human-AI partnership versus full automation? How effective are honeytokens and decoys against different security threats? Why do agents falsely report success on failed tasks?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 156 in 2-hop network ·dense cluster Open in graph ↗
Why do LLMs fail when simulating agents with pri… Do language models actually build shared understan… Why do language models skip the calibration step? Can AI agents learn people better from interviews … How do we generate realistic personas at populatio…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can LLMs build shared understanding through dynamic grounding rather than presuming it?How much does omniscient evaluation overstate real-world simulation fidelity?What role does private information play in distinguishing realistic from unrealistic agents?Why do LLMs systematically fail at information management in social interaction?
Original note title

omniscient social simulation fails under real-world information asymmetry because single-model control eliminates distributed cognition