Can controllable latent variables in simulators ground them to realistic conversation?
This explores whether giving an LLM user-simulator a few tunable 'knobs' — hidden variables like who the user is and what they want — is enough to make its fake conversations read as real, and what else the corpus says realism actually depends on.
This explores whether giving an LLM user-simulator a few tunable 'knobs' — hidden variables like who the user is and what they want — is enough to make its fake conversations pass as real. The most direct answer in the corpus is yes, with caveats. RecLLM shows that conditioning a simulator on two layers of latent variables — a session-level user profile and a turn-level user intent — produces synthetic dialogue that holds up under three independent realism tests: humans can't reliably tell it from real, discriminator models struggle, and the distribution of generated conversations matches real ones Can controlled latent variables make LLM user simulators realistic?. So controllability isn't just a convenience for steering the simulator; the act of grounding it to explicit profile-and-intent variables is what buys the realism.
But other notes suggest two latent variables is the floor, not the ceiling. One line of work argues realistic synthetic dialogue is *multiplicative*: you need subtopic specificity, Big Five personality variation, and around eleven contextual characteristics all stacked together, reaching ~90% of real in-domain performance only when the layers compound Can synthetic dialogues become realistic through layered diversity?. The more interesting failure isn't too few variables — it's that simulators drift away from whatever variables you set. A persona conditioned at turn one quietly decays over a long conversation, and training the simulator with multi-turn RL against consistency rewards cuts that drift by over 55% Can training user simulators reduce persona drift in dialogue?. Grounding, in other words, is not a one-time conditioning act; it has to be defended turn by turn.
Here's the thing the question doesn't anticipate: there may be no stable 'character' underneath for the latent variables to pin down. Shanahan's 20-questions regeneration test shows an LLM holds a *superposition* of consistent characters and samples one at generation time — regenerate the same reply and you get a different-but-still-consistent answer, meaning nothing is actually committed Do large language models actually commit to a single character?. A competing view pushes back: post-training can *install* personas robust enough to resist adversarial pressure, making them substrate-level dispositions rather than performances Are LLM personas realized or merely simulated through training?. Whether your latent variables are steering a real disposition or just biasing a sampler is genuinely unsettled — and it matters, because it determines whether 'controllable' means 'reliable.'
The corpus also flags a kind of realism that latent variables *can't* reach, because it isn't information at all. Smooth human conversation runs on implicit relational work — reference repair, topic hand-offs, grounding acts — that training rewards information prediction never teaches a model to do Why don't language models develop conversation maintenance skills?. Preference optimization makes this worse, eroding grounding acts by ~77.5% below human levels so models sound confident while silently failing in multi-turn exchanges Does preference optimization harm conversational understanding?. And the sharpest warning: simulators look socially competent mainly because one model secretly puppets every interlocutor. Introduce real information asymmetry — give one agent private knowledge the other lacks — and the apparent competence collapses, revealing the model was skipping the grounding work all along Why do LLMs fail when simulating agents with private information?.
So the honest synthesis: controllable latent variables *do* ground simulators to measurable conversational realism, and that's a real, validated result. But 'realistic' there means statistically indistinguishable surface dialogue — not the relational repair work that holds real conversations together, and not robustness to the information asymmetry of genuine two-party talk. The knobs get you a convincing transcript; they don't yet get you a competent interlocutor.
Sources 8 notes
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
Research shows that realistic synthetic dialogues require three multiplicative layers: subtopic specificity, Big Five persona variation, and 11 contextual characteristics via Chain of Thought reasoning. This structured approach captures 90.48% of in-domain dialogue performance.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
Humans keep conversations smooth through implicit techniques like reference repair and topic hand-off that sustain relational interaction, not convey information. Language models don't develop these because training signals reward information prediction, not relational work.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.