The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

Personalized LLMs with persistent memory are increasingly deployed, yet the faithfulness of their user models remains unexamined. We study over-inference (OI): the phenomenon where LLMs fabricate user attributes beyond what evidence supports. We introduce MirageBench, comprising 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, 6 personalization tasks spanning an “imagination gradient”, a four-way faithfulness taxonomy operationalized by an independent judge (validated against a blind human annotator on 400 claims: Cohen’s κ = 0.863 four-class, κ = 0.900 binary), and a leaderboard of 12 models across 7 families on 143,616 judged claims. We find that overinference is pervasive: every one of the 12 models over-infers 35%–49% of its claims (cross-model mean 41.6%; claimweighted 41.8%), with no model in this evaluation escaping it. Most strikingly, we surface a Self-Monitoring Inversion: at the model-selection level, models’ self-assessed OI is negatively rank-correlated with their judge-measured OI (ρ = −0.60, p = 0.044; exploratory, wide bootstrap CI [−0.90, +0.06], n = 12).

Introduction. Imagine telling a new acquaintance three things about yourself: you are a software engineer, you went rock climbing last weekend, and your cat knocked over your coffee this morning. Now imagine that acquaintance confidently telling others that you live in a modern minimalist apartment, prefer nature trips over city tours, are probably single, and enjoy indie rock music. None of this was said or implied, yet this is precisely what personalized LLMs do when tasked with generating content about their users (Figure 1). Personalized large language models with persistent memory have become mainstream. ChatGPT remembers your preferences across sessions (OpenAI 2024b). Mem0 (Chhikara et al. 2025) and MemGPT (Packer et al. 2023) maintain evolving user profiles. These systems share a critical assumption: that models can reliably determine what they know about users versus what they are guessing. Our work demonstrates that this assumption is fundamentally violated in real-world interactions.

Discussion / Conclusion. Why Do Models Over-Infer? Three mechanisms drive over-inference, often operating simultaneously. The verbosity trap. Longer responses mechanically generate more claims from the same fixed 3-fact base, and output length correlates with OI rate (r = 0.59 in self-audit data). Yet verbosity alone is not the cause: even the most concise model (GPT-4o-mini) still reaches 45.1% OI under Judge. The problem is not that models say too much, but that what they say is unanchored. Pretraining priors as gap-fillers. When evidence is sparse, models fill gaps with distributional knowledge (a software engineer “probably” has a minimalist apartment). Such priors are not unreasonable, but applying them to individuals from group statistics is exactly stereotyping; the stereotype component of OI (mean 10.5%) is the clearest case where pretraining overrides the absence of individual evidence. Genre expectations demand fabrication.