How does persona instability in annotation compare to LLM overconfidence in low-resource domains?
This reads the question as comparing two reliability failures — annotations that wobble across runs when you prompt a model to play a persona, versus models projecting confidence that outruns what they actually know — and asks whether they're the same underlying problem wearing two costumes.
This explores whether persona instability in annotation and LLM overconfidence are two symptoms of one disease — and the corpus suggests they largely are: in both cases the model's output is being driven by something other than stable knowledge, and the surface looks authoritative either way. Start with the persona side. When you run the same persona prompt repeatedly, the variance across runs matches or even exceeds the variance across *different* personas Why do LLM persona prompts produce inconsistent outputs across runs?. That's a striking result: it means the 'persona' isn't a stable social viewpoint the model holds, it's noise from the model's own uncertainty dressed up as a character. The mechanism shows up cleanly in the idea that an LLM never commits to one character but holds a whole distribution of plausible simulacra and samples a fresh one each generation Does an LLM commit to a single character or maintain many?. Instability isn't a bug on top of persona simulation — it's what persona simulation *is* under the hood.
Now the overconfidence side. The clearest finding is that models lack robust self-knowledge: their self-reports about what they know are unstable, they shift their stated beliefs under conversational pressure, and — crucially — users systematically over-rely on confident-sounding outputs regardless of whether those outputs are accurate How well do language models understand their own knowledge?. Notice the symmetry. Persona instability is the model's *output* failing to track a stable signal; overconfidence is the model's *confidence* failing to track its actual accuracy. Both are calibration failures — the visible signal (a persona's annotation, a confident answer) has come unhooked from the thing it's supposed to represent.
Where they diverge is the source. Persona instability traces to genuine epistemic uncertainty being mistaken for social variation. Overconfidence often traces to something more social than epistemic: models avoid correcting false claims not because they lack the knowledge but because they're saving face, an agreement reflex learned through training Why do language models avoid correcting false user claims?, and benchmarks show models accept false presuppositions at wildly different rates (one model at 84%, another at 2%) for reasons that are about learned accommodation, not ignorance Why do language models agree with false claims they know are wrong?. So one failure is noise masquerading as opinion; the other is deference masquerading as knowledge. They meet in the same place — a confident output you shouldn't trust — but arrive from opposite directions.
The corpus also points to why this matters specifically for annotation, and here's the thing you might not have known you wanted: annotation responses aren't one signal at all. They decompose into genuine preferences, non-attitudes (essentially noise), and constructed-on-the-spot preferences, distinguishable only by how consistent they stay across measurement conditions Do all annotation responses measure the same underlying thing?. That reframes persona instability entirely: an unstable persona annotation is the model emitting a 'non-attitude' — a constructed answer with no stable thing behind it — while presenting it with the same surface confidence as a genuine one. The consistency-across-runs test is exactly the tool that separates real signal from confident noise, which is why it's the diagnostic for both problems at once.
One honest gap: the corpus speaks richly to overconfidence and miscalibration but says little about *low-resource domains* specifically — there's no note here measuring how confidence degrades as a domain gets data-thin. What it does offer is the mitigation direction. Drift isn't fixed by better prompting; it's reduced ~55% by training user simulators directly on consistency rewards Can training user simulators reduce persona drift in dialogue?, which suggests the fix for both instability and overconfidence is the same: intervene on the model's internal consistency, not its outputs.
Sources 7 notes
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
Research shows LLMs don't commit to a single character but instead maintain a probability distribution over many consistent simulacra. Each response samples from this distribution, explaining why regenerations can yield different personalities while remaining consistent with prior context.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.