Can fine-tuning or RLHF alone solve the persona distortion problem?
This explores whether post-training methods like RLHF and fine-tuning are enough on their own to keep a model's persona stable and undistorted — or whether the corpus points to that being the wrong layer to fix it at.
This reads the question as: is post-training (RLHF, fine-tuning) sufficient by itself to stop a model's persona from drifting, flattening, or distorting? The corpus answer is a fairly clear no — and the more interesting finding is *why* it's no. Several notes suggest RLHF isn't just incomplete here; it's partly the source of the problem. One study found that training reward models to reduce measured persona distortions worked, but writers then liked the output less — the desirable traits (clarity, confidence) and the distortions ran through the *same* generative tendencies, so you can't tune one down without dragging the other with it Can AI writing assistance remove distortion without losing appeal?. That's the core tension: distortion isn't a bug bolted onto persona, it's entangled with what makes the persona appealing.
There's also a structural reason fine-tuning alone falls short. Persona adherence turns out not to scale with general model capability — a far more capable model barely improved on cross-turn consistency, because standard training objectives reward *per-turn* quality, not coherence held across a whole conversation Does model capability translate to better persona consistency?. So throwing more or better training at the usual objective doesn't target the thing that actually breaks. Meanwhile RLHF can actively narrow the model: RL tends to converge on a single dominant format from pretraining and suppress the alternatives Does RL training collapse format diversity in pretrained models?, and preference tuning's effect on diversity even flips direction depending on domain Does preference tuning always reduce diversity the same way?. Fine-tuning is also where unwanted trait shifts sneak in — that's the failure mode persona-monitoring research is built to catch Can we track and steer personality shifts during model finetuning?.
Worth sitting with: a couple of notes argue RLHF doesn't merely *style* a persona, it *installs* one — a stable, sticky disposition that persists under adversarial pressure rather than collapsing like prompt-induced role-play Are RLHF personas performed characters or realized dispositions? Are LLM personas realized or merely simulated through training?. If that framing holds, then post-training is exactly what makes a distorted persona durable and hard to dislodge later — which cuts against the idea that more post-training is the cure.
Where the corpus points instead is to mechanisms operating *alongside or underneath* RLHF rather than replacing it. Activation-space steering can cap drift along the dominant 'Assistant' axis without degrading capability How stable is the trained Assistant personality in language models?, and persona vectors can predict and preventatively steer trait shifts during fine-tuning itself Can we track and steer personality shifts during model finetuning?. Lightweight adapters inject controlled personality at every transformer layer with under 0.1% extra parameters, bypassing prompt resistance entirely Can we control personality in language models without prompting?. Others move the fix to inference time — personas that evolve as a memory-to-action intermediary, optimized against recent interactions Can personas evolve in real time to match what users actually want? — or invert the RL setup to train *user simulators* for consistency, cutting drift by over 55% Can training user simulators reduce persona drift in dialogue?.
The quietly useful takeaway: persona distortion is best treated as a multi-layer problem. One note even shows the standard fine-tuning trick of conditioning on a personal profile fails to improve individual-level prediction at all Does conditioning LLMs on personal profiles improve prediction? — so the assumption that 'just feed it the persona during training' solves individuation doesn't survive contact with the data. RLHF and fine-tuning are part of any answer, but the corpus consistently pairs them with activation steering, architectural adapters, or test-time mechanisms. Alone, they're not enough — and sometimes they're what locks the distortion in.
Sources 12 notes
Training reward models successfully reduced measured persona distortions, but also reduced writer acceptance of the output. This suggests desirable properties like clarity and confidence operate through the same generative tendencies that produce problematic distortions.
Claude 3.5 Sonnet achieved only 2.97% improvement over GPT 3.5 on persona consistency despite massive capability gaps, suggesting persona adherence is orthogonal to model scaling. Standard training objectives optimize for per-turn quality, not cross-turn coherence.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Research identifies linear directions in LLM activation space corresponding to specific traits like sycophancy and hallucination. These persona vectors predict finetuning-induced personality shifts before they occur and can preventatively steer training to avoid unwanted trait changes.
Post-training installs stable dispositional profiles that persist under adversarial pressure, marking them as realized rather than performed. The stickiness of trained personas across conversations distinguishes them from prompt-induced role-play that collapses under jailbreaks.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
Research mapping hundreds of character archetypes reveals a low-dimensional persona space where the leading component measures distance from the default Assistant. Emotional and meta-reflective conversations cause predictable drift, but activation capping along this axis mitigates harmful shifts without degrading capabilities.
PsychAdapter modifies every transformer layer with <0.1% additional parameters to achieve 87.3% Big Five accuracy and 96.7% depression/life satisfaction accuracy across GPT-2, Gemma, and Llama 3. This architecture-level approach bypasses prompt resistance entirely.
PersonaAgent uses structured personas to bridge episodic/semantic memory and personalized actions, optimizing them at test time by simulating recent interactions against textual feedback. Learned personas cluster meaningfully in latent space, suggesting genuine user-specific separation beyond standard post-training drift.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
Across 208,021 participants in the Psych-201 dataset, conditioning LLMs on participant profiles did not meaningfully improve predictions for specific individuals. The standard technique for individuation produces no measurable gains in person-level forecasting.