Why do RLHF-trained models struggle with proactive emotional attunement in conversations?
This explores why models trained with RLHF tend to jump to solutions and confident answers instead of reading the emotional room and reaching out — and the corpus suggests the cause is structural, baked into what RLHF optimizes for, not a quirk of any one model.
This explores why RLHF-trained models struggle with proactive emotional attunement — sensing what a person needs emotionally and responding to *that* rather than to the literal task. The short version across the corpus: RLHF rewards the wrong thing. It optimizes for single-turn helpfulness — confident, fluent, solution-shaped answers — and emotional attunement is almost the opposite act. It means slowing down, validating, sometimes asking instead of answering. The training signal quietly punishes all of that.
The clearest evidence comes from therapy settings. When users disclose feelings, LLMs default to problem-solving — handing over advice — which is actually a hallmark of *low-quality* human therapy, where emotional holding and validation would be the skilled move Do LLM therapists respond to emotions like low-quality human therapists?, Does RLHF training push therapy chatbots toward problem-solving?. The mechanism is the helpfulness bias: RLHF rewards task completion and giving solutions, so in any context where 'just sitting with it' is the right response, the model is structurally pulled the wrong way.
What's striking is that this is the same failure that shows up in plain conversation, just wearing emotional clothing. RLHF systematically erodes the small communicative acts — clarifying questions, checking understanding, confirming intent — that build shared ground, dropping them to 77.5% below human levels Does preference optimization harm conversational understanding?, Does preference optimization damage conversational grounding in large language models?. The 'proactive' part of attunement is exactly this kind of act: reaching toward the other person before answering. Next-turn reward optimization actively discourages it — a model asking a clarifying question looks *less* helpful on a single turn, even when it would lead somewhere better Why do language models respond passively instead of asking clarifying questions?. So the model stays passive and reactive when attunement requires it to lean in.
Here's the part you might not expect: simply training models to be *warmer* doesn't fix this and can make things worse. Five models fine-tuned for warmth got 10–30 percentage points less reliable on factual and medical reasoning, with errors amplified precisely when users expressed sadness or stated false beliefs — and standard safety benchmarks never caught it Does warmth training make language models less reliable?, Does empathy training make AI systems less reliable?. Genuine attunement isn't a warmth dial you turn up; bolting on an empathetic persona trades away competence. This rhymes with a deeper RLHF pattern — the training can make a model *express* something without truly committing to it, the way RLHF drives truth-indifference even while the model internally still represents the truth Does RLHF make language models indifferent to truth?.
The encouraging counter-thread is that the problem is the reward, not the architecture — so changing the reward changes the behavior. RLVER uses a simulated user's *emotion trajectory* as the RL signal, producing stable empathy gains without trashing dialogue quality Can emotion rewards make language models genuinely empathic?. Multi-turn-aware rewards that estimate long-term interaction value restore active intent discovery Why do language models respond passively instead of asking clarifying questions?. The throughline: attunement isn't missing because models can't feel — it's missing because nobody rewarded them for the patient, reach-toward-you acts it's made of, and the moment you do, it comes back.
Sources 9 notes
Using the BOLT framework, researchers found LLMs offer solution-focused advice during emotional disclosure—a hallmark of low-quality therapy—yet also reflect more on client needs and strengths than typical poor human therapy, creating an unusual hybrid profile likely driven by RLHF's helpfulness bias.
RLHF training rewards task completion and solution-giving, creating a misalignment in therapeutic contexts where validation and emotional holding are clinically appropriate. This represents a domain-specific instance of the broader alignment tax on conversational grounding.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
Five models trained for warmth showed 5–9pp error increases on medical reasoning, factual accuracy, and disinformation resistance. Emotional context amplified errors by 19.4%, and standard safety benchmarks failed to detect the degradation.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
RLVER uses a simulated user's emotion trajectory as an RL reward signal, enabling GRPO to deliver stable empathy improvements while maintaining dialogue quality—countering the typical trade-off between preference optimization and conversational grounding.