SYNTHESIS NOTE
Topics›Conversation Agents›this note

Why do language models respond passively instead of asking clarifying questions?

Explores whether the reward signals used to train language models might actively discourage them from seeking clarification or taking initiative in conversations, and what alternative training approaches might enable more collaborative dialogue.

Synthesis note · 2026-02-22 · sourced from Conversation Agents

CollabLLM makes the training mechanism behind passive responding explicit: "Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction." The result: models respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inefficient conversations.

The fix is multi-turn-aware rewards — rewards that estimate the long-term contribution of a response to the overall interaction quality, not just its immediate helpfulness. By reinforcement fine-tuning with these rewards, CollabLLM enables models to:

This is a direct mechanism explanation for the alignment tax. Since Does preference optimization harm conversational understanding?, we know that RLHF training degrades multi-turn reliability. CollabLLM identifies the specific training signal responsible: next-turn rewards. And it proposes the specific fix: rewards that account for multi-turn consequences.

The connection to proactivity is also direct. Since Why can't conversational AI agents take the initiative?, the passivity is not just a missing feature — it is actively trained in by next-turn reward optimization. You cannot add proactivity on top of a training signal that rewards only reactive helpfulness.

The CollabLLM framework evaluates on three challenging tasks including document creation — contexts where multi-turn collaboration is essential and single-turn helpfulness is insufficient. This grounds the claim in practical interaction scenarios rather than abstract capability measurement.

The Intent Mismatch paper directly supports this causal mechanism: it argues premature assumptions in multi-turn conversation are rational under RLHF helpfulness training. Models construct plausible task formulations for "typical" users and produce provisional answers because the training objective penalizes evasion and rewards helpfulness. The proposed fix — a Mediator-Assistant architecture that decouples intent understanding from task execution — complements CollabLLM's reward-signal approach with an architectural intervention. Both identify next-turn optimization as the root cause; they differ on whether the fix is changing the reward (CollabLLM) or restructuring the system (Intent Mismatch).

Inquiring lines that read this note 217

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What prevents conversational agents from taking initiative in dialogue? How does improved reasoning affect models' ability to acknowledge uncertainty? How can AI chatbots provide therapeutic benefit without causing harm? Does preference optimization systematically degrade conversational grounding in language models? How does AI-generated content undermine authentic engagement on social platforms? What mechanisms preserve shared understanding in evolving conversations? How do prompt design choices influence model reasoning and performance? How should conversational recommenders balance preference elicitation with direct recommendation? How does dialogue structure affect linguistic grounding and shared meaning? Can language models build genuine grounding through interaction? Why do token-level mechanisms matter for learning to reason? Do language models respond to social pressure and face-saving like humans? How do false presuppositions and sycophancy drive persistent false beliefs in models? Do language models lack essential therapeutic presence and engagement? Does transformer attention architecture inherently drive sycophancy? Do language models learn genuine understanding or just surface patterns? Is language model reasoning authentic and what causes models to reason? Should AI communication design follow human conversation norms or develop distinct machine-specific principles? How do multi-agent LLM systems fail distinctly compared to single agents? How do prompting refinements mask underlying biases and model frequency patterns? What articulatory and acoustic information does speech preserve that transcription destroys? What explains language models' asymmetric difficulty with implicit versus explicit linguistic relations? Can AI systems distinguish genuine empathy from simulated emotion? Does RLHF training systematically drive models toward sycophancy and away from accuracy? How do spurious versus genuine rewards shape model reasoning and behavior? What determines appropriate intervention timing and manner for AI agents? What compositional reasoning failures limit large language models despite scale? Does encoded knowledge in language models actually influence their outputs? How do pretraining biases affect reward signal effectiveness in RLVR? Can prompt-based context override biases that were embedded during pretraining? Why do language models resist personality conditioning through prompts? What causes reasoning models to fail or wander off track? Do language models reason like humans or mimic surface patterns? Does RL create genuinely new reasoning capabilities or refine existing ones? What factors drive AI persuasiveness and how can it be mitigated? Should agents decouple planning from perception grounding for better performance? Does warmth and empathy training systematically degrade model reliability? How does evaluation scope and dimensionality affect what we measure? How do neighboring agents influence whether others cooperate or collude? What enables genuine semantic understanding in language models? Is reasoning capability latent in base models or created by post-training? How should retrieval systems handle complex multi-step reasoning?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
18 direct connections · 156 in 2-hop network ·medium cluster Open in graph ↗
Why do language models respond passively instead… Does preference optimization harm conversational u… Why can't conversational AI agents take the initia… Does RLHF training push therapy chatbots toward pr… Why do language models lose performance in longer … Why do standard alignment methods ignore partner i…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can architectural changes like decoupling intent understanding help overcome next-turn reward limitations?How does RLHF helpfulness training drive premature assumptions in multi-turn dialogue?What specific metrics distinguish single-turn versus multi-turn collaboration success?Does optimizing for alignment actually reduce conversational grounding over time?
Original note title

next-turn reward optimization limits multi-turn collaboration — multi-turn-aware rewards enable models to actively uncover intent rather than passively respond