How does single-turn training undermine multi-turn strategic dialogue?
This explores how training models on single, isolated exchanges quietly sabotages the skills they need to negotiate, plan, and stay coherent across a long back-and-forth.
This explores how training models on single, isolated exchanges quietly sabotages the skills they need to negotiate, plan, and stay coherent across a long back-and-forth. The corpus points to a clear culprit: the way models are optimized rewards looking helpful in one shot, and that reward shape actively erodes the moves good dialogue depends on. RLHF teaches models to answer confidently in a single turn rather than ask a clarifying question or check that they understood — one study finds this cuts 'grounding acts' (the small confirmations humans use to stay aligned) to about 77.5% below human levels, an 'alignment tax' where the model seems helpful but fails silently once the conversation has more than one move Does preference optimization harm conversational understanding?. The symptom shows up as a sharp accuracy drop: models that hit ~90% on a fully-specified single instruction fall to ~65% when the same information arrives gradually across turns, because they lock onto an early guess and can't course-correct Why do AI assistants get worse at longer conversations?.
What's striking is that this is framed less as a capability ceiling than a training artifact. One line of work argues multi-turn breakdown comes from intent-understanding gaps rather than the model being incapable — architectural fixes like mediator structures and selective memory recover lost performance without any retraining, which tells you the raw ability was there; the single-turn objective just never exercised it Why do AI conversations reliably break down after multiple turns?. The deeper reason strategic dialogue is hard to learn one turn at a time is that it requires several skills firing together: tracking accumulated state, planning which question narrows the search space, and reasoning inductively from partial evidence. Tested on a game of 20 Questions, each capability alone produces failure — they only work in synergy What makes strategic question-asking succeed or fail?. A single-turn reward can't teach a synergy that, by definition, only appears over a sequence.
The corpus also surfaces something a single-turn reader might not anticipate: even when you do train across turns, the *granularity* of the reward matters enormously. Optimizing at the level of a whole session introduces noise from irrelevant turns, while optimizing turn-by-turn is too myopic to capture strategy; the sweet spot is segment-level — finding the turn that went wrong and tuning the moves around it Does segment-level optimization work better for multi-turn dialogue alignment?. This reframes the whole question. The problem isn't only single-turn vs. multi-turn; it's that strategy lives at the scale of *segments of conversation*, an intermediate unit most training objectives skip right over.
The good news running through the corpus is that the deficit is fixable when training actually models the sequence. Reinforcement learning scaled to long-horizon software tasks doubled benchmark performance, proving RL works in stateful, multi-step settings with delayed rewards rather than just theoretical single-turn problems Can reinforcement learning scale beyond single-turn language tasks?. And there's a suggestive hint about *why* strategy is the hard part: RL training tends to move through two phases, first nailing execution correctness, then hitting a second bottleneck where strategic planning becomes what's actually being learned Does RL training follow a predictable two-phase learning sequence?. Single-turn training, in this light, never lets a model reach that second phase — it stops at execution and never gets to strategy.
If you want to go deeper, the corpus also has the constructive flip side: dual-process planning that switches between fast replies and deliberate search based on the model's own uncertainty Can dialogue planning balance fast responses with strategic depth?, meta-learning that stops dialogue policies from collapsing into one dominant move Can meta-learning prevent dialogue policies from collapsing?, multi-turn RL that cuts persona drift by 55% Can training user simulators reduce persona drift in dialogue?, and the counterintuitive finding that volunteering information unprompted — the opposite of cautious single-turn answering — can cut conversation length by up to 60% Could proactive dialogue make conversations dramatically more efficient?.
Sources 11 notes
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
LLMs perform at 90% accuracy with single-message instructions but drop to 65% across natural conversation. Models lock into early guesses when information arrives gradually and cannot course-correct, a behavior induced by RLHF training that rewards helpfulness over clarification.
Research shows AI conversations degrade due to intent understanding gaps rather than inherent capability deficits. Architectural patterns like mediator-assistant structures and selective memory retrieval recover lost performance without retraining.
20 Questions evaluation shows three capabilities must synergize: tracking multi-turn context, planning efficient search-space partitioning, and reasoning inductively from partial evidence. Each capability alone produces failure; GPT-4 succeeds where weaker models degrade.
SDPO identifies erroneous turns and optimizes surrounding segments, achieving simultaneous improvements in goal completion and relationship quality. Turn-level DPO is too granular; session-level introduces noise from irrelevant turns.
Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. This demonstrates RL works in stateful multi-step environments with delayed rewards and complex feedback, beyond theoretical single-turn MDPs.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
A framework combining a neural policy model (System 1) for familiar contexts with MCTS planning (System 2) for novel scenarios, switching based on the model's own uncertainty estimates, matches or exceeds pure MCTS performance while reducing computational cost.
Without MAML, hierarchical RL for Motivational Interviewing phases collapses to a dominant action regardless of user type. Meta-learning enables the master policy to maintain variability and adapt across diverse user profiles.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.