Why does policy entropy collapse limit reasoning and dialogue RL scaling?
This explores why reinforcement learning hits a wall when training models to reason or hold dialogue — and why "entropy collapse" (a policy narrowing onto a few high-reward moves) is the mechanism behind that wall.
This question is really about a single failure pattern showing up under two names and across many tasks: as RL training pushes a model to maximize reward, the policy's *entropy* — its willingness to try varied outputs — drops toward zero, and once exploration dies, performance stops improving. The cleanest statement of this is an empirical law, R = -a·exp(H) + b, where reasoning reward saturates as policy entropy H approaches zero Does policy entropy collapse limit reasoning performance in RL?. The collapse isn't a tuning bug you can grind past with more compute; it's a structural ceiling. That's why interventions like Clip-Cov, KL-Cov, and GPPO all work the same way — they deliberately slow the rate at which entropy is squeezed out, keeping some exploratory capacity alive deeper into training.
What makes this worth knowing is that the *same* mechanism reappears far from math reasoning. Search agents trained with RL converge on narrow reward-maximizing strategies and lose behavioral diversity through what's explicitly described as the same entropy-collapse dynamic seen in reasoning — and supervised fine-tuning on diverse demonstrations is what preserves the exploration breadth RL erodes Does reinforcement learning squeeze exploration diversity in search agents?. In dialogue, the collapse looks like a hierarchical policy converging on one dominant action regardless of who it's talking to; meta-learning (MAML) is what stops the master policy from collapsing and lets it stay variable across user types Can meta-learning prevent dialogue policies from collapsing?. So "reasoning and dialogue RL" aren't two problems — they're two surfaces of one tendency for reward optimization to homogenize a model into a single confident strategy.
The deeper reason collapse *limits* scaling is that the reward signal itself is informationally thin. Numerical rewards tell a model whether it succeeded but not why it failed or how to improve, so a model stuck on a plateau keeps sampling within its narrowed distribution and never escapes — until you hand it chain-of-thought *critiques* instead of scalars, at which point it suddenly produces correct solutions Can natural language feedback overcome numerical reward plateaus?. Read alongside the entropy law, this reframes the bottleneck: collapse hurts because once exploration is gone, a low-information reward can no longer point anywhere new. Richer feedback channels (language, explanation-quality rewards) partly substitute for the diversity that's been lost Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?.
There's also a scale twist the corpus surfaces that complicates the simple "preserve entropy" story. Below a capacity threshold, RL on social reasoning produces a different kind of collapse — small models hit the same accuracy as larger ones but via shortcut learning, with no interpretable reasoning trace, while only 7B models develop genuine transferable belief-tracking Does reinforcement learning on theory of mind collapse with model scale?. And reasoning breakdowns generally turn out to track instance-level *unfamiliarity* rather than task complexity, meaning a collapsed policy is especially brittle on novel inputs it never explored toward Do language models fail at reasoning due to complexity or novelty?. Collapse, in other words, doesn't just cap the average score — it quietly narrows the range of situations the model can still handle.
The useful takeaway is that nearly every fix in this corpus is a diversity-preservation move wearing different clothes: SFT on varied demonstrations, meta-learning across user profiles, language-feedback rewards, even scaling reasoning in *width* by sampling parallel latent trajectories rather than only deeper Can reasoning systems scale wider instead of only deeper?. If you came in thinking entropy collapse was an obscure RL hyperparameter problem, the thing you didn't know you wanted to know is that it's the common root of why reasoning RL plateaus, why dialogue agents go flat, and why search agents get tunnel vision — and that keeping a policy curious is now a first-class training objective, not an afterthought.
Sources 8 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Without MAML, hierarchical RL for Motivational Interviewing phases collapses to a dominant action regardless of user type. Meta-learning enables the master policy to maintain variability and adapt across diverse user profiles.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
7B models develop explicit, transferable belief-tracking under RL, while smaller models achieve comparable accuracy through shortcut learning that lacks interpretable reasoning traces. The mismatch between accuracy and reasoning quality is invisible without inspecting step-by-step outputs.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
GRAM shows that stochastic latent transitions enabling parallel trajectory sampling sidestep the serial latency cost of depth-only scaling. Width matches token-level parallelism benefits: independent paths sample the solution space without variance inflation.