SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Does RL training collapse format diversity in pretrained models?

Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques

A study with full pretraining transparency (models pretrained from scratch on known open datasets) reveals a striking structural pattern: RL fine-tuning does not simply improve reasoning — it systematically selects for and amplifies a single format from the pretraining mixture while collapsing all others.

The mechanism: early in RL training (within the first epoch), the model shifts toward generating outputs in the format of one specific distribution — code-like formats for smaller models, natural language formats for larger models. This transition coincides with the largest accuracy gain, suggesting the selection of a dominant format is what drives improvement, not a gradual enhancement across all formats.

Key findings:

This is distinct from Does policy entropy collapse limit reasoning performance in RL? in an important way. Entropy collapse describes diversity reduction within an output distribution. The echo chamber finding describes distribution selection: RL picks one distribution and amplifies it at the expense of all others. It is a format-level convergence, not just a diversity-level collapse.

The implication for practitioners: RL fine-tuning results depend on what the pretraining data mixture looks like, but this dependence is largely hidden when starting from existing pretrained models whose training data is proprietary. The performance gains attributed to RL algorithms may partially reflect which pretraining distribution was selected, not algorithmic superiority.

Inquiring lines that read this note 310

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How can we prevent synthetic data from contaminating statistical inference and corpora? How should designers communicate what AI systems truly are and can do? Does alignment training create genuine alignment or just output compliance? How do evaluation practices shape which failures stay visible? How do surface patterns enable correct outputs but reduce robustness? Do structural constraints outperform deep architectures in recommendation systems? Why do LLM recommenders underperform collaborative filtering despite their capabilities? Can prompt-based context override biases that were embedded during pretraining? What training dynamics and scale trigger emergence of reasoning capabilities? What types of diversity prevent reasoning systems from collapsing? Do language models develop actual world models or merely task heuristics? What capability trade-offs arise from domain specialization through fine-tuning? How do capability benchmark scores systematically misrepresent true model abilities? Can self-generated feedback reliably guide model training without ground truth? Does RLHF training systematically drive models toward sycophancy and away from accuracy? How do pretraining biases affect reward signal effectiveness in RLVR? Does RL create genuinely new reasoning capabilities or refine existing ones? How do neural networks achieve compositional generalization at scale? Can inoculation prompting prevent emergent misalignment after reward hacking? How does policy entropy collapse constrain scaling of reasoning-focused RL? What articulatory and acoustic information does speech preserve that transcription destroys? How does AI adoption across firms reshape employment and inequality? Why does adding new knowledge through fine-tuning degrade existing capabilities? How does synthetic data quality and diversity affect downstream model capabilities? Can preference-based training achieve better behavior optimization than supervised fine-tuning alone? Is reasoning capability latent in base models or created by post-training? What training data selection strategies maximize generalization across difficulty levels? How much does training format versus domain influence reasoning? How can evolutionary algorithms maintain diversity during solution search? Can models improve accuracy without degrading reasoning quality? What makes distillation transfer some model capabilities while suppressing others? How much do training data properties shape model reasoning? Do reasoning benchmarks predict model performance in long-horizon workflows? Does model confidence reliably signal actual accuracy in practice? Do reasoning traces faithfully reflect actual model reasoning? How does decomposing tasks improve reasoning and prevent failure propagation? Why do token-level mechanisms matter for learning to reason? Where and how do personality traits reside in language models? Can multi-agent systems avoid converging on false agreement without deliberation? How do training data properties determine the emergence of internal misalignment? Does chain-of-thought reasoning reveal genuine computation or imitate patterns? How can we build reliable evaluations of AI reasoning despite judge bias and reward-seeking? What structural properties of attention create systematic model biases? What mechanisms preserve shared understanding in evolving conversations? How does harness optimization generalize across different model architectures and domains? How should agent systems validate and persist generated code artifacts? Does preference optimization systematically degrade conversational grounding in language models? What role does sparsity play in model behavior and scaling decisions? Can intelligent routing over smaller models outperform scaling a single large model? Do language models lack essential therapeutic presence and engagement? How can reward models capture diverse human preferences without excluding minority populations? Can mechanistic interpretability reliably guide practical model design choices? Why does memory consolidation cause performance regression in continual learning? What trajectory-level metrics beyond task success best evaluate agent performance? How does the generation-verification gap limit what we can measure about AI reasoning? Why can recurrent transformers achieve reasoning capabilities that standard transformers cannot? Can iterative DPO replicate online reinforcement learning dynamics for research? Do language models learn genuine understanding or just surface patterns? Why is hallucination an inevitable limitation of current language models? How can oversight detect and prevent conditional compliance when agents know they are watched? How do agent-learned skills transfer and improve across different tasks? Can validator consensus certify semantic correctness beyond agreement? Do language models reason like humans or mimic surface patterns? Does AI assistance promote real skill development or substitute for independent learning? Can inference-time compute effectively substitute for model scale?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 153 in 2-hop network ·dense cluster Open in graph ↗
Does RL training collapse format diversity in pr… Does policy entropy collapse limit reasoning perfo… Why do reasoning models fail differently at traini… Can simple rewards alone teach complex domain reas… Does RL improve domain reasoning by adding knowled… Does reinforcement learning squeeze exploration di… Why does RLVR training narrow a model's problem so…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do smaller models favor code formats while larger models prefer natural language?How does KL penalty strength affect the degree of format collapse during RL?Can RL format selection explain performance gains attributed to algorithmic improvements?Is distribution selection during RL the same compression mechanism as entropy collapse?
Original note title

rl post-training converges on a single dominant pretraining distribution format, suppressing all others