What role does KL penalty strength play in format selection?
This reads the question as: when you train a model with RL, the KL penalty keeps it tethered to its pretrained starting point — so how does loosening or tightening that leash decide which output format the model settles on?
This explores how the KL penalty — the RL knob that controls how far a model is allowed to drift from its pretrained self — interacts with which format wins out during training. Worth saying plainly up front: the corpus has no single note that isolates KL penalty strength as a dial and measures format outcomes against it. But several notes circle the mechanism closely enough to sketch what's going on, and the picture they paint is more interesting than the literal question assumes.
The key finding is that RL doesn't invent formats — it picks favorites among ones already latent in pretraining. RL training reliably converges on a single dominant pretraining format within the first epoch and suppresses the alternatives, and tellingly the winner depends on model scale, not on which format performs best Does RL training collapse format diversity in pretrained models?. That reframes the KL question: a strong KL penalty holds the model near its pretrained distribution, where many formats coexist; a weak one frees RL to collapse hard onto whichever format the reward gradient amplifies first. The penalty isn't selecting a format so much as setting how aggressively the model is allowed to throw the others away.
Why this matters becomes clear once you see how much rides on format. Training format shapes reasoning *strategy* about 7.5 times more than domain content does — multiple-choice training pushes models toward breadth-first exploration while free-form training produces depth-first reasoning Does training data format shape reasoning strategy more than domain?. So a format collapse during RL isn't cosmetic; it can quietly lock in an entire reasoning style. And format compliance has a real cost: strict output-format constraints measurably degrade reasoning, as if formatting and thinking compete for the same generation budget Do strict output formats hurt LLM reasoning ability?. A loose KL leash that collapses onto a rigid format could therefore trade reasoning capacity away without anyone noticing.
The deeper throughline across these notes is that the pretrained prior, not the RL algorithm, sets the ceiling. When two simple techniques let vanilla PPO match fancier methods like GRPO and DAPO, the lesson was that most RL tricks are setup-sensitive and the prior dominates the outcome Can two simple techniques match complex RL algorithms?. KL penalty strength is precisely the term that governs your relationship to that prior — turn it up and you inherit the prior's format diversity, turn it down and you let the reward signal sculpt freely (for better or worse). It's also worth knowing that RLHF's effect on diversity isn't even one-directional: it compresses lexical variety in code but expands it in creative writing, because each domain rewards different things Does preference tuning always reduce diversity the same way?. So the 'right' KL strength for format selection isn't a universal constant — it depends on whether your domain wants convergence or spread.
If you came here wanting a tuning recipe, the honest answer is the corpus doesn't have one. But what it does have is arguably more useful: the realization that format selection during RL is mostly a story about how tightly you stay bound to pretraining, and that the format you end up with can silently reshape how the model reasons.
Sources 5 notes
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Models trained on multiple-choice data adopt breadth-first exploration (Cohen's d up to 1.5), while free-form training produces depth-first reasoning. Format effect dwarfs domain effect, meaning presentation matters far more than content type.
Schema-specific format requirements cause measurable reasoning decline across multiple models. Removing schema constraints while keeping loose format type recovers most lost performance, suggesting format compliance and reasoning compete for the model's generation capacity.
Advantage normalization and token-level loss aggregation allow critic-free PPO to surpass more complex algorithms. Systematic evaluation shows most RL techniques are setup-sensitive; the pretrained prior, not algorithm choice, sets performance ceiling.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.