Does semantic diversity in output space compete with reward-component diversity?
This explores whether two different routes to keeping AI outputs diverse during RL training — rewarding semantic variety in what the model says, versus splitting the reward itself into multiple unscalarized components — are rival strategies or complementary ones operating in different spaces.
This pits two answers to the same problem against each other. The problem is well documented: reinforcement learning relentlessly compresses an AI's range. RL training squeezes exploration diversity in search agents through the same entropy-collapse mechanism seen in reasoning Does reinforcement learning squeeze exploration diversity in search agents?, and within the first epoch it amplifies a single pretraining format while suppressing all the alternatives Does RL training collapse format diversity in pretrained models?. The result, at the population level, is an 'Artificial Hivemind' where even very different models converge on near-identical outputs Do different AI models actually produce diverse outputs?. Diversity isn't a luxury here — it's the thing the standard objective actively destroys.
The two fixes attack from opposite ends. DARLING bolts a diversity reward onto the output space: a learned classifier scores how semantically distinct a model's responses are, and that score is optimized jointly with quality — which turns out to catalyze exploration and raise quality, not trade against it Can diversity optimization improve quality during language model training?. Vector Policy Optimization attacks the reward side instead: keep the reward as an unscalarized vector (per test-case, per criterion, per persona) and let solutions specialize across a Pareto frontier. Its central claim is pointed — that this yields 'competent diversity grounded in real task trade-offs rather than external regularizers' Can reward vectors be the hidden source of solution diversity?. Read literally, that's a shot at DARLING's approach: the vector-reward view says an added semantic-diversity term is an arbitrary regularizer, while diversity that falls out of the reward's own structure is meaningful difference rather than difference for its own sake.
So do they compete? Conceptually yes — they disagree about where diversity should *come from*. But the more interesting answer is that they aren't really fighting over the same territory. Semantic diversity is about the surface of the output (does the model say genuinely different things?); reward-component diversity is about the geometry of the objective (does the reward leave room for more than one right answer?). A scalar reward with a semantic-diversity bonus and a vector reward with no bonus can both be diverse, but they'll be diverse in different ways — one spreads across meanings, the other across task trade-offs. They'd only directly collide if you tried to add a semantic-diversity term on top of an already-decomposed vector reward and found the two pulling in opposite directions, which none of these papers actually tests.
What complicates any clean answer is that diversity isn't one thing and isn't always wanted. Preference tuning *reduces* lexical diversity in code (where convergence on the correct solution is the point) but *increases* it in creative writing Does preference tuning always reduce diversity the same way?. Vector rewards encode exactly that intuition structurally — they only manufacture diversity along axes where the task genuinely has trade-offs. And there's a quieter failure mode lurking underneath both: optimizing toward 'common' or high-frequency outputs systematically drifts toward abstraction and erases expert specificity Does word frequency correlate with semantic abstraction?, so a diversity score that rewards safe paraphrase variety could mask a loss of real precision.
The corpus also hints at a third route that sidesteps the rivalry entirely: don't engineer the reward at all, fix the exploration. Step-level critique inside the training loop counteracts tail-narrowing and preserves solution diversity across self-training iterations — a training-time benefit the authors argue is more fundamental than test-time accuracy Do critique models improve diversity during training itself?, and natural-language critique can break the very reward plateaus that scaling numerical rewards can't Can natural language feedback overcome numerical reward plateaus?. If you want to go deeper, the live tension to chase is this: DARLING says add diversity, VPO says decompose the reward, and the critique papers say neither — change what the model explores in the first place. They're three answers to one question that the field hasn't yet run head-to-head.
Sources 9 notes
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
DARLING jointly optimizes for quality and semantic diversity using a learned classifier, finding that diversity rewards catalyze exploration and produce higher-quality outputs than quality-only baselines across both creative and mathematical tasks.
Vector Policy Optimization shows that rewards decomposed per test-case, criterion, or persona provide an inherent diversity structure. Training solutions to span the Pareto frontier across these dimensions produces competent diversity grounded in real task trade-offs rather than external regularizers.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
WordNet analysis shows hypernyms (general concepts) occur more frequently than hyponyms (specific ones). Combined with LLMs' frequency bias, this means preferring common paraphrases systematically drifts toward abstraction, erasing expert-level specificity.
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.