Does preference tuning help or hurt the exploration of solution spaces in code?
This explores whether RLHF and preference optimization broaden or narrow how a model searches the space of possible code solutions — and the corpus says the effect is domain-specific, with code being exactly the domain where tuning narrows rather than widens.
This explores whether preference tuning broadens or narrows how a model searches for code solutions — and the most direct answer in the collection is that code is precisely the domain where tuning *narrows*. The same RLHF pass that *increases* lexical and syntactic diversity in creative writing *reduces* it in code generation Does preference tuning always reduce diversity the same way?. The reason is in what each domain rewards: creative writing pays off for being distinctive, while code pays off for converging on the one correct answer. So preference tuning isn't uniformly good or bad for exploration — it amplifies whatever the reward signal already points at, and in code that signal points at convergence.
Whether that convergence helps depends on what you think exploration is *for*. If a single correct solution exists, narrowing toward it is the point. But the collection raises a quieter worry: tuning may be sharpening the wrong thing. RL fine-tuning (even GRPO) tends to sharpen memorized template-matching rather than install a genuine search procedure — models that look strong in-distribution collapse on near-identical out-of-distribution variants Do fine-tuned language models actually learn optimization procedures?. Supervised fine-tuning shows the same pattern from a different angle: it teaches the *surface form* of good solutions without the reasoning to construct valid ones Does supervised fine-tuning actually improve reasoning on optimization problems?. If tuning collapses your search around polished-looking but shallow paths, you've lost exploration without gaining correctness.
The failure mode this sets up is visible in how reasoning models actually move through a solution space. They tend to wander into invalid branches and then abandon promising ones prematurely — a structural disorganization, not a compute shortage, since decoding-level nudges recover accuracy with no fine-tuning at all Why do reasoning models abandon promising solution paths?. That's a strong hint that good solutions are already reachable but are getting pruned too early — and a reward signal that prizes confident convergence would prune them harder.
The more interesting move in the corpus is the work that deliberately re-injects breadth that tuning would otherwise squeeze out. Training a model to generate diverse *abstractions* before solutions enforces a breadth-first search that beats simply sampling more solution attempts in parallel Can abstractions guide exploration better than depth alone?. The Darwin Gödel Machine keeps an evolutionary *archive* of agent variants rather than greedily keeping only the current best, which is what lets it discover genuinely new coding capabilities Can AI systems improve themselves through trial and error?. And a bilevel system that rewrites its own search code found new mechanisms specifically by *breaking* the inner loop's deterministic patterns Can an AI system improve its own search methods automatically?. All three treat preserved diversity as the engine of discovery — the opposite of what convergence-rewarding preference tuning does.
The through-line — and the thing worth taking away — is that this isn't really a code-specific quirk. Preference optimization erodes whatever doesn't serve its narrow target: in dialogue it strips out the grounding acts that build shared understanding Does preference optimization damage conversational grounding in large language models?, and in writing it can't be a clean alignment target at all because the same optimization that polishes also distorts Can user preference guide AI writing tool alignment?. For code, the lesson is that if you want a model to *explore* rather than just produce the most-rewarded-looking answer, exploration has to be protected architecturally — through abstractions, archives, or decoding-time search — because preference tuning, left alone, will quietly trade it away for convergence.
Sources 9 notes
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Supervised fine-tuning makes model outputs look correct—proper JSON structure, valid identifiers, expected sections—without making them physically feasible. The model learns surface features of solutions, not the reasoning to construct valid ones.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
Writers prefer AI rewrites 63% of the time but object to systematic persona distortions those same rewrites introduce. Mitigation studies show polish and distortion are entangled at the model level—preference optimization produces both simultaneously.