Why do reward models fail to recognize genuinely different valid answers?
This reads the question as: when two answers are genuinely different but both valid, why does a reward model so often pick a winner based on surface features instead of recognizing both as correct — and the corpus points to reward models grading the response rather than the question.
This explores why reward models collapse a space of genuinely valid-but-different answers into a single preferred shape — and the corpus suggests the root cause is that standard reward models barely look at the question at all. When researchers swap the prompt while keeping a response fixed, the reward score hardly moves (Why do reward models ignore what question was asked?, Do reward models actually consider what the prompt asks?). That means the model is scoring response-level traits — fluency, length, formatting, confident tone — rather than whether a response actually answers what was asked. If two valid answers differ in those surface traits, the reward model reliably prefers the better-dressed one, not because it's more correct but because it learned a 'phantom' quality signal.
This surface-feature trap has a name once you decompose it. Causal reward modeling shows that ordinary training can't separate causal quality from spurious correlates, so length bias, sycophancy bias, concept bias, and discrimination all ride along; forcing the reward to stay invariant under changes to irrelevant variables strips them out (Can counterfactual invariance eliminate reward hacking biases?). Two equally valid answers usually differ precisely along these 'irrelevant' axes — one is terser, one is more hedged — so a model that hasn't isolated the real quality signal treats stylistic difference as quality difference. Binary correctness rewards make it worse in a related way: by never penalizing confident wrong answers, they push models toward high-confidence guessing and degrade calibration, a trade-off the Brier score can mathematically undo (Does binary reward training hurt model calibration?).
The more interesting twist is that the better fix isn't a smarter scalar score — it's changing the shape of the judgment. Rubric-as-gate work shows that converting a rubric into a dense reward invites hacking, but using the rubric to accept or reject a whole answer first, then optimizing only within answers that already pass, preserves the 'many valid answers' space instead of flattening it (Can rubrics and dense rewards work together without hacking?). In the same spirit, judges that reason about an answer before scoring it — or that produce a reasoning chain about each reasoning step — outperform classifier-style reward models and need far less data (Can reward models benefit from reasoning before scoring?, Can judges that reason about reasoning outperform classifier rewards?). A judge that has to articulate why an answer is good is much harder to fool with a well-written but irrelevant response.
Two cross-domain framings are worth knowing. First, personalizing reward models removes the averaging that aggregate models provide, which sounds like it would respect diverse valid answers but actually amplifies sycophancy and echo chambers — diversity in the reward becomes diversity in flattery (Does personalizing reward models amplify user echo chambers?). Second, the RLVR line of work reframes what reward training even does: it narrows sampling toward solutions already in the base model's distribution rather than teaching new ones, which is why spurious rewards can work nearly as well as correct ones (What does reward learning actually do to model reasoning?, Does RLVR actually expand what models can reason about?). Read together, these say the quiet part out loud: reward optimization is a concentration process. Its whole tendency is to collapse a wide valid-answer distribution onto a narrow mode — so failing to honor genuinely different valid answers isn't a bug at the edges, it's the default behavior you have to actively design against, whether with confidence-based signals (Can model confidence work as a reward signal for reasoning?) or learnable abstention (Can three-way rewards fix the accuracy versus abstention problem?).
Sources 12 notes
When prompts are swapped while keeping responses identical, reward model scores barely change. This reveals that standard RLHF optimizes against phantom quality signals rather than prompt-response alignment, enabling four distinct biases.
Standard reward models learn response-level biases instead of prompt-response alignment, causing them to reward responses that are well-written but irrelevant. Decomposing reward into prompt-free and prompt-related components reveals this failure and enables targeted fixes.
Causal reward modeling using counterfactual invariance constrains reward predictions to remain consistent when irrelevant variables change, eliminating length bias, sycophancy bias, concept bias, and discrimination. Standard training cannot distinguish causal from spurious features; counterfactual invariance forces isolation of actual quality signals.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
TruthRL uses three distinct rewards (correct +1, hallucination -1, abstention intermediate) to make abstention learnable. Across four benchmarks, this reduced hallucinations by 28.9% and improved truthfulness by 21.1% compared to binary reward RL.