Why do outcome-based reward models fail at intermediate step evaluation?

The Reasoning Language Models Blueprint provides a precise taxonomy of the two primary reward model families and their failure modes:

Outcome-Based Reward Models (ORMs):

• Evaluate reasoning solely based on final outcome: P(correct(zT+1) | z0, ..., zT+1)
• Training objective is misaligned with intermediate step evaluation — they are trained on final outcomes only
• Systematically pessimistic for intermediate steps: a correct intermediate step can look "wrong" if a subsequent error occurs
• High false-negative rate: ORMs underestimate solvability of problems from intermediate states
• Cannot distinguish between "the chain got lucky" and "the chain reasoned correctly"

Process-Based Reward Models (PRMs):

• Evaluate reasoning step-by-step: P(correct(zt) | z0, ..., zt)
• Dense rewards enable error localization — can pinpoint which step went wrong
• Better alignment with MCTS, which requires per-action evaluation rather than per-trajectory evaluation
• Trade-off: require extensive step-level annotations from skilled annotators (expensive), or from LLM-generated annotations (lower quality due to limited self-evaluation capability)

Q-Value models (Q-VMs) vs V-Value models (V-VMs): A further split. Q-VMs evaluate Q(s, a) — expected cumulative reward for taking action a in state s — and are preferred for MCTS because they evaluate edges (actions), not just nodes (states). V-VMs evaluate V(s) — expected cumulative reward from state s — and provide a broader state-level view but less guidance for action selection.

Generative Reward Models (GRMs) as a third category: The RRM and DeepSeek-GRM papers introduce a third family alongside ORMs and PRMs. GRMs harness LLMs to produce interpretable, natural-language feedback rather than scalar scores. They can follow adaptive evaluation instructions, construct synthetic training data, and self-improve through iterative refinement. GRMs unify scoring of single, paired, and multiple responses within pure language representation. However, concerns persist about evaluation reliability — LLMs may produce biased or hallucinated judgments that diverge from human standards. Since Can reward models benefit from reasoning before scoring?, GRMs become most powerful when combined with extended reasoning before judgment.

This taxonomy explains why Can self-supervised process rewards replace human annotation? matters: the annotation cost is the primary bottleneck for PRMs, and self-supervised approaches address precisely this.

The ORM/PRM split is also the reason Can curriculum learning approximate expensive process supervision? is significant — R3 uses outcome supervision only but achieves process-supervision-like step feedback by decomposing the problem curriculum.