What happens when confident wrong answers become more rewarded than uncertain correct ones?
This explores what reward design does to a model when the training signal pays off confident bluffing over honest uncertainty — and how the corpus diagnoses and repairs that incentive.
This explores what happens to a model's behavior when its training reward favors a confident wrong answer over a hedged correct one. The corpus is unusually direct about the mechanism: standard binary correctness rewards — +1 if right, 0 if wrong — never penalize a confident miss any harder than a hesitant one, so they quietly teach the model that guessing boldly is the optimal policy. One result proves this isn't a side effect but a mathematical inevitability: binary rewards *provably* degrade calibration because they incentivize high-confidence guessing, and the fix is to add a proper scoring rule — the Brier score — as a second reward term that jointly optimizes accuracy and calibration without forcing a trade-off Does binary reward training hurt model calibration?.
The damage is hard to see because of where it hides. Confident wrong answers don't show up in aggregate accuracy — they concentrate in the rare, high-stakes cases (medical triage, legal interpretation, financial planning) where surface heuristics collide with unstated constraints, while overall scores still look strong Why do confident wrong answers hide in standard accuracy metrics?. So a reward scheme that rewards confidence is also a reward scheme whose failures are invisible to the metric you'd use to catch them. That's the trap: the incentive and the blind spot reinforce each other.
The corpus's most interesting move is to make abstention *learnable* rather than penalized. Ternary reward design hands out three distinct signals — correct, hallucination, and a middle reward for honestly saying "I don't know" — which cut hallucinations by nearly 29% while preserving accuracy Can three-way rewards fix the accuracy versus abstention problem?. The same instinct shows up in forecasting, where small models trained with uncertainty-aware objectives and the option to abstain match models ten times their size — suggesting calibration is a latent ability that standard training leaves undertrained, not one the model lacks Can models learn to abstain when uncertain about predictions?.
Here's the turn you might not expect: the model's *own confidence* can become the reward signal that fixes the problem the reward created. Several approaches use answer-span confidence or intrinsic token probability to rank reasoning traces, which restores calibration while strengthening step-by-step reasoning — and does it without human labels or external verifiers Can model confidence work as a reward signal for reasoning? Can model confidence alone replace external answer verification?. Confidence, in other words, is both the thing badly-designed rewards corrupt and the thing well-designed rewards can lean on.
If you want to go further sideways: there's evidence that training a model to *critique* wrong answers builds deeper understanding than training it to imitate correct ones, because engaging with failure modes teaches structure that surface-pattern matching never does Does critiquing errors teach deeper understanding than imitating correct answers?. And rubric-based work shows that *how* you wire a signal matters as much as the signal itself — using rubrics as accept/reject gates rather than dense rewards prevents the model from gaming them Can rubrics and dense rewards work together without hacking?. The throughline across all of it: reward whatever you want more of, and a model will give you exactly that — so if you reward confidence without rewarding being right about your confidence, you get a fluent, persuasive, and quietly unreliable system.
Sources 8 notes
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
TruthRL uses three distinct rewards (correct +1, hallucination -1, abstention intermediate) to make abstention learnable. Across four benchmarks, this reduced hallucinations by 28.9% and improved truthfulness by 21.1% compared to binary reward RL.
Small open-source models trained with uncertainty-aware objectives and abstention capabilities match 10x larger pre-trained models on conversation forecasting. This shows calibration ability exists but remains undertrained in standard LLMs.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
RLPR and INTUITOR successfully extend reinforcement learning for reasoning to general domains by using the model's own token probabilities and confidence levels as reward signals, eliminating the need for external verifiers or reference answers.
Training models to critique noisy responses outperforms training on correct answers because critique forces engagement with failure modes and structural reasoning. Even imperfect critique supervision beats correct-answer imitation, showing how weak surface-pattern learning is for building genuine understanding.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.