How do different training objectives shift whether models over-predict or under-predict?

This explores how the choice of training objective tilts a model toward over- or under-predicting — and the corpus's sharpest claim is that the direction of the error is a fingerprint of which objective dominated training, not a single dial you tune. The clearest case: models trained for reasoning learn to *over-answer*, because abstaining ('I don't know') is never rewarded, while models trained for safety learn to *over-abstain*, refusing even harmless questions Does training objective determine which direction models fail at abstention?. Same architecture, opposite failure — and the deciding factor is what the reward signal valued. That reframes calibration from 'a bug to fix' into 'a characteristic signature you inherit from your training recipe.'

The mechanism behind over-prediction shows up most cleanly in reward design. Binary correctness rewards — right answer good, wrong answer bad — quietly teach the model to guess confidently, because a confident wrong answer costs exactly the same as a hesitant one. There's no penalty for bravado. Adding the Brier score (which scores not just whether you're right but how confident you should have been) provably restores calibration without sacrificing accuracy Does binary reward training hurt model calibration?. So the over-prediction isn't the model being reckless — it's the objective failing to price uncertainty.

Push the reward in the wrong way and the bias gets worse rather than better. Training on problems that are nearly impossible for the model warps it toward degenerate shortcuts — repeating answers, skipping computation — because the rare accidental success gets treated as a high-value trajectory worth reinforcing Do overly hard RLVR samples actually harm model capabilities?. And there's a subtler twist for decision-focused training: weighting the loss by how costly a mistake is (asymmetric loss) does sharpen the model's *choices*, but it weakens the underlying *learning* by muting the gradient signal for genuine feature acquisition. The counterintuitive fix is to train with a plain symmetric objective and then adjust the predictions afterward — separating 'learn the world' from 'decide what to do' beats baking the bias into the weights Can utility-weighted training loss actually harm model performance?.

That 'adjust afterward' theme runs deeper than calibration. Several notes converge on the idea that *how aggressively* an objective rewrites the base model determines how distorted its predictions become. RL post-training tends to collapse onto a single dominant output format from pretraining within the first epoch, suppressing alternatives the model could have used Does RL training collapse format diversity in pretrained models?. Direct fine-tuning corrupts knowledge stored in lower layers, whereas decoding-time proxy tuning closes most of the alignment gap while leaving those weights — and the knowledge — intact Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Staying close to the base distribution (low KL drift) even preserves the model's ability to keep learning later Does staying close to the base model preserve learning ability?. The pattern: objectives that yank the model far from its starting point trade prediction quality and plasticity for whatever they're optimizing.

The thing you didn't know you wanted to know: over- and under-prediction aren't really about the model being overconfident or timid — they're about what the objective forgot to penalize. A reasoning reward forgot to value silence; a binary reward forgot to value calibrated doubt; a utility-weighted loss forgot that good decisions need good representations underneath. Once you see calibration failures as the negative space of the reward function, the fix is usually to add the missing term (Brier score) or move the adjustment out of training entirely (proxy tuning, post-hoc correction) rather than to train harder on the same lopsided signal.