Does negative reinforcement alone outperform full reinforcement learning?
Can training with only penalty signals for wrong answers match or exceed full RL approaches? This challenges the conventional assumption that reward design requires both positive and negative signals.
Decomposing RL's learning signal into positive sample reinforcement (PSR) and negative sample reinforcement (NSR) reveals a surprising asymmetry. Training with only negative samples — penalizing incorrect responses without ever reinforcing correct ones — consistently improves performance over the base model across the entire Pass@k spectrum (k up to 256), often matching or surpassing full PPO and GRPO.
The mechanism is straightforward through gradient analysis: NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines existing knowledge rather than introducing entirely new behaviors. This is because penalizing a wrong answer doesn't point toward any specific correct answer — it lets the model's own prior determine where the freed probability mass flows.
Positive-only reinforcement creates the opposite problem. It improves Pass@1 (the model gets better at its top-ranked answer) but degrades performance at higher k because it concentrates probability mass on rewarded trajectories, reducing diversity. Since Does policy entropy collapse limit reasoning performance in RL?, positive reinforcement actively contributes to the problem while negative reinforcement sidesteps it.
This reframes how we think about RL for reasoning. The conventional framing is that RL rewards correct behavior. But the evidence suggests that penalizing incorrect behavior may contribute more to performance than reinforcing correct behavior — especially when diversity matters. The model already contains good solutions in its prior; it just needs help avoiding the bad ones.
The practical implication is that reward design for reasoning RL may be over-engineered. If suppression alone gets you most of the way, the elaborate reward shaping and process supervision architectures may be solving a problem that's already largely solved by the base model's prior distribution.
Inquiring lines that read this note 83
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can we build reliable evaluations of AI reasoning despite judge bias and reward-seeking? Do structural constraints outperform deep architectures in recommendation systems? How do pretraining biases affect reward signal effectiveness in RLVR?- Why does binary reward forcing degrade model calibration?
- Can checklist-based rewards fix judgment problems in RL training?
- Can log-likelihood loss combined with binary rewards achieve calibration?
- What information do next-state signals contain beyond what scalar rewards capture?
- How does forced exploration through diversity rewards differ from suppression-based negative reinforcement?
- How does negative reinforcement redistribute probability without guiding toward correct answers?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- Can model confidence signals replace explicit external reward functions?
- How do reward model biases cascade into downstream optimization failures?
- Why do spurious rewards work nearly as well as correct ones?
- What makes Effective Rank Acceleration a stable training signal for dual-channel incentives?
- When does outcome reward signal become informative during model training?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- Can binary judge feedback replace external reward signals for skill learning?
- How do reward signals in RLVR interact with pretraining biases?
- How does 93% reward reliability compare to other RL noise sources?
- Why does scalarization of rewards fail for multi-objective GRPO training?
- What happens when variance in reward signals comes from a noisy model?
- Can the same variance signal work as both reward and query filter?
- How do you extract reward signals when all rollouts fail?
- How do relational reward signals compare to absolute preference encodings in RL?
- Are different reward signal sources substitutable in verifier-free RL?
- Why do majority-vote rewards amplify errors below an accuracy threshold?
- Can early experience replace external rewards as a learning signal?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- Can tree-GRPO work with extremely noisy or sparse outcome reward signals?
- When does a task lack a meaningful multi-dimensional reward structure?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- How do reward models and self-improvement mechanisms interact in training?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- What makes current learned reward models fail across different domains?
- What behavioral changes occur during reward learning training?
- Can meta-reinforcement learning explain why this bias pattern emerges rationally?
- How does reinforcement learning compare to differentiable joint training for RAG?
- Can negative reinforcement alone match full RL performance on domain tasks?
- Does negative reinforcement alone achieve what full RL training accomplishes?
- What makes pretraining composition more important than reward engineering?
- What distinguishes high-signal prompts from low-signal ones in RL training?
- Does reinforcement learning require sufficient pretraining to be effective?
- Can pretraining alone achieve chess performance without reinforcement learning?
- Why do spurious reward signals improve reasoning for some pretrained models?
- Do outcome-only reward signals miss step-level errors that compound later?
- Why do different models respond differently to spurious rewards?
- Why do spurious rewards work for some models but not others?
- Can structured rewards still teach models when spurious rewards also work?
- What makes binary rewards more effective than richer reward signals?
- How do internal model mechanisms escape token-level reinforcement signals?
- What makes process-level supervision better than outcome-only reward signals?
- How do process-level rewards compare to environment-extracted next-state signals?
- How does process-based reward differ from outcome-only reward in training?
- How do evaluative versus directive signals differ in next-state training?
- What distinguishes inductive inference from negative evidence versus positive patterns?
- What makes content informative and not-yet-mastered for reinforcement during pretraining?
- What makes utility-weighted training backfire in machine learning systems?
- Can a rejected-edit buffer work like hard negatives in contrastive learning?
- What makes preventative lessons from failures more valuable than success patterns?
- How does active selection of training content differ from random reinforcement sampling?
- Can rejected edits serve as negative feedback like hard negatives in contrastive learning?
- Can negative feedback through critiques achieve the same steering flexibility as positive preferences?
- Can light human signals steer already-learned behavior without preference labels?
- What makes abstention a learnable behavior instead of a default penalty?
- Can abstention behavior transfer from small models to frontier models?
- Do negative constraints require fundamentally different training signals than positive instructions?
- Why do structure-targeted training negatives fail to fix the underlying problem?
- Why does negative experience transfer better than positive examples alone?
- Can we reverse the instruction-following deficit through targeted training?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
directly supports: pruning IS negative reinforcement at the reasoning path level
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
extends: positive reinforcement actively causes entropy collapse; negative reinforcement avoids it
-
Does reinforcement learning update only a small fraction of parameters?
Investigating whether RL algorithms consistently modify only 5–30% of model parameters across different LLMs and RL methods, and what structural properties those sparse updates possess.
complementary: if RL only touches 5-30% of parameters, negative reinforcement may be the primary mechanism for this sparse selection
-
Does the choice of RL algorithm actually matter for reasoning?
Expert Iteration, PPO, and RC-RL show similar performance on reasoning tasks. The question is whether algorithm choice drives results or whether something deeper—like the pretrained model itself—sets the real limits.
supports: if negative reinforcement alone suffices, algorithm choice matters even less
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
- A Survey of Reinforcement Learning from Human Feedback
- Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Efficient Reinforcement Learning via Large Language Model-based Search
- Reinforcement Learning with Rubric Anchors
Original note title
negative reinforcement alone matches or exceeds full rl by suppressing incorrect trajectories and redistributing probability mass