Does RLVR success on math benchmarks reflect genuine reasoning improvement?
Explores whether RLVR's apparent effectiveness with spurious rewards on contaminated benchmarks like MATH-500 represents actual reasoning gains or merely data memorization retrieval.
The apparent success of RLVR with random, incorrect, or spurious reward signals on Qwen models may be an artifact of data contamination rather than evidence of genuine reasoning improvement.
The contamination evidence: prompting Qwen2.5-Math-7B with the first 60% of each MATH-500 question yields 54.6% exact-match reconstruction of the remaining 40% and 53.6% correct answers to these incomplete problems. On LiveMathBench — a benchmark released after Qwen2.5 — completion rate drops to 0.0%, consistent with Llama3.1-8B (3.8%/0.0% respectively). The model has memorized MATH-500.
On a fully clean benchmark (RandomCalculation — synthetic arithmetic expressions generated after Qwen's release): correct rewards deliver consistent gains surpassing the model's performance ceiling; random rewards make training highly unstable with no reliable improvement; inverse rewards rapidly erode mathematical reasoning ability.
This directly challenges Why do random rewards improve reasoning for some models but not others?. The prior interpretation — that any optimization pressure activates pretraining strategies — may confound two effects: genuine strategy activation (possible) and recall of memorized answers triggered by format-similar optimization (likely for contaminated benchmarks). On clean data, the "any reward works" finding evaporates for random and inverse signals.
The practical implication: RLVR research conclusions drawn from MATH-500 and similar benchmarks for Qwen models should be interpreted with caution. Reward engineering may matter more than the spurious-reward literature suggests — we were measuring memorization recovery, not reasoning improvement.
Inquiring lines that read this note 69
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do capability benchmark scores systematically misrepresent true model abilities?- How much RLVR improvement comes from benchmark data memorization?
- Why do benchmark designers treat content effects as confounds?
- Can high test performance mask a complete absence of understanding?
- How much of MATH-500 improvement comes from data contamination versus real reasoning gains?
- What capability dimension does a closed-ended exam actually fail to measure?
- Can benchmark scores on verifiable tasks transfer to unseen problems outside the training domain?
- Why do benchmarks become saturated so quickly after initial launch?
- How much improvement comes from caching versus actual capability gain?
- Can an average-case validator score hide poor performance on critical tasks?
- How should benchmarks balance verifiability against outcome resolution?
- Why do hidden test partitions matter more than open evaluation sets?
- Can expert-frontier exams discriminate frontier capability better than saturated benchmarks?
- When does measured progress on an evaluator conceal actual performance decline?
- Why does adopting benchmarks one at a time produce non-comparable scores?
- What specific benchmarks show wins versus ties in the equals-or-surpasses claim?
- Can infrastructure records restore meaning to a single benchmark score?
- Can memorization inflate apparent capability on benchmarks with available solutions?
- Does measured performance gain reflect true task improvement or evaluator exploitation?
- What distortions do automated benchmarks introduce compared to real tasks?
- Can clean benchmarks reveal true RLVR reasoning gains?
- Does the Heuristic Override Benchmark measure enumeration or world knowledge?
- How does tool access change what we measure in reasoning tests?
- What is the gap between benchmark performance and real workplace task completion?
- What makes the Brier score mathematically better than log-likelihood here?
- Do current math benchmarks measure outcomes or rhetorical plausibility?
- What reporting standards would make interactive evaluation scores comparable across benchmarks?
- Why does a series of improving scores differ from a single score rise?
- What would a diagnosable evaluation look like compared to a scalar score?
- How do surface correlations between narratives and answers mislead benchmark validity?
- How do open-world evaluations correct distortions that automated benchmarks introduce?
- Can automated benchmarks accurately capture progress on real-world long-horizon tasks?
- Can solution traces substitute for process-level reward signals in math reasoning?
- How can we measure whether process rewards actually align with reasoning quality?
- How do partial credit grading systems accidentally reward reasoning theater?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Can defenders tighten the total-variation bound in practice with measured benign activation rates?
- Does RLVR reward structure create pressure toward traces that look right?
- How does 93% reward reliability compare to other RL noise sources?
- Why do six different RLVR algorithms converge on similar performance levels?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- Are RLVR models worse than non-reasoning models for subjective annotation?
- What limits RLVR effectiveness beyond mathematical and coding domains?
- Does RLVR expand model capability or reorganize existing capability?
- Can one training example activate mathematical reasoning in RL-trained models?
- Why does medium difficulty outperform both easy and hard RLVR training samples?
- Does RLVR teach new reasoning or activate existing pretraining capabilities?
- Can combining SRL with RLVR outperform either method used alone?
- How does RLSVR differ from using model probability or self-judgment?
- Can a single correct example seed exponential improvement in mathematical reasoning?
- How does RPT compare to learning when versus how to deploy reasoning?
- What pretraining formats encode latent reasoning strategies that RLVR can surface?
- Why do benchmark scores rise while reasoning quality declines?
- Can mathematical reasoning improvements transfer across problem subdomains?
- Can contamination-free evaluation distinguish between memorization and genuine prediction ability?
- Can trustworthy scoring prevent persistent iteration from compounding errors?
- How do test harnesses guide reflection better than transcripts alone?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do random rewards improve reasoning for some models but not others?
When RLVR training uses meaningless reward signals, some models gain reasoning improvements while others don't. What determines which models can benefit from optimization pressure without meaningful feedback?
directly challenged: the "code reasoning" activation story may be contamination-assisted memorization recall
-
Why does RLVR work with completely random rewards?
RLVR improves reasoning performance even with incorrect or random reward signals. This challenges the assumption that reward quality determines learning outcomes and raises questions about what RLVR is actually doing.
writing angle that needs qualification: the reward may matter after all, on clean benchmarks
-
Does RLVR actually expand what models can reason about?
Explores whether reinforcement learning from verifiable rewards teaches models genuinely new reasoning skills or simply makes existing capabilities more reliable. Pass@k analysis suggests the latter.
consistent: RLVR narrows rather than expands, and contamination inflates apparent gains
-
How much of LLM few-shot ability comes from training data?
Do large language models genuinely learn from a few examples, or are they mostly recognizing patterns from their training data? This matters for understanding what LLMs can actually do.
broader contamination phenomenon: RLVR contamination is benchmark-specific memorization; task contamination challenges the entire few-shot evaluation paradigm
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- Spurious Rewards: Rethinking Training Signals in RLVR
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- LLMs can implicitly learn from mistakes in-context
- Escaping the Verifier: Learning to Reason via Demonstrations
Original note title
RLVR effectiveness on contaminated benchmarks is primarily data memorization — clean benchmarks eliminate spurious reward gains