SYNTHESIS NOTE
Does RLVR success on math benchmarks reflect genuine reasoning improvement?
Explores whether RLVR's apparent effectiveness with spurious rewards on contaminated benchmarks like MATH-500 represents actual reasoning gains or merely data memorization retrieval.
The apparent success of RLVR with random, incorrect, or spurious reward signals on Qwen models may be an artifact of data contamination rather than evidence of genuine reasoning improvement.
The contamination evidence: prompting Qwen2.5-Math-7B with the first 60% of each MATH-500 question yields 54.6% exact-match reconstruction of the remaining 40% and 53.6% correct answers to these incomplete problems. On LiveMathBench — a benchmark released after Qwen2.5 — completion rate drops to 0.0%, consistent with Llama3.1-8B (3.8%/0.0% respectively). The model has memorized MATH-500.
On a fully clean benchmark (RandomCalculation — synthetic arithmetic expressions generated after Qwen's release): correct rewards deliver consistent gains surpassing the model's performance ceiling; random rewards make training highly unstable with no reliable improvement; inverse rewards rapidly erode mathematical reasoning ability.
This directly challenges Why do random rewards improve reasoning for some models but not others?. The prior interpretation — that any optimization pressure activates pretraining strategies — may confound two effects: genuine strategy activation (possible) and recall of memorized answers triggered by format-similar optimization (likely for contaminated benchmarks). On clean data, the "any reward works" finding evaporates for random and inverse signals.
The practical implication: RLVR research conclusions drawn from MATH-500 and similar benchmarks for Qwen models should be interpreted with caution. Reward engineering may matter more than the spurious-reward literature suggests — we were measuring memorization recovery, not reasoning improvement.
Inquiring lines that read this note 89
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What explains the gap between benchmark scores and true reasoning capability?- How much RLVR improvement comes from benchmark data memorization?
- Can clean benchmarks reveal true RLVR reasoning gains?
- Does the Heuristic Override Benchmark measure enumeration or world knowledge?
- Can high test performance mask a complete absence of understanding?
- Do current math benchmarks measure outcomes or rhetorical plausibility?
- Why do benchmark scores rise while reasoning quality declines?
- How does tool access change what we measure in reasoning tests?
- What is the gap between benchmark performance and real workplace task completion?
- How much of MATH-500 improvement comes from data contamination versus real reasoning gains?
- Can benchmark scores on verifiable tasks transfer to unseen problems outside the training domain?
- Why do benchmarks become saturated so quickly after initial launch?
- How much improvement comes from caching versus actual capability gain?
- Can an average-case validator score hide poor performance on critical tasks?
- Why do hidden test partitions matter more than open evaluation sets?
- Can memorization inflate apparent capability on benchmarks with available solutions?
- Does measured performance gain reflect true task improvement or evaluator exploitation?
- What distortions do automated benchmarks introduce compared to real tasks?
- What biases affect how we measure progress on research leaderboards?
- What makes the Brier score mathematically better than log-likelihood here?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Why do six different RLVR algorithms converge on similar performance levels?
- Can defenders tighten the total-variation bound in practice with measured benign activation rates?
- Why do benchmark designers treat content effects as confounds?
- What reporting standards would make interactive evaluation scores comparable across benchmarks?
- What capability dimension does a closed-ended exam actually fail to measure?
- Why does a series of improving scores differ from a single score rise?
- How should benchmarks balance verifiability against outcome resolution?
- When does measured progress on an evaluator conceal actual performance decline?
- Why does adopting benchmarks one at a time produce non-comparable scores?
- What specific benchmarks show wins versus ties in the equals-or-surpasses claim?
- Can infrastructure records restore meaning to a single benchmark score?
- What would a diagnosable evaluation look like compared to a scalar score?
- How much does medal age matter when predicting Kaggle performance?
- Do medals from retired competition formats lose predictive power faster than others?
- Does this stranding problem appear when other evaluation platforms retire their formats?
- How would longitudinal measurement reveal sustained effects of friction?
- Why did the PR lift persist longer than in comparable studies?
- How should researchers document validity gaps in their benchmarks?
- How do statistical noise and revalidation bias speedrun benchmark claims?
- Why do cumulative-best reporting inflate progress compared to actual validation?
- How do surface correlations between narratives and answers mislead benchmark validity?
- How do open-world evaluations correct distortions that automated benchmarks introduce?
- Can automated benchmarks accurately capture progress on real-world long-horizon tasks?
- Can expert-frontier exams discriminate frontier capability better than saturated benchmarks?
- Can a single benchmark score capture both progress and readiness?
- Can solution traces substitute for process-level reward signals in math reasoning?
- How can we measure whether process rewards actually align with reasoning quality?
- How do partial credit grading systems accidentally reward reasoning theater?
- What information do numerical rewards fail to provide for reasoning tasks?
- Does RLVR reward structure create pressure toward traces that look right?
- How does 93% reward reliability compare to other RL noise sources?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- Are RLVR models worse than non-reasoning models for subjective annotation?
- What limits RLVR effectiveness beyond mathematical and coding domains?
- Does RLVR expand model capability or reorganize existing capability?
- Can one training example activate mathematical reasoning in RL-trained models?
- Why does medium difficulty outperform both easy and hard RLVR training samples?
- Does RLVR teach new reasoning or activate existing pretraining capabilities?
- Can combining SRL with RLVR outperform either method used alone?
- How does RLSVR differ from using model probability or self-judgment?
- Can a single correct example seed exponential improvement in mathematical reasoning?
- How does RPT compare to learning when versus how to deploy reasoning?
- Can trustworthy scoring prevent persistent iteration from compounding errors?
- Can win rates alone measure whether a move is genuinely better?
- What evidence should benchmark operators attach to completion claims?
- Can infrastructure evidence ground benchmark claims better than terminal scores alone?
- Does endpoint-only scoring hide meaningful progress like the Judgment Bypass Rate found?
- How do test harnesses guide reflection better than transcripts alone?
- What distinguishes authentic consistency from Hawthorne-effect confounds in benchmarks?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does RLVR success on math benchmarks reflect gen…
Why do random rewards improve reasoning for some m…
Why does RLVR work with completely random rewards?
Does RLVR actually expand what models can reason a…
How much of LLM few-shot ability comes from traini…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Why do random rewards improve reasoning for some models but not others?
When RLVR training uses meaningless reward signals, some models gain reasoning improvements while others don't. What determines which models can benefit from optimization pressure without meaningful feedback?
directly challenged: the "code reasoning" activation story may be contamination-assisted memorization recall
-
Why does RLVR work with completely random rewards?
RLVR improves reasoning performance even with incorrect or random reward signals. This challenges the assumption that reward quality determines learning outcomes and raises questions about what RLVR is actually doing.
writing angle that needs qualification: the reward may matter after all, on clean benchmarks
-
Does RLVR actually expand what models can reason about?
Explores whether reinforcement learning from verifiable rewards teaches models genuinely new reasoning skills or simply makes existing capabilities more reliable. Pass@k analysis suggests the latter.
consistent: RLVR narrows rather than expands, and contamination inflates apparent gains
-
How much of LLM few-shot ability comes from training data?
Do large language models genuinely learn from a few examples, or are they mostly recognizing patterns from their training data? This matters for understanding what LLMs can actually do.
broader contamination phenomenon: RLVR contamination is benchmark-specific memorization; task contamination challenges the entire few-shot evaluation paradigm
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- Spurious Rewards: Rethinking Training Signals in RLVR
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- LLMs can implicitly learn from mistakes in-context
- Escaping the Verifier: Learning to Reason via Demonstrations
Original note title
RLVR effectiveness on contaminated benchmarks is primarily data memorization — clean benchmarks eliminate spurious reward gains