SYNTHESIS NOTE
Does RLVR actually expand what models can reason about?
Explores whether reinforcement learning from verifiable rewards teaches models genuinely new reasoning skills or simply makes existing capabilities more reliable. Pass@k analysis suggests the latter.
The strongest empirical challenge to the "RL teaches reasoning" narrative comes from pass@k analysis. At small k (e.g., k=1), RLVR models outperform their base models — they produce correct answers more reliably on any given attempt. But as k increases, base models consistently surpass RLVR models across all benchmarks and model families. The reasoning paths that RLVR models generate are already present in the base model's sampling distribution.
This reframes what RLVR actually does. Rather than expanding the frontier of solvable problems, RLVR narrows the sampling distribution toward correct solutions that were already accessible. The model learns to find correct paths more efficiently, not to reason in fundamentally new ways. Manual inspection confirms: for most problems where RLVR models succeed, the base model can produce at least one correct chain-of-thought.
Six popular RLVR algorithms (including GRPO, PPO variants) perform similarly and all remain far from optimal in leveraging the base model's potential — they converge on similar subsets of the base model's capability space. This suggests the bottleneck is not algorithmic but structural: on-policy RL with verifiable rewards optimizes sampling, not capability.
The contrast with distillation is sharp. Distillation from a stronger teacher can transfer genuinely new reasoning patterns, expanding the student's reasoning scope beyond what the base model could sample. Since Does RL teach reasoning or just when to use it?, the RLVR finding fits: activation is not creation. But distillation is creation — it writes new patterns into the model's distribution.
The practical implication: if you need capabilities the base model doesn't have, distillation from a stronger model is the path. If the base model can already solve the problem (given enough samples), RLVR makes it reliable. These are different tools for different gaps.
Inquiring lines that read this note 124
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI systems achieve real improvement without external human feedback?- Does good simulation eventually count as genuine realization?
- Can neural grafts reliably reveal hidden capabilities in AI models?
- Why does binary reward forcing degrade model calibration?
- Can checklist-based rewards fix judgment problems in RL training?
- Do spurious rewards activate reasoning without teaching new skills?
- How do probability-based rewards compare to self-consistency as training signals for reasoning?
- What distinguishes verifiable rewards from preference-based rewards in unified training?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- Can model confidence signals replace explicit external reward functions?
- Does RLVR reward structure create pressure toward traces that look right?
- Can reward design fix the conflict between reasoning accuracy and abstention calibration?
- When does outcome reward signal become informative during model training?
- What reward mechanisms make thinking-based compression budget-controllable and reliable?
- How does belief-shift reward compare to curiosity-driven and process reward approaches?
- Why does belief-shift reward enable smaller models to match larger baselines?
- How do reward signals in RLVR interact with pretraining biases?
- Why do reward models fail to recognize genuinely different valid answers?
- How does 93% reward reliability compare to other RL noise sources?
- Does outcome-based reinforcement learning improve explanation faithfulness?
- What other downstream metrics could serve as RL reward sources?
- Can verifiable rewards during pretraining replace costly human preference labeling?
- Are different reward signal sources substitutable in verifier-free RL?
- Can structured rewards still teach models when spurious rewards also work?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- What makes reward models fundamentally different from policy discriminators?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- What makes user-decision rewards better than model-confidence rewards?
- Why do dense rewards plus hard constraints outperform single fixed rewards?
- What makes current learned reward models fail across different domains?
- Can belief editing alone distinguish reward-optimization from instruction-following behavior?
- What makes some tasks bounded enough for reliable RL?
- Can RLVR expand a model's reasoning capabilities beyond its training ceiling?
- Why do current RLVR methods fail to expand reasoning capability beyond base model boundaries?
- Can reinforcement learning add missing domain knowledge to fine-tuned reasoning models?
- Can RL teach when to use reasoning versus when to respond directly?
- How does reinforcement learning compare to differentiable joint training for RAG?
- Does reinforcement learning learn optimal per-turn reasoning discipline?
- How do Q-value models improve action selection compared to value models?
- How does RL refine reasoning paths without simply adding model capability?
- Does RL refine existing knowledge or discover entirely new capabilities?
- Why does RL improve sampling efficiency but not expand capability boundaries?
- Can extended RL training unlock genuinely new reasoning strategies models cannot discover otherwise?
- Are RLVR models worse than non-reasoning models for subjective annotation?
- What limits RLVR effectiveness beyond mathematical and coding domains?
- Can models learn both what and how to study through reinforcement learning?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- Can reinforcement learning teach AI when to ask clarifying questions?
- Does RLVR expand model capability or reorganize existing capability?
- Does RL teach models when to use reasoning or how to reason?
- Why does prolonged RL discover strategies absent from any base model sample?
- How does Supervised RL bridge the gap between SFT and RLVR?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- Does RL amplify existing reasoning or create genuinely new computational strategies?
- Does reinforcement learning teach models how to reason or when to reason?
- How do verifier-free and adversarial approaches compare in extending reasoning RL?
- Can reinforcement learning close the gap between LLM reasoning and action?
- What does RL post-training actually teach reasoning systems?
- Can verifier-free RL work without manual preference labels or task-specific training?
- How can verifier-free reinforcement learning handle reasoning without task-specific checks?
- How does prolonged RL training differ from standard RLVR approaches?
- How do verifier-free RL patterns differ from traditional RLHF approaches?
- Does RLVR teach new reasoning or activate existing pretraining capabilities?
- Why does reinforcement learning training degrade model calibration?
- What makes a task at the edge of competence optimal for RL?
- Can RL create new reasoning primitives that pretraining never established?
- How do extrapolative and contextual generalization measure RL reasoning gains?
- When does reinforcement learning actually produce true reasoning gains in models?
- Why do reasoning gains from RL require models trained with headroom and edge-of-competence data?
- Can reinforcement learning add new capabilities or only remove inaccurate knowledge?
- What role does reinforcement learning play in optimizing inference compute?
- Can reinforcement learning improve how accurately models explain themselves?
- Does RL teach models new reasoning or just better timing?
- How does RLSVR differ from using model probability or self-judgment?
- Should larger compute budgets allocate more resources to reinforcement learning?
- Does the reinforcement learning improvement rate depend on model initialization?
- Can RL training on small verifiable tasks transfer to real-world AI research?
- What hard-to-verify tasks will remain resistant to reinforcement learning?
- How much RLVR improvement comes from benchmark data memorization?
- Can clean benchmarks reveal true RLVR reasoning gains?
- Can outcome-based rewards fully replace per-step likelihood in diffusion RL training?
- Why does a relativistic critic outperform absolute scoring in adversarial reasoning training?
- Can RL with verifiable rewards improve dialogue quality better than preference optimization?
- Can proper scoring rules fix RLVR's degradation on disagreement prediction?
- What scaling properties emerge from RL training dynamics beyond verification?
- What makes a sub-goal verifiable enough to provide dense feedback signals?
- What information do next-state signals contain beyond what scalar rewards capture?
- What makes exploration and reflection rewards verifiable in agentic environments?
- Can early experience replace external rewards as a learning signal?
- Why does early experience provide better warm-starts for downstream reinforcement learning?
- How does the pretrained prior set a capability ceiling for reward model exploration?
- What distinguishes RL that creates new capabilities from RL that merely teaches timing?
- How do self-evolving curricula help RL break beyond base model capability boundaries?
- What training duration is actually needed for RL to expand capabilities?
- What pretraining formats encode latent reasoning strategies that RLVR can surface?
- How does the expert demonstration ceiling compare to the generation-verification gap bound?
- At what capability level does the generation-verification gap make intrinsic rewards insufficient?
- Does the generation-verification gap actually limit self-improvement in verifiable tasks?
- What stability techniques prevent collapse in policy-critic adversarial training?
- Can UCB-style bonuses over outcome space prevent policy entropy collapse?
- What happens to model reasoning when policy entropy collapses during RL?
- What role do high-entropy minority tokens play in RLVR?
- Do high-entropy RLVR tokens correspond to MI-peak tokens during inference?
- How do dense token-level rewards compare to sparse task-level verification signals?
- Can process supervision improve agentic RL through meta-reasoning rewards?
- How much data do generative process reward models actually need?
- Why do certain tokens at certain difficulties drive most of RLVR's learning signal?
- How does off-policy data reuse inside trust regions affect convergence guarantees?
- Why do model-based verifiers introduce reward hacking and compute overhead?
- What role does verifier design play in reasoning capability gains?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does RLVR actually expand what models can reason…
Does RL teach reasoning or just when to use it?
Do base models already contain hidden reasoning ab…
Can reinforcement learning discover reasoning stra…
Can simple rewards alone teach complex domain reas…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does RL teach reasoning or just when to use it?
Does reinforcement learning in thinking models actually create new reasoning abilities, or does it simply teach existing capabilities when to activate? This matters for understanding where reasoning truly emerges.
RLVR confirms the timing-not-capability thesis with pass@k evidence
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
RLVR finding shows the latent capability is an upper bound, not a floor
-
Can reinforcement learning discover reasoning strategies base models cannot?
Does RL training truly expand what models can do, or does it just find solutions already hidden in base models? ProRL tests this by running RL longer and on diverse tasks beyond mathematics.
tension: this claims RL does expand boundaries under prolonged training
-
Can simple rewards alone teach complex domain reasoning?
Does reinforcement learning on difficult problems with basic accuracy rewards produce sophisticated reasoning strategies without explicit chain-of-thought training? This challenges assumptions about what domain AI models need to learn effectively.
emergence may operate at a different level than sampling efficiency
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data
- Escaping the Verifier: Learning to Reason via Demonstrations
- Spurious Rewards: Rethinking Training Signals in RLVR
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
Original note title
rlvr does not expand reasoning capability boundaries beyond the base model — it improves sampling efficiency within existing boundaries