Can generative reasoning beat discriminative models with less training data?
Do process reward models that generate reasoning before judging achieve better performance than traditional discriminative approaches when trained on dramatically smaller datasets? This tests whether generative verification can scale more efficiently.
Process reward models (PRMs) are central to test-time scaling but face three limitations: limited generalization across models and tasks, dependence on scalar value prediction that ignores LLM generative abilities, and inability to scale test-time verification compute. Two converging approaches solve these by reframing process supervision as a generative task.
GenPRM integrates Chain-of-Thought reasoning and code verification before providing judgment for each reasoning step. Using Relative Progress Estimation (RPE) — a relative criterion for label estimation rather than hard labels — and a rationale synthesis framework with code verification, GenPRM achieves strong results with only 23K training examples from MATH. A 1.5B GenPRM outperforms GPT-4o on ProcessBench; a 7B version surpasses Qwen2.5-Math-PRM-72B.
ThinkPRM capitalizes on the inherent reasoning abilities of long CoT models, fine-tuning with as few as 8K synthetic verification chains. Using only 1% of the process labels in PRM800K, ThinkPRM outperforms LLM-as-a-Judge and discriminative verifiers across ProcessBench, MATH-500, and AIME '24. In out-of-domain evaluation (GPQA-Diamond, LiveCodeBench), it surpasses discriminative PRMs trained on the full PRM800K by 8% and 4.5% respectively.
The key structural advantage: generative PRMs uniquely support simultaneous scaling of both generator and verifier compute. Discriminative PRMs output a fixed scalar; generative PRMs can be forced to think longer, producing more thorough verification. Under the same token budget, ThinkPRM scales verification compute more effectively than LLM-as-a-Judge, outperforming it by 7.2% on ProcessBench.
Since Can judges that reason about reasoning outperform classifier rewards?, GenPRM and ThinkPRM provide the strongest evidence and specific mechanisms. Since Can reward models benefit from reasoning before scoring?, generative PRMs establish the paradigm: the verifier should think before judging, just as the generator should think before answering.
Inquiring lines that read this note 42
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do language models encode knowledge that influences generation, or primarily imitate surface patterns? Why does AI verification capability persistently exceed generation capability?- How does the expert demonstration ceiling compare to the generation-verification gap bound?
- Why does human validation become the bottleneck when AI generation scales?
- At what capability level does the generation-verification gap make intrinsic rewards insufficient?
- Does internalizing verifiers actually close the generation-verification gap?
- Why does the generation-verification gap disappear for factual recall tasks?
- Does the generation-verification gap actually limit self-improvement in verifiable tasks?
- Why does AI generation outpace verification across the research lifecycle?
- Does the verification gap widen exactly where judgment replaces checkability?
- Can automated tools close the gap between AI generation and verification?
- How does generation-verification asymmetry create the need for verifiable reporting?
- How does test-time verification decouple the act of checking from reasoning generation?
- Why can generative verifiers scale verification compute more effectively than fixed-output discriminative models?
- Can verification tools keep pace with AI artifact generation speed?
- Why does strengthening the judge improve the actor's generation performance?
- Does the generation-verification gap limit how far AI can improve itself?
- Where does the generation-verification gap appear in test-time compute?
- What structural changes help AI generation keep pace with verification?
- Why is verification harder than generation across the research lifecycle?
- Why does verification of AI work consistently lag behind AI generation?
- How does the generation-verification gap erode when verifier and generator are coupled?
- Why does verification take longer than generation across research workflows?
- How much does model verification capability exceed generation capability?
- How does the generation-verification gap change as models scale up?
- Why do generative reward models produce more interpretable evaluations than scalar scores?
- How do generative PRMs ensure their reasoning actually influences judgment instead of decorating outputs?
- What distinguishes generative reward models from outcome-based and process-based approaches?
- How much data do generative process reward models actually need?
- How do process reward models compare to token-level variance filtering?
- What are the actual limits of sibling comparison versus trained process reward models?
- Can process reward models reason before judging more data efficiently?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can judges that reason about reasoning outperform classifier rewards?
Can process reward models generate explanations about why steps are correct rather than simply classifying them? This explores whether meta-reasoning about reasoning improves both accuracy and generalization in step-level evaluation.
GenPRM/ThinkPRM provide the strongest implementations
-
Can reward models benefit from reasoning before scoring?
Does allowing evaluator models to generate reasoning traces before producing reward scores improve alignment and enable adaptive compute allocation? Three independent research teams converged on this insight simultaneously.
generative PRMs operationalize reward-compute scaling
-
Can self-supervised process rewards replace human annotation?
Self-supervised PRMs learn from outcome labels alone, avoiding expensive step-level annotation. The key question is whether this approach generalizes beyond math and code to domains with ambiguous correctness.
GenPRM's RPE and ThinkPRM's synthetic chains reduce annotation dependence
-
Does chain of thought reasoning actually explain model decisions?
When language models show their reasoning steps in agentic pipelines, does the quality of those steps predict or explain the quality of final outputs? This matters for trusting and debugging AI systems.
generative PRMs must ensure their CoT actually drives judgment, not just decorates it
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
- Process Reward Models That Think
- StepWiser: Stepwise Generative Judges for Wiser Reasoning
- Test-Time Scaling with Reflective Generative Model
- Reward Reasoning Model
- The Future of Facts: Tracing the Factual Generation-Verification Gap
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Reasoning Language Models: A Blueprint
Original note title
generative process reward models that reason before judging outperform discriminative prms with orders of magnitude less data