Can adaptive guidance from solution traces reduce reward sparsity in RL?

RLVR faces a capacity-difficulty mismatch: when training data complexity outpaces the model's current capabilities, all rollout responses are incorrect, producing zero advantage and vanishing policy gradients. This creates two compounding problems. Training inefficiency — computational effort on failed rollouts is entirely wasted. Training instability — the number of "effective" queries fluctuates dramatically between updates, injecting noise into gradient estimates.

GHPO (Guided Hybrid Policy Optimization) addresses this by conditioning the model on partial ground-truth solution traces, steering its output distribution closer to correct answers and alleviating reward sparsity. The key insight: solution traces are available for most math training data but are typically ignored during RLVR in favor of final-answer-only verification.

The framework dynamically balances two learning modes. For problems the model can likely solve, GHPO uses standard on-policy RL — encouraging exploration and self-discovery. For harder problems beyond current capability, it provides explicit solution traces — a form of imitation learning. The transition is adaptive: difficulty assessment determines how much guidance each problem receives.

This achieves approximately 5% performance gain across six mathematics benchmarks, consistently outperforming both standard RL and curriculum learning baselines. The improvement is particularly significant for smaller, resource-efficient LLMs where the capacity-difficulty mismatch is most acute.

Since Does gradually tightening token budgets beat fixed budget training?, GHPO provides the mechanism for curriculum adaptation — the guidance level is the curriculum variable. Since Can curriculum learning approximate expensive process supervision?, GHPO offers the complementary approach: instead of starting near the solution and backing up, it provides partial traces and lets the model complete them.

The practical lesson: RLVR training wastes substantial compute on problems the model cannot currently solve. Providing adaptive guidance for those problems — using solution traces that already exist in the training data — converts wasted compute into learning signal.

Source: RLVR