Can UCB-style bonuses over outcome space prevent policy entropy collapse?
This explores whether borrowing the bandit idea of an exploration bonus — rewarding under-visited answers (UCB-style) — could keep an RL-trained reasoning model from collapsing into a narrow, overconfident output distribution; the corpus doesn't test that exact mechanism, but it maps the problem and several adjacent fixes.
This explores whether a UCB-style exploration bonus over the space of possible outcomes could stop policy entropy collapse. None of the corpus papers run that exact experiment, but together they explain why the question matters and hint at why an outcome-space bonus alone might not be enough. The anchor is Does policy entropy collapse limit reasoning performance in RL?, which shows performance in reasoning RL follows a clean empirical law — gains saturate as policy entropy drains toward zero. The fixes that work there (Clip-Cov, KL-Cov, GPPO) are all entropy-management techniques: they constrain *how fast* the policy is allowed to sharpen, rather than handing out a bonus for novel outcomes. That's a quiet signal that the field's current best answers operate on the gradient/update side, not by adding a UCB term over answers.
Why might an outcome-space bonus underdeliver? Look at Does RLVR actually expand what models can reason about?. Its pass@k analysis shows RLVR doesn't add new solvable problems — it just concentrates probability mass on solutions the base model could already reach. If entropy collapse is really the policy *narrowing toward what it already knows*, then rewarding rare outcomes risks chasing exploration into regions the model can't actually solve. A UCB bonus rewards novelty whether or not novelty is useful; in a verifiable-reasoning setting, most novel outcomes are simply wrong. So the bonus could preserve entropy while degrading accuracy — exploration for its own sake.
There's also a calibration trap worth knowing about. Does binary reward training hurt model calibration? shows that binary correctness rewards actively *push* models toward high-confidence guessing, because nothing penalizes a confident wrong answer — which is a direct driver of the overconfident, low-entropy collapse you'd be trying to fight. Their fix isn't an exploration bonus at all; it's adding a proper scoring rule (Brier score) as a second reward term that mathematically couples accuracy and calibration. That suggests the leverage point is the *reward's information content*, not a count-based novelty bonus bolted on top.
The more interesting lateral move in the corpus is that several papers attack the same collapse by enriching the learning signal rather than the exploration term. Can natural language feedback overcome numerical reward plateaus? shows models stuck on a plateau (a collapse symptom) break free when given chain-of-thought critiques — because numerical rewards lack the information about *why* a failure happened. Can scalar rewards capture all the information in agent feedback? makes the structural version of the argument: feedback carries both evaluative and directive information, and scalar rewards throw the directive half away. And Can an agent's own beliefs guide credit assignment without critics? offers a dense intrinsic reward built from the agent's own shifting beliefs — a per-step signal that keeps learning alive without a critic. These all point to the same conclusion: entropy collapse is downstream of *thin* reward signals, and the corpus's bet is on denser, more directional feedback rather than count-based outcome bonuses.
So the honest answer: a UCB-style bonus is a plausible lever on the entropy side of Does policy entropy collapse limit reasoning performance in RL?'s law, but the corpus's accumulated evidence suggests it would treat a symptom. The papers that actually move plateaus do it by giving the policy richer reasons to update, not more reasons to wander — and the calibration result warns that naive outcome bonuses can preserve entropy while quietly rewarding confident nonsense.
Sources 6 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.