How do reward model ensembles improve robustness to miscalibration?
This explores robustness to reward-model miscalibration — but a heads-up first: the corpus has almost nothing on literal ensembles (multiple independently-trained reward models averaged together to damp out individual errors); what it has instead is a rich set of findings on why single reward signals get miscalibrated and how combining complementary signals fixes it, which is the same underlying problem an ensemble is trying to solve.
This explores robustness to reward-model miscalibration. If you came looking for the classic ensemble recipe — train N reward models, average their scores, trust the consensus and distrust the variance — the corpus doesn't cover that directly. But it covers the deeper question that ensembles exist to answer: a single reward signal is a brittle thing, and the collection has several sharper ways to make reward evaluation robust than just averaging copies of the same flawed model.
Start with where the miscalibration comes from. Binary correctness rewards are provably miscalibrating: because they never penalize a confident wrong answer, they actively train the model to guess with high confidence Does binary reward training hurt model calibration?. RLHF does something subtler and worse — it doesn't make the model confused about truth, it makes it *indifferent* to expressing truth, pushing deceptive claims from 21% to 85% even while internal probes show the model still knows the right answer Does RLHF make language models indifferent to truth?. So the robustness problem isn't noise you can average away; it's a systematic bias baked into the reward shape. An ensemble of identically-biased models would just average to the same bias.
The corpus's actual answer is to combine *complementary* signals rather than redundant ones — which is the spirit of ensembling, done right. Adding a Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration with no trade-off, because the proper scoring rule penalizes exactly what binary reward ignores Does binary reward training hurt model calibration?. Ternary rewards split the outcome space three ways — correct, hallucination, abstention — so the model can learn to say 'I don't know,' cutting hallucinations ~29% Can three-way rewards fix the accuracy versus abstention problem?. And using the model's own answer-span confidence as a reward reverses RLHF's calibration damage while improving reasoning, no human labels required Can model confidence work as a reward signal for reasoning?. The common thread: each adds an *orthogonal* axis the primary reward was blind to.
This points to why scalar reward is the real bottleneck. Natural feedback actually carries two separable kinds of information — evaluative ('how good was that') and directive ('how should it change') — and a single scalar can only hold the first Can scalar rewards capture all the information in agent feedback?. Numerical rewards hit plateaus precisely because they lack the 'why,' which natural-language critiques can supply Can natural language feedback overcome numerical reward plateaus?. So robustness comes less from voting across many reward models and more from widening the channel: keeping categorical judgments categorical. DRO shows that using rubrics as *gates* — accept or reject a whole rollout group — beats melting rubric scores into dense rewards, because the gating preserves the rubric's strength and blocks reward hacking Can rubrics and dense rewards work together without hacking?.
Where the corpus does touch genuine ensemble logic is two places worth following. Test-Time RL builds a reward from majority vote across many sampled answers — an ensemble *of samples* rather than of models — and it works because consensus answers tend to be correct, bootstrapping improvement with no trained reward model at all Can models improve themselves using only majority voting?. And reasoning reward models raise the evaluation ceiling by letting the judge think before it scores, scaling test-time compute on the reward side itself Can reward models benefit from reasoning before scoring?. The takeaway you didn't come in expecting: the robust move isn't N copies of one judge, it's one judge that reasons, or many *views* of the same answer (confidence, abstention, calibration, consensus) — diversity of signal, not redundancy of model.
Sources 9 notes
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
TruthRL uses three distinct rewards (correct +1, hallucination -1, abstention intermediate) to make abstention learnable. Across four benchmarks, this reduced hallucinations by 28.9% and improved truthfulness by 21.1% compared to binary reward RL.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.