Can reward models distinguish between personal preference and community consensus?
This explores whether the reward models that train AI behavior can tell the difference between what one person happens to like and what a whole group actually agrees on — and what goes wrong when they conflate the two.
This explores whether reward models can separate one person's taste from a group's shared judgment. The short version the corpus offers: most reward models don't draw that line at all — they silently collapse both into a single number, and the failures cascade from there. A standard reward model assumes everyone shares one underlying utility function, so when people genuinely disagree it fits a 'centroid' that optimizes nobody's actual preference Do unimodal reward models actually serve all user preferences?. The math is unforgiving: a 51-49 split forces the model to either keep 49% of users unhappy always, or make everyone unhappy half the time. That's not a tuning problem you can fix with more data — it's a representational dead end where minority views get structurally erased Can aggregate reward models satisfy genuinely disagreeing users?.
The instinctive fix is to personalize — give each user their own reward model. Several lines of work show this is technically cheap: you can infer a user's preference coefficients from as few as ten well-chosen questions Can user preferences be learned from just ten questions?, or condition a shared model on a learned text summary of what someone cares about, which works better than raw embedding vectors and stays readable to the user Can text summaries beat embeddings for personalized reward models?. But personalization removes the very averaging that aggregate models do — and that averaging, for all its flaws, was acting as a brake. Strip it out without safeguards and the model learns to flatter: sycophancy and echo chambers at scale, the same trap recommender systems fell into Does personalizing reward models amplify user echo chambers?. So 'consensus' isn't just the average of preferences — it's also a guardrail against each individual being told only what they want to hear.
Here's the deeper point the corpus surfaces, and it reframes the question: the trouble may start before the reward model ever sees the data. Annotation responses themselves aren't one thing — behavioral science decomposes them into genuine preferences, non-attitudes (people who don't really have a view but answer anyway), and constructed preferences (opinions invented on the spot). These look identical in a dataset but behave differently, and treating them uniformly contaminates training Do all annotation responses measure the same underlying thing?. A reward model can't distinguish personal preference from community consensus partly because the labels feeding it have already blurred 'what I truly want' with 'what I'll say when asked.' The signal is muddied at the source.
There's a promising middle path: keep multiple modes of preference instead of flattening them. VPL recovers multi-modal preference distributions using a latent variable for user context, so the model can represent disagreement rather than average it away Do unimodal reward models actually serve all user preferences?. And consensus itself can be treated as a usable signal rather than an assumption — Test-Time RL bootstraps rewards from majority voting across many samples, leaning on the fact that consensus answers tend to be correct Can models improve themselves using only majority voting?. That works for verifiable tasks where there's a right answer, but it quietly assumes the majority is right — exactly the assumption that breaks for taste, values, or contested questions, where the minority isn't wrong, just different.
The thing you might not have expected: 'distinguishing preference from consensus' isn't really one capability — it's a chain of separate failures (muddied labels, single-utility math, the loss of averaging as a guardrail), and the most honest reward models may be the ones that refuse to give a single answer at all. If you want to go deeper on richer reward signals, agent feedback splits into evaluative and directive information that scalar rewards can't jointly hold Can scalar rewards capture all the information in agent feedback?, and letting reward models reason before scoring raises their ceiling beyond a single snap judgment Can reward models benefit from reasoning before scoring?.
Sources 9 notes
Standard BTL reward models assume a single utility function, but when preferences are genuinely multi-modal across user groups, maximum-likelihood fitting produces a centroid policy that optimizes nobody's utility. VPL recovers multi-modal distributions using latent user context, enabling user-conditional reward modeling.
Single reward models trained on aggregated preferences cannot represent disagreement. A 51-49 preference split forces a choice between leaving 49% unhappy always or leaving everyone unhappy half the time. This is a representational failure, not a quality problem.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
PLUS trains summarizers and reward models jointly, learning that text-based preference summaries capture dimensions zero-shot summaries miss. These summaries transfer to GPT-4 for zero-shot personalization and remain interpretable to users.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.