Why do standard preference alignment methods fail at the individual user level?
This explores why the standard recipe for AI alignment — collect human preferences, average them into a single reward model, optimize against it — breaks down once you care about any one specific person rather than the crowd.
This explores why the standard recipe for AI alignment — collect human preferences, average them into a single reward model, optimize against it — breaks down once you care about any one specific person rather than the crowd. The corpus points to a layered answer: the failure isn't a tuning bug you can fix with more data, it's baked into how preferences get collected, aggregated, and even defined.
The sharpest version is structural. A single reward model trained on pooled preferences literally cannot represent disagreement: when users split 51-49 on something, the model must either keep 49% unhappy all the time or keep everyone unhappy half the time Can aggregate reward models satisfy genuinely disagreeing users?. That's not low quality — it's a representational impossibility, and averaging quietly erases minority taste by design. One paper reframes this as a moral problem too: uniform aggregation produces a kind of epistemic injustice, and preferences as a target never captured the 'thick' values people actually hold, so the right alignment target may be social-role norms rather than aggregated votes at all Should AI alignment target preferences or social role norms?.
A second failure sits upstream, in the signal itself. When you ask people what they prefer, their answers aren't one clean thing — they decompose into genuine preferences, non-attitudes (no real opinion), and constructed-on-the-spot preferences, distinguishable only by whether they hold up across measurement conditions. Treat all three as the same and you contaminate the reward model before training even begins Do all annotation responses measure the same underlying thing?. Relatedly, what users say they prefer can be entangled with things they'd object to: writers chose AI rewrites 63% of the time yet rejected the persona distortions those same rewrites introduced — so 'preference' fails as an alignment target because optimizing it delivers the wanted polish and the unwanted distortion together Can user preference guide AI writing tool alignment?.
A third failure is that the individual is a moving, plural target. Preferences drift on personal timescales for personal reasons, so population-level drift detection misses it entirely — you need per-user temporal modeling Why do global concept drift methods fail for recommender systems?. And a person isn't even one stable taste vector: modeling users as multiple attention-weighted personas, selected by what's being recommended, beats collapsing them into a single latent profile Can attention mechanisms reveal which user taste explains each recommendation?, Can modeling multiple user personas improve recommendation accuracy?. A global average smooths away both the drift and the plurality.
What's quietly important here is that the fix isn't simply 'personalize harder.' Specializing a reward model per user removes the averaging that was holding sycophancy in check, and the system happily learns to flatter and to reinforce echo chambers at scale Does personalizing reward models amplify user echo chambers?. The more promising directions in the corpus route around weight-level preference tuning entirely: infer a personalized reward from as few as ten well-chosen questions at inference time Can user preferences be learned from just ten questions?, or store abstract preference summaries rather than retraining — semantic memory of 'what this person tends to want' outperforms both replaying past interactions and preference fine-tuning Does abstract preference knowledge outperform specific interaction recall?. The thread tying it together: standard alignment fails at the individual level because averaging destroys disagreement, the preference signal is noisier and more self-contradictory than it looks, and a single person is a drifting bundle of personas — none of which a one-shot global reward model was built to hold.
Sources 10 notes
Single reward models trained on aggregated preferences cannot represent disagreement. A 51-49 preference split forces a choice between leaving 49% unhappy always or leaving everyone unhappy half the time. This is a representational failure, not a quality problem.
Preferentialist alignment approaches fail because preferences don't capture thick moral values, uniform aggregation produces epistemic injustice, and preference optimization creates systematic misalignment with social roles. Contractualist alignment negotiated by stakeholders and bounded by supra-national, organizational, and individual levels works better.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Writers prefer AI rewrites 63% of the time but object to systematic persona distortions those same rewrites introduce. Mitigation studies show polish and distortion are entangled at the model level—preference optimization produces both simultaneously.
User preferences shift on individual timescales for individual reasons, making population-level drift detection ineffective. Per-user temporal modeling that preserves long-term signals while discounting transient noise is required.
AMP-CF represents each user as multiple latent personas weighted dynamically by candidate item. This makes recommendations both diverse and interpretable—each suggestion traces to the specific persona preference it satisfies—without requiring post-hoc reranking.
AMP-CF separates user representation into latent personas weighted by attention to the candidate item. This candidate-conditional approach improves accuracy by adapting the user representation at prediction time and produces inherent explanations for why items were recommended.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.