Can alignment procedures be redesigned to serve multiple preference groups?
This explores whether alignment methods like RLHF and DPO — which tend to optimize toward one aggregate preference — can be rebuilt to fairly serve groups with different values, dialects, and cultures.
This explores whether alignment methods like RLHF and DPO can be redesigned to serve groups with different values rather than collapsing everyone into one average preference. The first thing the corpus makes clear is that today's disparities are not accidents of nature — they're design choices. Standard RLHF and DPO create measurable gaps across English dialects and global opinions, and those gaps trace back to who gets picked as an annotator and how the task is defined, not to anything inevitable about the method How does LLM alignment affect representation across dialects?. If the problem is design, then redesign is on the table.
But redesign hits a deeper wall: the assumption that a single preference signal even exists to optimize. When you treat all the thumbs-up/thumbs-down data as measuring one thing, you contaminate the reward model — annotations actually decompose into genuine preferences, throwaway 'non-attitudes,' and preferences people construct on the spot, and these need different handling Do all annotation responses measure the same underlying thing?. Worse, even when people clearly prefer an output, optimizing for that preference can smuggle in harms they'd reject: writers picked AI rewrites 63% of the time yet objected to the persona distortions baked into those same rewrites, because polish and distortion are entangled at the model level Can user preference guide AI writing tool alignment?. Preference, in other words, is a noisy and sometimes self-contradictory target — which is exactly why serving multiple groups isn't just a sampling fix.
The most direct answer in the corpus argues the framing itself should change: stop aggregating preferences and instead align to the normative standards appropriate to a social role, negotiated by stakeholders and bounded at supra-national, organizational, and individual levels Should AI alignment target preferences or social role norms?. This 'contractualist' move is interesting because it sidesteps the impossibility of fairly averaging incompatible values — it lets different groups, roles, and jurisdictions hold different standards rather than fighting over one global setting. The cross-cultural evidence supports the pessimism about any single policy: alignment effects are documented almost entirely in WEIRD (Western) samples, and communication norms vary enough that one alignment policy is unlikely to land uniformly across cultures Does linguistic alignment work the same way across cultures?.
There's also a quieter, more technical line of evidence that multi-group alignment is feasible, because researchers are already learning to align along separable axes rather than one blob. Alignment 'dimensions' aren't interchangeable — lexical alignment buys task efficiency while emotional and prosodic alignment build trust, and conflating them produces category errors like cold service bots Do different types of alignment serve different conversational goals?. Methods are getting more surgical too: segment-level DPO beats turn- and session-level approaches by optimizing the right slice of a conversation Does segment-level optimization work better for multi-turn dialogue alignment?, and counterfactual-invariance training can produce agents that genuinely weigh a partner's input by causal impact instead of steamrolling it Why do standard alignment methods ignore partner interventions?. If alignment can be decomposed and targeted this precisely, serving distinct groups becomes a question of which axes to tune for whom.
The encouraging twist for anyone worried this requires impossibly large pluralistic datasets: LIMA showed that 1,000 carefully curated examples can match models trained on orders of magnitude more data, because post-training activates capabilities the model already has rather than building them Can careful curation replace massive alignment datasets?. That suggests bespoke alignment for a specific community or role could be cheap — a small, well-chosen, group-specific dataset rather than a massive crowd vote. And while crowd-scale pairwise voting does produce credible rankings Can crowdsourced votes reliably rank language models?, the corpus's overall message is that scale alone averages groups away; what serves multiple preference groups is structure — separable dimensions, role-bounded standards, and curated rather than aggregated signals.
Sources 10 notes
RLHF and DPO alignment create measurable disparities between English dialects and global opinions, while improving some languages. These disparities reflect deliberate design choices in annotator selection and task definition, not inevitable outcomes.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Writers prefer AI rewrites 63% of the time but object to systematic persona distortions those same rewrites introduce. Mitigation studies show polish and distortion are entangled at the model level—preference optimization produces both simultaneously.
Preferentialist alignment approaches fail because preferences don't capture thick moral values, uniform aggregation produces epistemic injustice, and preference optimization creates systematic misalignment with social roles. Contractualist alignment negotiated by stakeholders and bounded by supra-national, organizational, and individual levels works better.
A 2020–2025 systematic review found that alignment effects are documented almost exclusively in WEIRD samples using inconsistent outcome measures, with mechanisms rarely directly measured. Communication norms vary substantially across cultures, making single alignment policies unlikely to produce uniform effects globally.
A 2020–2025 systematic review shows lexical alignment drives task efficiency and comprehension, while emotional and prosodic alignment drive relational warmth and trust. Conflating them in design produces category errors—cold customer-service bots and evasive mental-health assistants.
SDPO identifies erroneous turns and optimizes surrounding segments, achieving simultaneous improvements in goal completion and relationship quality. Turn-level DPO is too granular; session-level introduces noise from irrelevant turns.
Regularizing agents to maintain consistency when intervention pathways are nullified forces them to evaluate suggestions by causal impact rather than surface plausibility. Common ground alignment emerges as a byproduct without explicit reward.
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.