Do all annotation responses measure the same underlying thing?
Explores whether RLHF's treatment of all annotations as equivalent signals overlooks fundamental differences in what those responses actually represent—stable preferences versus non-attitudes versus context-dependent constructions.
Behavioral science's six-decade accumulation of preference elicitation research produces a taxonomy that RLHF practice collapses into a single signal. The three categories matter because they require different treatment — and treating them uniformly is the upstream mistake that Are RLHF annotations actually measuring genuine human preferences? argues contaminates the entire pipeline.
Genuine preferences manifest stably across equivalent measurement conditions. Ask the same question with different surface wording, different framing, different order, and the response stays the same. This is what the reward model is supposed to be learning. Only this category is safe to aggregate in the way standard RLHF aggregates.
Non-attitudes are responses generated to satisfy the question without any stable underlying opinion. The respondent has never formed a view on the matter, but the measurement protocol demands an answer, so one gets produced. Non-attitudes are especially pervasive for value-laden questions — precisely the questions that matter most for alignment. Non-attitudes look like genuine preferences in a single measurement but fail the consistency test: re-ask the same respondent and you get a different answer because there was never a stable view to retrieve. Current RLHF treats these as noise to filter or minority views to downweight. The behavioral science view is different: non-attitudes contain no signal at all and should be excluded, not averaged with genuine preferences.
Constructed preferences are assembled on the spot from contextual cues and framing. The respondent is not uncertain (as in a non-attitude); they are producing a coherent answer that depends on the measurement context. Change the context — different anchoring, different comparison class, different framing — and you get a different coherent answer. This category carries real information, but about the interaction between person and context, not about a stable property of the person. RLHF treats constructed preferences as context-independent preferences and trains reward models on them as if they were. The result: reward models that look good on in-distribution evaluation but fail when the deployment context differs from the annotation context.
Measurement artifacts form a fourth related category: same question measuring different constructs for different respondents. One annotator interprets "helpful" as "completes the task"; another interprets it as "gives correct information even when unasked"; a third interprets it as "avoids making the user feel incompetent." They provide coherent, stable responses — each tracking a real preference of theirs — but they are not tracking the same thing. RLHF aggregates them as if they were.
The diagnostic criterion that separates these is consistency across equivalent measurement conditions. Genuine preferences pass; non-attitudes, constructed preferences, and measurement artifacts each fail in distinctive ways. Non-attitudes fail on re-ask (no stable view). Constructed preferences fail on context perturbation (context-dependent). Measurement artifacts fail on question rephrasing (different construct elicited). These are distinguishable empirically, and the distinction determines what should be done with each.
The operational implication is a pre-aggregation filtering step that RLHF currently lacks. Before training the reward model, submit annotation tasks to consistency protocols: re-ask selected items, perturb framings, rephrase questions. Responses that fail consistency tests are not aggregated as preferences; they are either excluded (non-attitudes), contextualized (constructed preferences), or routed to separate annotators (measurement artifacts). This is operationally demanding but conceptually necessary: the alternative is the status quo, in which Why do preference models favor surface features over substance? documents 40% divergences without being able to attribute them to a specific upstream cause.
The taxonomy also suggests why Can models learn to ignore irrelevant prompt changes? works as an output-side intervention. If the upstream measurement problem is consistency failure across equivalent conditions, then training models to be invariant to equivalent-condition perturbations is a downstream patch for the same underlying phenomenon: the system's current robustness against irrelevant cue variation.
Inquiring lines that read this note 146
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does preference optimization undermine conversational grounding in language models?- Why does RLHF alignment reduce the diversity of viewpoints in AI output?
- How does preference optimization create systematic bias toward emotional accommodation?
- Why does RLHF degrade model calibration despite improving preference alignment?
- Can preference optimization and faithfulness measurement coexist as separate alignment objectives?
- What unmeasured side channels emerge from RLHF preference optimization?
- When does RLHF reduce diversity and when does it preserve semantic variation?
- How does constitutional alignment compare to RLHF in removing human annotation costs?
- Can alignment procedures be redesigned to serve multiple preference groups?
- How does RLHF labeler identity shape the values AI systems learn?
- How does RLHF-trained sycophancy manifest differently across feedback and review contexts?
- How does Peircean Secondness differ from what RLHF actually provides?
- Why does RLHF alone fail to fully prevent opinion copying?
- How should historical preferences be weighted when users change their stated intent?
- What design changes if we separate behavior description from adoption justification goals?
- Can cross-view learning align semantic, entity, and item representations of the same user?
- How well does semantic similarity preserve survey response nuance?
- Why do embeddings measure association instead of actual task relevance?
- Can embedding similarity reliably map language model outputs to survey response scales?
- What other hidden biases might aggregate metrics fail to distinguish from reasoning?
- Can standard accuracy metrics miss the real constraints on user consumption?
- How do static benchmarks fail to capture human preference alignment?
- How do self-generated preference pairs from a strong teacher compare to human feedback?
- How can consistency across measurement conditions identify genuine versus constructed preferences?
- Why do ranking metrics fail to capture distributional properties of user taste?
- Why is the Judging preference constant while other traits vary slightly?
- Why do standard preference alignment methods fail at the individual user level?
- How do text-based preference summaries compare to embedding vectors for conditioning?
- Do high-disagreement items signal contested values or measurement noise?
- Can reward models be personalized if annotators lack stable preferences?
- What consistency tests could distinguish constructed from genuine preferences?
- Can counterfactual data augmentation fully eliminate preference model miscalibration?
- Should AI alignment use normative standards instead of aggregate preferences?
- Why does multi-objective ranking make the political dimensions of weight choices more visible?
- When does low-dimensional preference factorization miss important user variation?
- What preference dimensions do base reward functions typically capture?
- Can we distinguish between genuine alignment and response quality bias in reward signals?
- How do per-user concept drift and per-period periodicity combine in time-varying preferences?
- Can preference model training be redesigned to prioritize factual correction over user agreement?
- Can preference learning fix the rigid output format problem better than supervised training?
- How do reward features learned from group data generalize to new users?
- What makes minority preferences disappear in aggregated single-distribution reward models?
- What makes preference distributions unimodal versus genuinely disagreement-heavy?
- Why does preference measurement validity matter more than aggregation methods?
- Can smaller judge models better capture human preferences than larger prompted models?
- How do reward models as policy discriminators differ from labeled preferences?
- How do aggregate reward models fail to capture minority user preferences?
- Can reward models distinguish between personal preference and community consensus?
- How do adversarial IRL and policy discrimination differ in rejecting preference labels?
- How do relational reward signals compare to absolute preference encodings in RL?
- What makes policy discrimination scalable where preference annotation hits bottlenecks?
- How do pairwise comparisons convert subjective quality into trainable ranking signals?
- Can variational inference recover user-specific reward models from preference comparisons?
- How do binary comparisons constrain reward scale in multi-user preference learning?
- Why does single-reward RLHF fail to represent diverse human preferences?
- What validity threats exist in crowdsourced preference signals?
- How does typicality bias in human annotation affect downstream model behavior?
- How does preference learning differ from supervised finetuning for reasoning?
- How do aggregate reward models systematically exclude minority preferences?
- Can latent-variable reward models capture multimodal preference distributions?
- Why does preference measurement validity matter before any aggregation?
- How does preference measurement error propagate through RLHF training?
- Can evaluators detect value-driven output biases without comparing paired questions?
- Can a single AI judge capture diverse human preferences or does it collapse them?
- What explicit concept annotations would improve cross-concept preference reasoning?
- Does reward model training data quality determine fair preference aggregation outcomes?
- Can social welfare functions avoid unjustified assumptions about comparing people's preferences?
- Does measured opinion stability reflect genuine preference or elicitation artifact?
- Why does model uncertainty dominate persona-specific knowledge in annotation tasks?
- Can persona-based approaches capture genuine disagreement in expert annotations?
- How does persona instability in annotation compare to LLM overconfidence in low-resource domains?
- Can we detect superposition in LLM personality traits and stated preferences?
- Can systems recognize and abstain on judgments rather than hallucinating preferences?
- Why do automated selection methods outperform human judgments of relevant context?
- How do annotation artifacts get mistaken for genuine human values?
- How does comparing answers differ from answering when activating company preference?
- How should comprehension failures during preference application be measured and operationalized?
- How does unidimensionality in assessments affect measurement validity?
- What measurement artifacts emerge when annotators interpret the same question differently?
- What happens when we use a single response per condition?
- Why do non-attitudes cluster around value-laden questions most relevant to alignment?
- Can alignment methods model loss aversion without creating unintended sophistry?
- What happens when alignment targets measure only the preferred dimension of entangled properties?
- How does upstream value embedding differ from downstream alignment patches?
- Can constitutional AI alignment work without preference labels by maximizing input-response mutual information?
- How do alignment priors drive similar outputs across different models?
- What role do multi-dimensional quality frameworks play in assessing arguments versus single-metric approaches?
- What fine-grained distinctions matter most for human situated action in categories?
- How does the valence task distinguish whether values support or oppose actions?
- Can evaluation criteria be reliably encoded in labeled data without ground truth standards?
- How much do different detection frameworks disagree on adoption rates?
- How do retrieval systems handle feedback expressed as negations rather than preferences?
- Can graded relevance assumptions hold when user ratings are temporally inconsistent?
- Why do untrained summarizers focus on topics rather than preference dimensions?
- Should emotion systems preserve ambiguity instead of resolving it to one label?
- What makes emotion scores more stable than human preference labels?
- How do implicit signals like clicks capture preference more reliably than explicit ratings?
- What anchoring effects shape how users rate items in sequence?
- Why do explicit ratings fail to capture uncertainty in user preferences?
- Why do online ratings fail to represent independent individual preferences?
- Does the U-shaped distribution of raters compound the negativity bias from public posting?
- How do rating anchors shift meaning within short temporal windows for individual users?
- Can implicit view signals distinguish between consumer preference and confidence?
- How do short-term user satisfaction metrics diverge from long-term user empowerment goals?
- Can side information alone predict preferences without rating history?
- How should unobserved items differ from items rated zero preference?
- Can curiosity rewards about user type complement general social motivation frameworks?
- What structural signals in user language reveal their unstated preferences and context?
- Can elicited user responses measure true preferences or just elicitation artifacts?
- How do human annotators disagree systematically on ambiguous examples?
- Why does fairness depend on context and who you ask?
- Why are ground truth labels missing from unlabeled domain evaluations?
- What information is lost when majority labels discard minority interpretations?
- Why do high-disagreement tasks benefit from broad rater pools over deep annotation?
- What distinguishes genuine user preferences from similar-user preferences in sparse data?
- What happens when personalization aggregates preferences across diverse populations?
- What preference data do different personalized alignment methods actually need?
- Why do health and therapy preferences show weaker utilization than other preference types?
- Can rich environment feedback replace human preference labels entirely?
- How does temporal anchoring maintain learning signals when preference gaps collapse?
- Can information-gain principles improve how we choose what to label?
- How much does preference data freshness matter compared to data source in DPO?
- How does the CAP framework determine a model's initial stance and label reversals?
- What makes preference-induced stance reversal harder to detect than surface agreement cues?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Are RLHF annotations actually measuring genuine human preferences?
RLHF trains on annotation responses as stable preferences, but behavioral science shows humans often construct answers without holding real opinions. Does this measurement gap undermine the entire approach?
the parent argument this taxonomy operationalizes
-
Why do preference models favor surface features over substance?
Preference models show systematic bias toward length, structure, jargon, sycophancy, and vagueness—features humans actively dislike. Understanding this 40% divergence reveals whether it stems from training data artifacts or architectural constraints.
the 40% divergence as downstream symptom; this taxonomy points upstream
-
Why do reasoning models fail at predicting disagreement?
RLVR models optimize for single correct answers, but many real tasks involve legitimate disagreement among annotators. Does this optimization fundamentally suppress the model's ability to capture when humans reasonably disagree?
disagreement that should be preserved vs disagreement that signals non-attitude — current RLHF conflates them
-
Can models learn to ignore irrelevant prompt changes?
Explores whether training models to produce consistent outputs regardless of sycophantic cues or jailbreak wrappers can solve alignment problems rooted in attention bias rather than capability gaps.
consistency-as-diagnostic maps to consistency-as-training-objective
-
Why do LLM persona prompts produce inconsistent outputs across runs?
Can language models reliably simulate different social perspectives through persona prompting, or does their run-to-run variance indicate they lack stable group-specific knowledge? This matters for whether LLMs can approximate human disagreement in annotation tasks.
unstable-across-runs is the constructed-preference signature in simulated annotators
-
Why do LLM judges fail at predicting sparse user preferences?
When LLMs judge user preferences based on limited persona information, what causes their predictions to become unreliable? Understanding persona sparsity's role in judgment failure could improve personalization systems.
verbal uncertainty estimation as an abstention analog for identifying non-attitudes
-
Should AI alignment target preferences or social role norms?
Current AI alignment approaches optimize for individual or aggregate human preferences. But do preferences actually capture what matters morally, or should alignment instead target the normative standards appropriate to an AI system's specific social role?
the normative critique; this note is the measurement refinement that specifies what the inputs actually contain
-
Can text summaries beat embeddings for personalized reward models?
When training reward models on diverse user preferences, does conditioning on learned text-based summaries of user preferences outperform embedding vectors? This matters because better representations could make personalization more interpretable and portable.
text summaries preserve the context that constructed preferences depend on, where scalar rewards lose it
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Measuring Human Preferences in RLHF is a Social Science Problem
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
- Capturing Individual Human Preferences with Reward Features
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
- The Emotion-Memory Link: Do Memorability Annotations Matter for Intelligent Systems?
- Beyond Preferences in AI Alignment
- Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It
Original note title
annotation responses decompose into three distinct signal types — genuine preferences non-attitudes and constructed preferences — each requiring fundamentally different handling