How do annotation artifacts get mistaken for genuine human values?

This explores how the artifacts of how we collect feedback get treated as if they were genuine human values. The sharpest answer in the corpus comes from decades of behavioral science: people routinely produce survey responses without having any underlying preference at all. RLHF, the standard method for aligning models to humans, largely ignores this — it takes whatever annotators click and trains reward models on it as stable signal, even when that signal is a non-attitude or a preference constructed on the spot by the act of asking Are RLHF annotations actually measuring genuine human preferences?. The mistake isn't bad annotators; it's a missing distinction between measuring a value and manufacturing one.

What makes this concrete is that annotation responses aren't one thing. They decompose into at least three signal types — genuine preferences, non-attitudes (no real opinion behind the answer), and constructed preferences (an answer invented in the moment) — and you can tell them apart by whether they stay consistent across different ways of measuring. Treating all three uniformly is exactly how the artifact contaminates the pipeline: the noise gets averaged in with the signal and the reward model can't tell the difference Do all annotation responses measure the same underlying thing?.

There's a subtler trap too. Not all annotation disagreement is error to be cleaned away. When people interpret the same socially-loaded sentence differently, that spread often reflects real, valid differences in where the readers stand — meaningful information, not measurement failure Why do readers interpret the same sentence so differently?. So the field has two opposite failure modes at once: it treats *real* variation (genuine perspective differences) as noise to be flattened, while treating *real* noise (non-attitudes, elicitation artifacts) as values to be preserved. Both come from the same root assumption that an annotation is a clean readout of a stable inner preference.

If you're tempted to fix this by simulating annotators with LLMs, the corpus closes that door: run the same persona prompt repeatedly and the output varies as much across runs as it does across different personas. That means the model's own uncertainty — not any stable social knowledge — is driving the answers, so synthetic annotations reproduce the very artifact you were trying to escape Why do LLM persona prompts produce inconsistent outputs across runs?. And it matters downstream, because models don't stay neutral: at scale they settle into surprisingly coherent value systems, which makes it all the more important that the values they're absorbing are real ones rather than measurement residue Do large language models develop coherent value systems?.

The thing you might not have known you wanted to know: the problem is symmetric. "Annotation artifact mistaken for value" and "genuine value mistaken for artifact" are the same bug seen from two sides — a pipeline that has no principled way to ask whether a click means anything before it learns from it.