When a test's questions and answer keys come from a few experts, it may measure their way of asking, not the real world.

This explores who writes the questions and answers in expert-built benchmarks, and how that narrow pool of authors and sources limits what the test can stand for. The retrieved notes don't include a study that audits benchmark authors directly, such as their countries, languages, professions or institutions. What they do show, from several angles, is why a small and uniform group of writers produces a skewed picture.

The clearest statement of the problem comes from work on user diversity. Can simulated users reveal what offline benchmarks miss? argues that fixed, outcome-only benchmarks leave out two things: how different people phrase their requests, and how differently they judge whether an answer is good. An expert-written benchmark builds in one way of asking (the expert's) and one standard of success (the expert's answer key). The proposed fix is telling. Instead of recruiting more experts, it simulates billions of persona records to put back the variety that a small group of authors can't supply.

Crowdsourced evaluation comes at the same problem from the other side. Can crowdsourced votes reliably rank language models? shows that 240K+ votes from ordinary users produce rankings that agree with expert raters, and the reason given is that the crowd's questions are diverse and good at separating models. The surprise is that agreeing with experts is the check on the crowd, not the other way round. Breadth of sources does work a small expert pool can't. But a crowd has its own demographic skew, since the votes come from whoever shows up to a chatbot leaderboard.

The curation and consensus notes add a less obvious point: careful selection narrows a dataset as well as improving it. Can careful curation replace massive alignment datasets? shows that 1,000 hand-picked examples can rival huge datasets. The flip side is that the curators' taste becomes the definition of quality. Can models trained on many imperfect experts outperform everyone? explains why many imperfect experts can beat any single one: their random, unrelated errors cancel out. That only works when the errors are unrelated. If benchmark writers share a background, language or training, their blind spots line up and don't cancel. Averaging over them then makes the shared bias stronger instead of removing it.

If you want to go further, the practical lesson is to ask two questions of any expert benchmark. Who wrote it, and would those people tend to make the same mistakes? The corpus gives you the reasoning for why that matters, but not a dataset-by-dataset audit of benchmark authorship.