How do self-generated preference pairs from a strong teacher compare to human feedback?
This explores whether preference signals a model generates for itself — through a strong teacher, self-play, tree search, or self-judging — can stand in for human-labeled feedback, and where each path quietly breaks.
This explores whether preference signals a model generates for itself — via a strong teacher model, self-play, or self-judging — can substitute for human feedback, and the corpus has a more interesting answer than "yes" or "no": self-generated signal often matches or beats human feedback, but only after you account for who the learner is and what kind of preference you're actually capturing.
The strongest case for self-generation is that human annotation isn't the gold standard people assume. One thread shows annotation responses aren't a single clean signal at all — they decompose into genuine preferences, non-attitudes, and preferences constructed on the spot, and treating them uniformly contaminates reward-model training Do all annotation responses measure the same underlying thing?. Worse, preference data isn't even i.i.d.: how well a reward model generalizes depends on the number of *raters*, not just examples, so noisy human pools have a built-in ceiling Does preference data need more raters than examples?. Against that backdrop, machine-generated signal looks competitive. Tree search can manufacture dense, process-level reward without any annotation oracle Can tree search replace human feedback in LLM training?, self-play with a neutral judge co-evolves skills with no human in the loop Can language models learn skills without human supervision?, and models can even judge their own pairwise outputs and improve from ranking consistency alone — one method climbed from 52% to nearly 60% win rate on AlpacaEval with zero external signal Can models learn to judge themselves without external rewards?.
But the "strong teacher" framing in the question hides a trap the corpus names clearly: a stronger teacher is not automatically a better teacher. Teacher-refined data degrades the student when the refinements sit past the student's learning frontier — objectively higher quality, but incompatible — so the student has to *selectively* absorb only what fits its own statistical profile Does teacher-refined data always improve student model performance?. The flip side is just as surprising: with enough teacher-labeled data, a small student can overtake the very teacher that supervised it, because broad input coverage smoothed by teacher predictions generalizes better than the teacher itself Can smaller models outperform their LLM teachers with enough data?. So the teacher-vs-human comparison is the wrong axis — fit between signal and learner matters more than the source's raw strength.
There's also a quality dimension where self-generated signal can exceed numerical human preference labels rather than merely replace them. Plain reward numbers — human or machine — carry no information about *why* something failed; chain-of-thought critiques break performance plateaus that scaling numerical rewards can't Can natural language feedback overcome numerical reward plateaus?. Models can also internalize evaluation entirely, learning to compute their own reward in the unused space after their output at zero inference cost Can models learn to evaluate their own work during training?.
The honest limit is personalization. Self-generated pairs can teach general competence, but genuine individual preference still seems to need a human anchor — though strikingly little of it: roughly ten well-chosen adaptive questions can pin down a person's reward coefficients Can user preferences be learned from just ten questions?. The reader's takeaway: self-generated preference doesn't beat human feedback in the abstract — it beats *bad* human feedback, matches good human feedback on general skill, and the live question is no longer "machine or human?" but "is this signal something the learner can actually use?"
Sources 10 notes
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Preference data is not i.i.d. across raters with different preferences. PAC bounds for personalized reward models decompose into terms depending on both examples per rater and number of raters, showing rater diversity matters as much as data volume.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
Ctx2Skill's three-role self-play loop manufactures missing feedback through internal signals: the Challenger escalates difficulty as curriculum, the Judge gives binary verdicts as reward, and both sides evolve via natural-language skill edits. Success requires balancing adversarial pressure against a generalization safeguard to prevent collapse.
SERL enables self-improving language models by having them alternate between generating responses and judging them pairwise, deriving rewards from ranking consistency and self-consistency of judgments. On AlpacaEval, this reached 59.90% win rate without external signals, up from 52.37%.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.