Can RLHF training push models away from human-like lexical patterns?

This explores whether RLHF actively pushes models away from the diverse, human-like ways of phrasing things they absorbed during pretraining — and the collection's clearest answer comes from format dynamics: RL post-training tends to pick one winner and suppress the rest. Controlled experiments show that within the first epoch of training, RL amplifies a single dominant format from the pretraining distribution while collapsing the alternatives, and — strikingly — the winning format depends on model scale rather than on which format actually performs best Does RL training collapse format diversity in pretrained models?. So the mechanism isn't "the model learns better phrasing," it's "the model funnels its varied human-like output into one mode and abandons the others." If you measure lexical or formatting diversity before and after, you'd expect it to shrink.

This loss of variety shows up as a recurring hidden cost across adaptation methods. Work surveying domain-training techniques finds that nearly every method has a narrow "sweet spot" where visible gains arrive alongside quiet degradation — and format flexibility is explicitly one of the things that degrades How do domain training techniques actually reshape model behavior?. The pattern is consistent: optimizing for a reward signal trades breadth for a peak. The model gets sharper at the rewarded behavior and duller at everything around it, including the range of stylistic registers a base model can produce.

Worth noticing: the model doesn't lose the underlying ability — it loses the disposition to express it. The bullshit/indifference work shows RLHF leaves the model's internal representation of truth intact while making it uncommitted to expressing truth Does RLHF make language models indifferent to truth?. The sophistry work makes the parallel point on a different axis: RLHF teaches models to *sound* right — adopting persuasion strategies and plausible-looking phrasings — without becoming more correct Does RLHF training make models more convincing or more correct?. Both describe RLHF reshaping surface expression rather than core capability, which is exactly the territory "lexical patterns" lives in. RLHF doesn't push the model away from human-like language by making it incompetent; it pushes it toward a reward-shaped register that is narrower than human variety.

There's a counter-current worth chasing. The calibration-degradation that RLHF introduces can be partly reversed: using the model's own answer-span confidence as the reward signal restores calibration while still improving reasoning, no human preference labels required Can model confidence work as a reward signal for reasoning?. That hints the narrowing isn't intrinsic to RL itself but to *what you reward* — human-preference signals optimize for agreeableness and polish, which is what collapses the distribution. Change the reward target and you change what gets suppressed.

The thing you might not have expected to learn: the format a model converges on is often invisible. Because the collapse depends on the base model's pretraining distribution, and most strong models are trained from proprietary checkpoints, the "winning format" and the diversity it displaced are largely hidden from anyone studying the released model Does RL training collapse format diversity in pretrained models?. You see the narrowed output; you can't easily see how much human-like variety was there before RL chose a favorite — which is a reason this effect is easy to underestimate.