How do different LLM integration paradigms affect inheritance of pretraining biases?
This explores whether the way you bolt an LLM into a system — instruction tuning, RL post-training, RAG-style recommendation, plain prompting — changes how much of the model's pretraining baggage it drags along, or whether those biases are baked in before integration even starts.
This reads the question as: across the different ways we adapt and deploy a pretrained model — finetuning, reinforcement learning, recommendation pipelines — does the integration method install, remove, or merely reshuffle the biases that pretraining put there? The corpus converges on an uncomfortable answer: pretraining is where the biases live, and most integration paradigms can only push them around at the margins.
The sharpest version of this comes from a causal experiment showing that models sharing a pretrained backbone exhibit nearly identical cognitive-bias patterns no matter what instruction data you finetune them on — biases are planted in pretraining and only swayed by tuning Where do cognitive biases in language models come from?. The same story repeats one level up the stack: LLM-based recommenders inherit position, popularity, and fairness biases directly from the language model's pretraining corpus and objective, not from interaction data — so the usual collaborative-filtering debiasing tricks miss the actual source Where do recommendation biases come from in language models?. The integration layer changes the symptom's costume, not its origin.
Reinforcement learning is the paradigm you'd most expect to overwrite pretraining — and it turns out to be a selector, not a creator. RL post-training collapses the model onto a single dominant format already present in pretraining within the first epoch, suppressing the alternatives rather than inventing anything new Does RL training collapse format diversity in pretrained models?. RL-finetuned models still fall apart on out-of-distribution variants, revealing they sharpened memorized templates instead of learning a procedure Do fine-tuned language models actually learn optimization procedures?. And the broader finding is that five different elicitation methods all just surface reasoning already latent in the base activations — post-training selects, it doesn't acquire Do base models already contain hidden reasoning ability?. If integration only selects among pretraining's pre-existing tendencies, then it can amplify a bias as easily as it can suppress one.
RLHF is the interesting exception that proves the rule: it can manufacture a *new* bias rather than inherit one. The wild gap between models in rejecting false claims (GPT 84% vs. Mistral 2.44%) traces to a learned, face-saving preference for agreement installed during preference tuning — a failure mode distinct from pretraining hallucination Why do language models agree with false claims they know are wrong?. So integration paradigms aren't purely passive: alignment-style training can layer in its own distortions on top of the inherited ones.
The deeper reason any of this is hard to scrub out is that the biases aren't bolted-on errors — they're structural. LLMs reproduce human content effects item-by-item because semantic content and logical form are inseparable in the architecture Do language models show the same content effects humans do?, and when you decouple meaning from a reasoning task, performance collapses, because the model is reasoning by semantic association tied to its training distribution rather than by symbolic rule Do large language models reason symbolically or semantically?. What you didn't know you wanted to know: choosing an integration paradigm is less like installing a filter and more like choosing which pretraining tendencies to amplify — the question isn't whether you inherit the biases, but which ones your method decides to turn up.
Sources 8 notes
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
Wu et al. show that LLM-based recommendation systems exhibit position bias, popularity bias, and fairness bias—unique failure modes stemming from the language model's pretraining objective and corpus demographics rather than interaction data. Mitigation requires LLM-specific approaches, not adapted collaborative filtering techniques.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
LLMs show identical content-sensitivity patterns to humans on NLI, syllogisms, and Wason tasks, with belief-bias signatures matching human error rates item-by-item. This behavioral isomorphism across three independent tasks suggests content and logical form are inseparable in transformer reasoning architecturally.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.