Why does safety alignment break after only 10 harmful examples?
This explores why safety alignment is so fragile that a handful of harmful fine-tuning examples can undo it — and the corpus suggests the answer is that alignment was never deep to begin with.
This explores why safety alignment is so fragile that a handful of harmful fine-tuning examples can undo it. The corpus doesn't have a paper that runs the exact "10 examples" experiment, but read laterally it points at one explanation again and again: alignment is a thin behavioral layer activated on top of capabilities the model already has — not a deep removal of anything. If alignment is just *surfacing* a disposition, then a few examples can re-surface the opposite one just as cheaply.
The sharpest evidence for this is LIMA, which showed that 1,000 carefully curated examples produce alignment competitive with datasets orders of magnitude larger, because post-training "activates existing capabilities rather than building new ones" Can careful curation replace massive alignment datasets?. That's usually read as good news about data efficiency — but it cuts both ways. If a thousand examples can switch alignment *on* by activation, the symmetric implication is that very few examples can switch the unsafe behavior back on, because the underlying capability was never gone. MAGPIE makes the point even more starkly: aligned models will auto-regressively generate fluent instruction-response data when handed *only* the pre-query formatting tokens, no prompt at all Can aligned LLMs generate their own training data?. Alignment that can be steered by formatting tokens is a shallow overlay, not a structural change.
The persistence literature confirms the base layer stays intact. Pretraining poisoning at just 0.1% of data survives standard safety alignment for denial-of-service, context-extraction, and belief-manipulation attacks — alignment only reliably suppresses jailbreaking How much poisoned training data survives safety alignment?. So the safety pass doesn't reach down and edit what the model learned in pretraining; it sits on top of it. And the Moral RolePlay work shows what that overlay actually looks like from the inside: aligned models handle villainy by "superficial substitution," swapping crude aggression in for nuanced malevolence rather than genuinely lacking the trait Does safety alignment harm models' ability to roleplay villains?. The capability is suppressed, costumed over — not absent.
There's a deeper version of this fragility worth knowing about. Ethical alignment turns out to be a *separate axis* from other competencies entirely — HHH-trained models still violate basic conversational pragmatics, which means RLHF is changing one narrow behavioral channel and leaving the rest untouched Can ethically aligned AI systems still communicate poorly?. Narrow, channel-specific training is exactly the kind of thing a narrow, channel-specific counter-example can reverse. And guardrails themselves turn out to be contextual and inconsistent — refusal rates shift with the user's apparent demographics and ideology Do AI guardrails refuse differently based on who is asking? — which is what you'd expect from a learned surface heuristic rather than a robust internal constraint.
The thing you didn't know you wanted to know: the same property that makes alignment cheap to install (you can align a strong model with a tiny curated set) is the property that makes it cheap to remove. Fragility isn't a bug in the fine-tuning recipe — it's the flip side of how shallow post-training works. If you want alignment that resists 10 bad examples, the corpus implies you'd need something that changes the model's underlying capabilities or values rather than just activating a disposition — which is the harder, less-solved problem.
Sources 6 notes
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
MAGPIE shows that aligned models like Llama-3-Instruct auto-regressively generate diverse, high-quality instructions when given only pre-query formatting tokens, without prompt engineering. 4M generated pairs matched human-curated datasets in quality and outperformed external sources in downstream fine-tuning.
Denial-of-service, context extraction, and belief manipulation attacks persist through standard safety alignment at 0.1% poisoning rates, while jailbreaking attacks are successfully suppressed, contradicting sleeper agent persistence hypotheses.
The Moral RolePlay benchmark shows LLM performance drops from 3.21 for moral paragons to 2.62 for villains, with largest degradation between flawed-but-good and egoistic characters. Models fail most on deception and manipulation traits, substituting crude aggression for nuanced malevolence.
Research shows that HHH-aligned models can violate Gricean maxims, lose common ground, and mishandle context despite being honest and harmless. Pragmatic competence requires architectural changes that RLHF alone cannot deliver.
GPT-3.5 refuses requests at different rates for younger, female, and Asian-American personas, and sycophantically declines to engage with political positions users would disagree with. Sports fandom and other non-political signals also shift refusal sensitivity.