Contaminating just 0.1% of an AI's training data can plant behaviors that still survive its later safety training.

This explores how what goes into a model's training data affects its security: how easily the model can be compromised, and how well it handles security-related tasks. The corpus has little direct evidence on the second part. It has no studies of models trained to find vulnerabilities or classify threats where the data mix is varied and performance is measured. What it does have is a sharper version of the question: training data composition is itself an attack surface, and small parts of the mix can matter far more than their size suggests.

The clearest example is that contaminating just 0.1% of pretraining data is enough to plant behaviors that survive later safety training How much poisoned training data survives safety alignment?. The surprising part is which behaviors survive. Attacks that make the model refuse service, leak its context, or shift its beliefs persisted through alignment. Jailbreaking was the one attack that alignment reliably removed. So the safety training most people picture as the defense is tuned to one kind of threat and largely misses the others. A related threat class goes after distribution rather than the data itself. Backdoored checkpoints or hijacked model hubs can quietly insert promotional or malicious content while leaving benchmark accuracy untouched Can language models be hijacked to embed hidden advertisements?. That means the usual way of checking a model, its test scores, can't show whether its data has been tampered with.

A second lesson comes from research that isn't framed as security at all: later training amplifies some parts of the pretraining mix and erases others. RL post-training tends to lock onto one dominant format from pretraining within the first epoch and suppress the alternatives. Which format wins depends on model scale, not on which one works best Does RL training collapse format diversity in pretrained models?. Similarly, only about 20% of tokens carry most of the learning signal during reasoning training Do high-entropy tokens drive reasoning model improvements?. Neither study looks at security directly. But if a small slice of data or tokens can steer a whole training run, then a small poisoned or skewed slice could have outsized effects too. This is an inference, not a tested result.

If the goal is to build a model that's good at a security domain, the domain-adaptation research warns that "more and better data" isn't the whole answer. Each adaptation method has domain-specific conditions where it works best, and visible gains often come with hidden losses in reasoning faithfulness or flexibility How do domain training techniques actually reshape model behavior?. Data that is objectively better can still hurt a smaller model if it sits beyond what that model can learn, so students do better when they filter a teacher's refinements by fit Does teacher-refined data always improve student model performance?. Including explicit wrong examples also helps. In one study, DPO on a teacher's correct and incorrect function calls taught small models better than imitation alone Can small models match large models on function calling?. Security work, with its strict formats and costly mistakes, looks like the kind of domain where that would matter.

In short, the corpus can't tell you how much data composition shifts a security model's benchmark scores. It does suggest a more useful question: which small parts of the mix are doing the steering, and would anyone notice if they changed? For the defensive side of this, the work on judging agent actions as linked coordination episodes rather than isolated events is a natural next stop Should defence units span multiple executions and agents?.