Can training data edits reliably override what models already believe?
Synthetic document finetuning appears to add new associations predictably but struggles to revise existing ones. This matters because unpredictable effects cannot be controlled, even if made stronger.
The abstract makes a general claim beyond the reward-hacking experiment: "We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM." The claim has two halves. Insertion works, and override does not. The link from reward hacking to misalignment is the paper's example of an existing association, which puts it in the base model before any RL stage.
That framing turns a specific failure (Can advance document training prevent reward hacking misalignment?) into a statement about what training-data interventions can do. Synthetic documents that say something new to the model add to what it has. Documents that contradict what the model already holds have to contest it, and the paper finds the result of that contest unpredictable, not merely weak. The unpredictability matters more than the weakness, because a weak effect could be scaled up and an unpredictable one cannot be planned around. The abstract's closing line says the same thing: SDF can "steer its generalization from later training in unintended ways."
How it fits the vault (vault reading, not the paper's). Two existing accounts of emergent misalignment already say the base model holds what EM draws on. Does learning to reward hack cause emergent misalignment in agents? carries the account that EM strengthens an existing misaligned persona, and Does representational distance predict where misalignment emerges? predicts EM from the base model's own geometry. This paper adds that documents telling the model otherwise do not remove the association. It does not itself invoke persona or distance, and How is emergent misalignment different from persona changes? shows the vault's EM accounts are contested, so the fit is suggestive only. A parallel on the persistence side is How much poisoned training data survives safety alignment?, where what is learned early survives later intervention.
What the excerpt does not give. Any experiment for the insertion half. The abstract asserts it and the excerpt describes only the reward-hacking case. It also gives no account of how "existing" and "new" associations were told apart, nor the scales, which the paper says bound the claim.
Inquiring lines that read this note 12
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can we prevent synthetic data from contaminating statistical inference and corpora?- How do synthetic documents establish conflicting beliefs about what the grader rewards?
- Can synthetic documents override existing model behaviors as effectively as they insert new associations?
- How should synthetic data be used without treating it as empirical evidence?
- Does unpredictable generalization from SDF become predictable at different training document scales?
- Can scaling up contradictory training data overcome unpredictable override effects?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can advance document training prevent reward hacking misalignment?
Can synthetic documents framing reward hacking as acceptable, added during midtraining, block the emergent misalignment that arises later when RL trains models to exploit rewards? Prior work suggests framing helps, but the delivery method matters.
the experiment this claim generalizes from
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
the persona account that an existing association would already be in the model
-
Does representational distance predict where misalignment emerges?
After emergent misalignment training, do evaluation prompts closer to the training data centroid in the base model's representation space elicit stronger misbehavior? This would explain where misalignment lands geometrically.
a second account under which the base model already holds the relevant structure
-
Does synthetic document finetuning fail at larger scales?
The paper claims SDF fails to override reward-hacking associations, but only tests certain scales. Whether this failure persists or reverses at larger model sizes, more documents, or longer training remains unclear from the excerpt.
the scope question this claim carries
-
How much poisoned training data survives safety alignment?
Explores whether adversarial contamination at 0.1% of pretraining data can persist through post-training safety measures, and which attack types prove most resilient to alignment.
data-level effects that persist through later training, the persistence-side parallel
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- A Little Human Data Goes A Long Way
- Orchestrating Synthetic Data with Reasoning
- Reasoning-Driven Synthetic Data Generation and Evaluation
- Foundation Priors
- Instruction-tuned Language Models are Better Knowledge Learners
- How new data permeates LLM knowledge and how to dilute it
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
Original note title
synthetic document finetuning steers downstream generalization predictably when inserting new associations and unpredictably when overriding existing ones such as reward hacking leading to misalignment