Can aligned LLMs generate their own training data?

MAGPIE discovers that the alignment process itself encodes extractable instruction-generation capability. When Llama-3-Instruct receives only its pre-query template — the formatting tokens before user input, like <|start_header_id|>user<|end_header_id|> — it auto-regressively generates high-quality user queries. No prompt engineering, no seed questions, no few-shot examples required.

This observation yields a fully automated pipeline: (1) feed pre-query template, (2) model generates instruction, (3) feed instruction back, (4) model generates response. 4 million instruction-response pairs were generated this way, with quality and diversity comparable to human-curated datasets.

The deeper insight is what this reveals about alignment training: the aligned model has internalized not just how to respond to instructions, but what good instructions look like. The alignment process creates a bidirectional capability — the model learns both the instruction→response mapping AND the response→instruction mapping. Auto-regressive prediction of the next token after user-role formatting tokens generates the kinds of queries the model was trained to handle.

Fine-tuning on MAGPIE-generated data achieves higher AlpacaEval win rates than ShareGPT, Open Orca, Alpaca-GPT4, and Self-instruct datasets. The generated instructions span task categories from information-seeking and reasoning to role-playing and creative writing, with quality filtering available through task categorization, difficulty estimation, and neighbor distance metrics.

This complements Does self-generated training data improve model learning?. SEAL shows self-generated data matches the learner's representational needs; MAGPIE extends this to instruction data specifically, showing the model can generate its own training curriculum.

Source: Alignment