Does synthetic training data need to be good, varied, or just mixed in alongside real data the right way?

This explores whether the way synthetic data enters training (swapped in for real data, or added alongside it) changes how much its quality and diversity matter. The collection has no single paper that tests all three together. Read side by side, though, the notes suggest something unexpected: the main reason the schedule matters may be diversity. The clearest result on schedules is Does model collapse depend on how we schedule training data?. If each generation's synthetic output replaces the real data, test error grows without limit. If synthetic data accumulates on top of the original real corpus, error stays bounded. This held for language models, diffusion models and VAEs. One plausible reading is that the real data works as a diversity anchor: it keeps the full spread of the original distribution present, so nothing can quietly drop out.

That fits a separate line of work that splits "good synthetic data" into parts. How do quality, diversity, and complexity affect synthetic data differently? finds that quality mainly helps a model on data like what it has seen. Diversity is what lets it cope with situations it hasn't seen. Complexity helps both. Because common evaluations fold all three into one quality score, self-improvement loops tend to keep raising quality while losing diversity, and that loss is hard to reverse. Put the two notes together and you get a hypothesis the corpus suggests but doesn't directly test. Replacement schedules are dangerous because each round filters for quality and lets the variety shrink. Accumulation schedules survive because the original variety is never thrown away.

The same pattern of variety vanishing without anyone noticing shows up in other places. Does synthetic content in search results hide ecosystem decay? shows a search corpus that becomes 67% synthetic. Over 80% of retrieved results then come from synthetic sources, yet answer accuracy stays high. The monoculture shows up only when it breaks. Inside training, Does RL training collapse format diversity in pretrained models? finds that RL amplifies one output format from pretraining within the first epoch and suppresses the rest. The pattern repeats: optimizing for 'good' answers shrinks variety, and the usual metrics don't flag it. One response is to reward diversity directly. Can diverse mediocre traces outperform redundant expert traces? scores a whole set of outputs together, and finds that varied mediocre traces can beat a set of near-identical strong ones.

If diversity is what's at risk, the next question is how to generate it on purpose instead of hoping real data keeps supplying it. Can we generate synthetic data without any seed examples? (Simula) handles broad coverage with a topic taxonomy and handles complexity separately, so quality, diversity and complexity can each be tuned. Can synthetic dialogues become realistic through layered diversity? builds realistic dialogue variety from several layers: subtopic, personality and context. What makes synthetic data work across different domains and models? adds a caveat: the right balance changes with domain, model and scale, so no single schedule or mix will be best everywhere.

One more idea changes the question. Should we treat LLM outputs as real empirical data? argues that model-generated text reflects the model's learned patterns and the prompt it was given, not new observations about the world. It should count only through an explicit trust weight. Seen this way, "accumulate, don't replace" is a rough version of a better rule: never let synthetic data outweigh the real evidence it came from. The collection does not yet hold a study that varies the accumulation schedule and synthetic-data diversity at the same time. That gap is worth flagging.