Can organizing knowledge structures beat raw training data volume?

StructTuning's efficiency result challenges the standard assumption that more domain training data produces proportionally better domain performance. The two-stage approach — Structure-aware Continual Pre-Training (SCPT) followed by Structure-aware Supervised Fine-Tuning (SSFT) — achieves 50% of traditional full-corpus knowledge injection performance using only 0.3% of the training data. The key variable is not volume but structure.

The insight driving this: standard knowledge injection concatenates text chunks and trains on them, discarding the organizational structure of the source material (textbook chapters, topic hierarchies, concept taxonomies). StructTuning instead auto-generates a domain knowledge taxonomy from the corpus using an LLM, then trains the model to predict text chunks in the context of their taxonomy location. Each chunk is treated as a knowledge point linked to the broader knowledge graph. The model learns not just the text content but its position in the domain's conceptual structure.

The SSFT phase leverages this structural awareness for task performance: the model is explicitly prompted to reveal the underlying knowledge structure in its outputs before applying it to solve problems. This is the mechanism that makes structural injection efficient — the taxonomy acts as a retrieval scaffold at inference time, allowing the model to navigate domain knowledge rather than pattern-match through it.

The inspiration is explicitly drawn from how human students learn from textbooks: students don't memorize raw text sequentially; they build hierarchical understanding (chapter → section → concept) that enables targeted retrieval. The analogy captures something real about the difference between storing knowledge and organizing it for use.

The efficiency implication is significant for practical domain specialization. Full-corpus fine-tuning on domain data is expensive, slow, and requires large proprietary datasets. If structure-aware injection can achieve 50% performance with 0.3% of the corpus, even if you need to add more data to approach full performance, the efficiency curve favors structured injection at every scale. This is consistent with Can formal language pretraining make language models more efficient? — structured input improves efficiency not just for syntax but for knowledge injection.

KG curriculum as a more powerful instance of structure > volume. The KG curriculum approach (QwQ-Med-3) extends this principle: instead of auto-generating a taxonomy from text, it derives reasoning tasks directly from KG structure — random walks produce multi-hop reasoning chains, and entity-relation triples provide compositional primitives. With just 24K KG-derived reasoning tasks, a 3B model approaches frontier medical AI performance. Both StructTuning and KG curriculum demonstrate the same core insight: knowledge organization drives learning efficiency more than knowledge volume. But KG curriculum goes further by making the relational structure itself the training signal rather than just the organizational scaffold. See Can knowledge graphs teach models deep domain expertise?.

Source: Domain Specialization; enriched from Knowledge Graphs