Why do models verify facts better than they generate them?

This paper traces how the "generation-verification gap" (GV-gap) for factual knowledge — the finding that language models judge a factual statement more accurately than they produce it — develops across a model's training life cycle. Fine-tuning "four open-source model families across two scales each" on synthetic facts, the authors measure generation and verification accuracy through three phases: "acquisition, continual learning, and updating." Three findings recur across every model and scale: "verification is consistently learned before generation," "verification is more robust to continual learning than generation," and factual updates "can leave models in a multi-verse state, simultaneously verifying both old and new answers as correct." "Natural experiments on frontier models" — exploiting how real-world topics and time periods vary in data coverage — reproduce the same three dynamics at scale, and additionally surface "residual verification biases on well-covered facts."

The paper frames this as a training-mechanism account rather than a purely architectural one: because verification "reduces to a decision over a small token space (e.g., a binary True/False)" while generation "requires sampling a sequence from the joint distribution over the full vocabulary, with each step compounding the difficulty," the two capabilities are learned from the same training data at different rates and forgotten at different rates. The authors place this "factual" GV-gap alongside a "computational" gap (P-vs-NP style, hard to trace to data) and an "aesthetic" one (unmeasurable, diffuse) to argue factual knowledge is the cleanest testbed because both capabilities can be "traced to specific training data points and their strength objectively measured." The multi-verse finding follows from this asymmetry: an update can overwrite what a model generates while its verifier keeps accepting the superseded answer as true, because the two capabilities decay on different schedules.

This sits alongside What limits how much models can improve themselves?, which formalizes the GV-gap as a quantity that shrinks for factual tasks with scale; this paper supplies the training-mechanism story behind that convergence — showing it arises because verification is acquired first and decays more slowly, not because the gap is a static property of the task. It also complicates Can AI verify research outputs as fast as it generates them?: that pattern describes generation outrunning verification for research artifacts and judgments, the mirror image of what this paper finds for simple factual claims, where verification is the capability that comes first and lasts longest. The difference suggests the GV-gap's direction is domain-dependent — favoring verification for well-defined factual triplets, favoring generation-over-verification difficulty for open-ended research outputs — rather than a single universal asymmetry.

The study is confined to "single-hop facts" presented as dense, explicit synthetic sentences, injected during post-training/"mid-training" rather than pretraining, and the frontier-model results are natural experiments, not controlled interventions, with the authors noting the most capable models show a confound of "recognizing evaluation contexts." Mitigations like RAG and Best-of-N sampling are not tested. The excerpt therefore establishes that the ordering and asymmetric decay of generation versus verification is a robust training-mechanism effect in a controlled synthetic setting and appears in frontier-scale natural data, but it does not establish that this holds for multi-hop facts, naturally-phrased training text, or pretraining-scale fact injection — nor whether known mitigations close it.