Does teaching question patterns before document training improve knowledge access?

To keep an assistant current, the standard recipe is continued pretraining on new documents followed by instruction-tuning on QA pairs. The paper finds this fails: LLMs trained this way struggle to answer questions even when the perplexity of the documents is minimized — the knowledge is encoded but not accessible. The diagnosis is a granularity mismatch: QA pairs are simple and direct, while documents weave many facts together intricately, so encoding document knowledge without knowing how it will be queried produces representations that don't surface under questioning.

The fix inverts the order. Pre-instruction-tuning (PIT) instruction-tunes on questions before continued pretraining on documents, so the model learns how knowledge is accessed before it encodes the knowledge — and the encoding then takes the access pattern into account. PIT outperforms standard instruction-tuning for later factual recall.

The keeper is a principle about knowledge encoding: what the model learns from a document depends on whether it already knows how that knowledge will be retrieved — encoding and access are coupled, not sequential. This connects to Can we predict keyword priming before learning happens? (how new facts get recruited) and to Does procedural knowledge drive reasoning more than factual retrieval?: both, with PIT, point to how knowledge is represented for use mattering more than raw exposure.