Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning

By creating rankings for 200 such questions over 5 million pretraining documents based on their influence on the likelihood of the completions, we conclude that the generalisation strategy for reasoning is unlike retrieval

We find that instead, the positively influential documents often contain procedural knowledge on how to get to a solution. Further, the models rely less on individual documents when reasoning than when answering factual questions, and the set of documents they rely on is more general. Finally, documents often have a similar influence on reasoning queries that require applying the same procedure to different numbers.

Perhaps a model needs to generalise from a broader set of documents for factual retrieval than for reasoning because it needs to see the same information more often to memorise it.