Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?

SFT on domain knowledge treats all tokens equally. A training example of a medical question answered correctly does not distinguish between the tokens that encode critical clinical reasoning and the tokens that are boilerplate formatting. CPT (continual pre-training) is worse: it processes entire domain documents without targeting clinically critical information. Both approaches fail at knowledge coherence — the model may learn isolated facts without integrating them into the connected knowledge structures needed for complex reasoning.

RLAG (Reinforcement Learning from Augmented Generation) takes a different approach. For each question, generate two responses: one with retrieved domain context as prefix, one without. The augmented response is the "preferred" response (the model sees what the correct answer looks like with evidence support). The unaugmented response is what the model can produce from parametric knowledge alone. The reward signals: answer accuracy and explanation rationality — not just whether the final answer is right but whether the reasoning that produced it is coherent.

The iterative cycle: sample → compute rewards → optimize → repeat. With each cycle the model internalizes the knowledge patterns from retrieved context, gradually reducing the gap between its unaugmented performance and augmented performance. The retrieved context during training becomes scaffolding that the model eventually internalizes.

The key difference from SFT: RLAG rewards the model for the quality of its knowledge representations, not just for reproducing training examples. A model that gets the right answer through incoherent reasoning is not rewarded. A model that produces a coherent explanation from genuinely integrated knowledge is.

This adds a new mechanism to the How do knowledge injection methods trade off flexibility and cost?: RL-from-augmentation is not purely dynamic (inference-time RAG) nor purely static (SFT/CPT) — it uses dynamic context during training to progressively embed what it learned into weights, creating models that can reason coherently without retrieval at test time.

Source: RAG