Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?
Explores whether rewarding coherent reasoning patterns during training helps models internalize domain knowledge better than standard fine-tuning approaches that treat all tokens equally.
SFT on domain knowledge treats all tokens equally. A training example of a medical question answered correctly does not distinguish between the tokens that encode critical clinical reasoning and the tokens that are boilerplate formatting. CPT (continual pre-training) is worse: it processes entire domain documents without targeting clinically critical information. Both approaches fail at knowledge coherence — the model may learn isolated facts without integrating them into the connected knowledge structures needed for complex reasoning.
RLAG (Reinforcement Learning from Augmented Generation) takes a different approach. For each question, generate two responses: one with retrieved domain context as prefix, one without. The augmented response is the "preferred" response (the model sees what the correct answer looks like with evidence support). The unaugmented response is what the model can produce from parametric knowledge alone. The reward signals: answer accuracy and explanation rationality — not just whether the final answer is right but whether the reasoning that produced it is coherent.
The iterative cycle: sample → compute rewards → optimize → repeat. With each cycle the model internalizes the knowledge patterns from retrieved context, gradually reducing the gap between its unaugmented performance and augmented performance. The retrieved context during training becomes scaffolding that the model eventually internalizes.
The key difference from SFT: RLAG rewards the model for the quality of its knowledge representations, not just for reproducing training examples. A model that gets the right answer through incoherent reasoning is not rewarded. A model that produces a coherent explanation from genuinely integrated knowledge is.
This adds a new mechanism to the How do knowledge injection methods trade off flexibility and cost?: RL-from-augmentation is not purely dynamic (inference-time RAG) nor purely static (SFT/CPT) — it uses dynamic context during training to progressively embed what it learned into weights, creating models that can reason coherently without retrieval at test time.
Inquiring lines that read this note 84
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI agents improve their skills through accumulated experience and reuse? Does reinforcement learning create genuinely new reasoning capabilities or only refine existing ones?- What role does natural language play in breaking reinforcement learning performance plateaus?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Can reinforcement learning add missing domain knowledge to fine-tuned reasoning models?
- How does reinforcement learning compare to differentiable joint training for RAG?
- How does reinforcement learning differ from chain-of-thought distillation?
- Can smaller models achieve domain expertise through focused RL training?
- Can negative reinforcement alone match full RL performance on domain tasks?
- Can models learn both what and how to study through reinforcement learning?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- How does reinforcement learning on outcomes reinforce template-matching rather than computation?
- Can out-of-distribution tests expose memorization in reinforcement learning fine-tuned models?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- Does reinforcement learning teach models how to reason or when to reason?
- Can reinforcement learning close the gap between LLM reasoning and action?
- What makes supervised fine-tuning worsen RL exploration later?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- When does reinforcement learning actually produce true reasoning gains in models?
- What role does reinforcement learning play in optimizing inference compute?
- Can reinforcement learning improve how accurately models explain themselves?
- Should larger compute budgets allocate more resources to reinforcement learning?
- How much does organized knowledge improve learning efficiency versus raw data?
- Can knowledge graph structure alone generate sufficient training signals for domain reasoning?
- What techniques work best for injecting domain knowledge at training time?
- How do training-time and inference-time knowledge injection techniques compare?
- Does domain training degrade reasoning ability even when benchmark scores rise?
- How does cross-domain reasoning transfer differ from domain-specific knowledge transfer?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- Does knowledge structure matter more than knowledge volume for model training?
- How should rapidly evolving domains choose knowledge injection methods?
- Can in-context learning substitute for domain-specific training altogether?
- What makes knowledge-rich specialized domains structurally different from general reasoning tasks?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- What training cost tradeoffs exist between fine-tuning and other knowledge injection methods?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Can models maintain reasoning-output coupling while improving domain accuracy?
- What makes some contexts learnable as rules versus requiring model retraining?
- Can expert-derived knowledge bases scale to other high-stakes domains?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- Which domains need knowledge injection versus reasoning-focused training?
- Why does structuring knowledge into taxonomies outperform larger unorganized training sets?
- What knowledge injection routes trade flexibility against training cost?
- Which medical tasks benefit most from domain-adaptive pretraining?
- Why does natural language feedback break performance plateaus that numerical rewards alone cannot?
- Can multi-turn reinforcement learning improve tool use in language models?
- Can structured natural language feedback outperform scalar rewards in RL?
- Can environmental rewards directly refine natural language descriptions of actions?
- Can distillation and reward optimization happen in a single training loop?
- Can prompting alone inject new domain knowledge into a model?
- How does explicit exploratory prompting compare to fine-tuned reinforcement learning for in-context adaptation?
- Why does early experience provide better warm-starts for downstream reinforcement learning?
- How does pretrained knowledge constrain what adaptation strategies can achieve?
- What makes pretraining composition more important than reward engineering?
- How does preference-based training compare to supervised fine-tuning for function calling?
- How does task-oriented fine-tuning compare to preference tuning methods?
- Why does policy entropy collapse limit reasoning and dialogue RL scaling?
- Does policy entropy collapse limit how many iterations of reasoning training work?
- How does behavioral fine-tuning differ from factual knowledge encoding in models?
- Why does fine-tuning function as character training rather than capability training?
- How does the functional separation of knowledge and reasoning affect adaptation methods?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Why does adversarial training force deeper reasoning than surface imitation?
- Can articulating latent reasoning processes improve transfer across domains?
- How do procedural versus factual knowledge differ in pretraining versus fine-tuning?
- Does looped pretraining build reasoning more efficiently than supervised fine-tuning?
- Why does reconstructing hidden thought processes during training improve cross-domain transfer?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do knowledge injection methods trade off flexibility and cost?
When and how should domain knowledge enter an AI system? This explores the speed, training cost, and adaptability trade-offs across four injection paradigms, and when each approach suits different deployment constraints.
RLAG is a hybrid: uses dynamic retrieval at training time to drive static weight updating; adds a fifth mechanism to the taxonomy
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
same pruning dynamic: RL removes incoherent knowledge pathways; RLAG uses augmented generation as the reference signal for what coherent pathways look like
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
RLAG's explanation-rationality reward is a direct response to this SFT failure mode
-
Why do specialized models fail outside their domain?
Deep domain optimization creates sharp performance cliffs at domain boundaries. Specialized models generate plausible-sounding but ungrounded responses when queries fall outside their training scope, and often fail to signal their own ignorance.
RLAG's retrieval-augmented training mitigates the cliff by anchoring knowledge to retrieved evidence rather than memorized patterns; models that internalize structured knowledge through RL are less likely to generate plausible-but-wrong outputs at domain boundaries than those trained purely on static data
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Eliciting Reasoning in Language Models with Cognitive Tools
- Efficient Reinforcement Learning via Large Language Model-based Search
- LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
Original note title
rl from augmented generation embeds domain knowledge more effectively than sft by rewarding coherent knowledge structures