Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?
Explores whether rewarding coherent reasoning patterns during training helps models internalize domain knowledge better than standard fine-tuning approaches that treat all tokens equally.
SFT on domain knowledge treats all tokens equally. A training example of a medical question answered correctly does not distinguish between the tokens that encode critical clinical reasoning and the tokens that are boilerplate formatting. CPT (continual pre-training) is worse: it processes entire domain documents without targeting clinically critical information. Both approaches fail at knowledge coherence — the model may learn isolated facts without integrating them into the connected knowledge structures needed for complex reasoning.
RLAG (Reinforcement Learning from Augmented Generation) takes a different approach. For each question, generate two responses: one with retrieved domain context as prefix, one without. The augmented response is the "preferred" response (the model sees what the correct answer looks like with evidence support). The unaugmented response is what the model can produce from parametric knowledge alone. The reward signals: answer accuracy and explanation rationality — not just whether the final answer is right but whether the reasoning that produced it is coherent.
The iterative cycle: sample → compute rewards → optimize → repeat. With each cycle the model internalizes the knowledge patterns from retrieved context, gradually reducing the gap between its unaugmented performance and augmented performance. The retrieved context during training becomes scaffolding that the model eventually internalizes.
The key difference from SFT: RLAG rewards the model for the quality of its knowledge representations, not just for reproducing training examples. A model that gets the right answer through incoherent reasoning is not rewarded. A model that produces a coherent explanation from genuinely integrated knowledge is.
This adds a new mechanism to the How do knowledge injection methods trade off flexibility and cost?: RL-from-augmentation is not purely dynamic (inference-time RAG) nor purely static (SFT/CPT) — it uses dynamic context during training to progressively embed what it learned into weights, creating models that can reason coherently without retrieval at test time.
Inquiring lines that read this note 78
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do agent-learned skills transfer and improve across different tasks? Does RL create genuinely new reasoning capabilities or refine existing ones?- What role does natural language play in breaking reinforcement learning performance plateaus?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Why does early experience provide better warm-starts for downstream reinforcement learning?
- Can reinforcement learning add missing domain knowledge to fine-tuned reasoning models?
- How does reinforcement learning compare to differentiable joint training for RAG?
- How does reinforcement learning differ from chain-of-thought distillation?
- Can smaller models achieve domain expertise through focused RL training?
- Can negative reinforcement alone match full RL performance on domain tasks?
- Can models learn both what and how to study through reinforcement learning?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- What makes pretraining composition more important than reward engineering?
- Can out-of-distribution tests expose memorization in reinforcement learning fine-tuned models?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- Does reinforcement learning teach models how to reason or when to reason?
- Can reinforcement learning close the gap between LLM reasoning and action?
- What makes supervised fine-tuning worsen RL exploration later?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- When does reinforcement learning actually produce true reasoning gains in models?
- What role does reinforcement learning play in optimizing inference compute?
- Can reinforcement learning improve how accurately models explain themselves?
- Should larger compute budgets allocate more resources to reinforcement learning?
- How much does organized knowledge improve learning efficiency versus raw data?
- Does knowledge structure matter more than knowledge volume for model training?
- Why does structuring knowledge into taxonomies outperform larger unorganized training sets?
- What techniques work best for injecting domain knowledge at training time?
- What hidden costs emerge when you fine-tune models for a single domain?
- How should rapidly evolving domains choose knowledge injection methods?
- How does behavioral fine-tuning differ from factual knowledge encoding in models?
- How does the functional separation of knowledge and reasoning affect adaptation methods?
- How does pretrained knowledge constrain what adaptation strategies can achieve?
- What training cost tradeoffs exist between fine-tuning and other knowledge injection methods?
- Why does natural language feedback break performance plateaus that numerical rewards alone cannot?
- Can structured natural language feedback outperform scalar rewards in RL?
- How does reinforcement learning on outcomes reinforce template-matching rather than computation?
- Can environmental rewards directly refine natural language descriptions of actions?
- Does outcome-based reinforcement learning improve explanation faithfulness?
- How do internal model mechanisms escape token-level reinforcement signals?
- Does reasoning ability help agents learn from feedback faster?
- Can prompting alone inject new domain knowledge into a model?
- How does explicit exploratory prompting compare to fine-tuned reinforcement learning for in-context adaptation?
- How do training-time and inference-time knowledge injection techniques compare?
- What causes catastrophic forgetting during domain knowledge embedding?
- What makes some contexts learnable as rules versus requiring model retraining?
- Does domain training degrade reasoning ability even when benchmark scores rise?
- How does cross-domain reasoning transfer differ from domain-specific knowledge transfer?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- What makes knowledge-rich specialized domains structurally different from general reasoning tasks?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Can models maintain reasoning-output coupling while improving domain accuracy?
- Can expert-derived knowledge bases scale to other high-stakes domains?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- How do procedural versus factual knowledge differ in pretraining versus fine-tuning?
- What interaction patterns preserve human learning when AI provides domain answers?
- Can multi-turn reinforcement learning improve tool use in language models?
- How does preference-based training compare to supervised fine-tuning for function calling?
- How does task-oriented fine-tuning compare to preference tuning methods?
- How does preference learning differ from supervised finetuning for reasoning?
- Why does policy entropy collapse limit reasoning and dialogue RL scaling?
- Does policy entropy collapse limit how many iterations of reasoning training work?
- Can correct outputs mask reliance on surface heuristics rather than deep understanding?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do knowledge injection methods trade off flexibility and cost?
When and how should domain knowledge enter an AI system? This explores the speed, training cost, and adaptability trade-offs across four injection paradigms, and when each approach suits different deployment constraints.
RLAG is a hybrid: uses dynamic retrieval at training time to drive static weight updating; adds a fifth mechanism to the taxonomy
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
same pruning dynamic: RL removes incoherent knowledge pathways; RLAG uses augmented generation as the reference signal for what coherent pathways look like
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
RLAG's explanation-rationality reward is a direct response to this SFT failure mode
-
Why do specialized models fail outside their domain?
Deep domain optimization creates sharp performance cliffs at domain boundaries. Specialized models generate plausible-sounding but ungrounded responses when queries fall outside their training scope, and often fail to signal their own ignorance.
RLAG's retrieval-augmented training mitigates the cliff by anchoring knowledge to retrieved evidence rather than memorized patterns; models that internalize structured knowledge through RL are less likely to generate plausible-but-wrong outputs at domain boundaries than those trained purely on static data
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Eliciting Reasoning in Language Models with Cognitive Tools
- Efficient Reinforcement Learning via Large Language Model-based Search
- LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
Original note title
rl from augmented generation embeds domain knowledge more effectively than sft by rewarding coherent knowledge structures