How does explicit exploratory prompting compare to fine-tuned reinforcement learning for in-context adaptation?
This explores the trade-off between two ways of getting a model to adapt to a new task on the fly — feeding it instructions and examples in the prompt (exploratory prompting / in-context learning) versus updating its weights through reinforcement learning — and what each can and can't actually change.
This explores the trade-off between adapting a model through what you put in its context window versus adapting it by changing its weights with RL — and the corpus suggests the two aren't competitors so much as tools that touch different layers of the model. The most useful framing comes from the idea that adaptation has two channels: a slow one (weights) and a fast one (text in context). Routing task-specific lessons into optimized prompts while leaving parameters mostly untouched reaches the same performance faster and with far less catastrophic forgetting, which reframes forgetting as a misallocation problem rather than an unavoidable cost of learning Can splitting adaptation into two channels reduce forgetting?. So the first thing to know is that the choice isn't binary — you can deliberately split the work.
But prompting has a hard ceiling that RL doesn't. Optimizing a prompt can only reorganize and activate knowledge the model already absorbed during training; it cannot inject anything genuinely new Can prompt optimization teach models knowledge they lack?. RL, by contrast, can actually embed new domain knowledge into the weights — and one approach (rewarding both answer accuracy and the rationality of the explanation) does this more effectively than ordinary supervised fine-tuning, because it rewards coherent reasoning rather than just token-level correctness Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. There's a subtle wrinkle here: instruction tuning, the cheap supervised cousin, may teach mostly the output format rather than real task understanding — models trained on deliberately wrong instructions perform almost as well as those trained on correct ones Does instruction tuning teach task understanding or output format?. That hints prompting and light tuning often just teach the model what shape of answer to produce, not new competence.
The most counterintuitive part is that weight-based RL carries a hidden cost: it narrows the model's behavioral range. RL training compresses exploration diversity in search agents through the same entropy-collapse mechanism seen in reasoning — policies converge on a few reward-maximizing strategies, while supervised fine-tuning on diverse demonstrations keeps exploration broad Does reinforcement learning squeeze exploration diversity in search agents?. So the very process that bakes in skill can also flatten the variety you'd want for genuine exploration. Techniques that deliberately structure exploration — training abstractions that force breadth-first search instead of deeper-and-deeper single chains — exist partly to counter this collapse Can abstractions guide exploration better than depth alone?.
Where pure in-context adaptation shines is in tasks where the model learns from experience without any weight update at all. Storing verbal self-reflections from failures in episodic memory lets agents improve across attempts using only a success/failure signal, no gradients involved Can agents learn from failure without updating their weights?. This works best with structure: in-context learning for sequential decisions needs whole trajectories from the same environment, not isolated examples Why do trajectories matter more than individual examples for in-context learning?, and treating successful and failed episodes differently — successes as concrete demos, failures as abstracted lessons — beats dumping everything in uniformly Should successful and failed episodes be processed differently?. The reader's takeaway: exploratory prompting is fast, reversible, and forgetting-free but capped by what the model already knows; RL can genuinely extend capability but at the price of diversity and the risk of overwriting old skills — and the most interesting work is learning to assign each kind of learning to the channel it's actually good at.
Sources 9 notes
Fast-Slow Training routes task-specific lessons into optimized prompts while keeping parameter updates minimal, reaching equivalent performance 1.4–3x faster with substantially less catastrophic forgetting and plasticity loss, demonstrating that forgetting is a misallocation problem rather than an inherent cost.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.