SYNTHESIS NOTE
Does training order reshape how models handle different task types?
Explores whether the sequence of multi-task RL training systematically affects model capabilities across structured and creative domains, and whether this ordering effect can be predicted and optimized.
The standard framing of Does policy entropy collapse limit reasoning performance in RL? treats entropy collapse as a uniform phenomenon — RL training decreases entropy. Omni-Thinker (2025) reveals this is domain-dependent: structured domains (math, coding) decrease output entropy, while open-ended domains (creative writing, dialogue) increase it.
This is not a minor observation — it makes training order a mechanistic variable, not just a scheduling convenience. If you train creative writing first and structured reasoning second, the structured training will collapse the entropy that creative training expanded, potentially degrading creative capability. If you train structured reasoning first and creative writing second, the creative training preserves and expands the model's expressive range. The ordering effect is predictable from backward transfer (BWT) measurements.
Omni-Thinker uses BWT-guided scheduling: order tasks so that later tasks experience minimal negative backward transfer from earlier tasks. The approach uses hybrid rewards — verifiable (rule-based) for deterministic domains + preference-based (LLM-as-Judge) for subjective domains — enabling unified training across domain types within a single policy. The "short-form" QA tasks condition on distractors to reduce reward hacking from random guessing.
The gains are substantial: 6.2% over joint multi-task training, 12.4% over model merging. The accuracy of final multi-task models is well-predicted by forgettability rankings, even under simplifying assumptions — suggesting BWT-guided scheduling has principled theoretical grounding.
This extends Does gradually tightening token budgets beat fixed budget training? from temporal budgets to task ordering: the dimension that matters for multi-task RL is not just how much compute per task, but which tasks come first. And it enriches the entropy collapse understanding: entropy collapse is not a bug to fix everywhere — in structured domains, it reflects desirable precision. The problem is when structured-domain entropy collapse propagates to damage open-ended capabilities.
Inquiring lines that read this note 104
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do language models develop actual world models or merely task heuristics?- Can world models form from aggregated partial information across training distributions?
- How do training objectives shape what a world model actually learns?
- What makes some tasks bounded enough for reliable RL?
- Does task ordering affect multi-task reinforcement learning outcomes?
- Can in-context learning replicate the timing effects that RL teaches models?
- How do residual connections and layer norm stabilize training in deep RL?
- Can smaller models achieve domain expertise through focused RL training?
- Does RL refine existing knowledge or discover entirely new capabilities?
- Can negative reinforcement alone match full RL performance on domain tasks?
- How does RL compress reasoning path diversity during training?
- What makes software engineering environments better suited for RL than other interactive domains?
- Which recipe choices determine the asymptotic ceiling in RL training?
- What distinguishes RL that creates new capabilities from RL that merely teaches timing?
- Can continuous spectrum training outperform sequential SFT-then-RL stages?
- What makes pretraining composition more important than reward engineering?
- How do RL training and base models differ in creating MI peaks?
- Does format-based pretraining determine how models respond to reinforcement learning?
- How do self-evolving curricula help RL break beyond base model capability boundaries?
- Why does RL behavior differ between standard reasoning tasks and complex planning domains?
- Can explicitly optimizing for semantic diversity during RL training improve both quality and variation?
- What scaling properties emerge from RL training dynamics beyond verification?
- Why do single-turn RL methods fail to generalize to multi-turn tasks?
- Why do overtrained domains show different RL training outcomes than novel tasks?
- What training duration is actually needed for RL to expand capabilities?
- How does prolonged RL training differ from standard RLVR approaches?
- Why does supervised fine-tuning on diverse demonstrations expand exploration diversity compared to RL?
- Does the pretrained model prior limit RL search capability more than the optimization algorithm itself?
- Why does outcome-based RL specifically lose diversity during training?
- Does RL teach models new reasoning or just better timing?
- What causes models to develop domain capability cliffs after specialization?
- What task structures benefit most from geometric parameter merging?
- Do different domains require different types of model investment?
- What capabilities actually require massive scale versus specialized training regimes?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Does specialized training in one domain create capability cliffs elsewhere?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- What trade-offs emerge between training objectives and model reliability?
- How much of the combinatorial task space must training data cover?
- How do task difficulty and skill type interact in model performance?
- Does knowledge structure matter more than knowledge volume for model training?
- How does training data distribution create asymmetric competence across relation types?
- Why does training order matter across different domain types?
- Why does curriculum order matter when information theory says data order is irrelevant?
- How do task frequency and complexity interact with model capacity during training?
- Can intentional data-mixture design replace model scaling for rare task learning?
- Can training order and structure shape what networks retain and learn?
- Can curriculum graphs as training data improve model understanding of prerequisite chains?
- Can training data organization by capability outperform source or task-based mixing?
- Why does full multi-task fine-tuning perform worse than sequential training?
- Can backward transfer measurements reliably predict optimal multi-task training order?
- Does training on granular tasks beat training on the full function calling problem?
- How does stage-wise training scheduling resolve conflicts between constraint-following and creative tasks?
- Why does decomposition ability transfer across domains but solving ability does not?
- Why does training data format matter more than domain content?
- Why does training data format matter more than its domain content?
- Can demo placement be tuned as a task-specific hyperparameter?
- Does prompt-to-design speed benefit depend on the type of design task?
- Do task-specific heuristics improve gradually or appear suddenly at scale?
- How can weak-to-strong progressive training target planning without interfering with grounding?
- Do different function-calling subtasks have different entropy profiles during training?
- Does critique training improve exploration diversity during model training or only test time?
- How does trajectory burstiness compare to other structural properties that shape emergent capabilities?
- What structural differences emerge between early generic skills and later meta-strategy skills?
- How does a challenger's escalating difficulty function as curriculum?
- Why does the order of training examples matter for what models learn?
- How do weight visualizations reveal temporal structure in cyclic training?
- How does model scale affect anticipatory behavior in structured training?
- What distinguishes surface mechanisms from the training regimes that produce them?
- What specific tasks should evaluate whether models understand pedagogical sequencing?
- Does constraining AI access during early task phases preserve skill formation?
- Does AI training preserve learning that transfers to independent subsequent tasks?
- Can diversity-aware RL objectives prevent format convergence?
- Does self-generated training data reduce a model's capability diversity?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- Can feedback loop frequency harm performance on finite task sets?
- Can models converge on similar experience descriptions across different architectures?
- How should skill libraries coordinate with gradient-based weight optimization?
- Can training on diverse related tasks be more efficient than task-specific training?
- How does training order affect knowledge acquisition in language models?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does training order reshape how models handle di…
Does policy entropy collapse limit reasoning perfo…
Does gradually tightening token budgets beat fixed…
Why do reasoning models fail differently at traini…
Does RL training collapse format diversity in pret…
Can isolating task-specific parameters prevent mul…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
entropy dynamics are domain-dependent, not uniformly negative; structured tasks decrease entropy while creative tasks increase it
-
Does gradually tightening token budgets beat fixed budget training?
Can models learn reasoning more efficiently by starting with generous token allowances and progressively constraining them, rather than training with fixed budgets from the start? This matters because it addresses how to teach models to think effectively while remaining concise.
BWT-guided scheduling extends curriculum insight from temporal budgets to task ordering
-
Why do reasoning models fail differently at training versus inference?
Reasoning models exhibit two distinct failure modes—entropy collapse during training and variance inflation during inference—that appear unrelated but may share underlying causes. Understanding these dual problems could reveal whether separate or unified solutions are needed.
Omni-Thinker adds that entropy direction depends on task type, further complicating the dual problem
-
Does RL training collapse format diversity in pretrained models?
Exploring whether RL fine-tuning systematically selects one output format from pretraining while suppressing others, and how this selection mechanism drives performance gains.
multi-task training with BWT scheduling may partially address format convergence by exposing the model to diverse task types
-
Can isolating task-specific parameters prevent multi-task fine-tuning interference?
Explores whether identifying and protecting task-specific parameter regions can prevent the performance degradation that occurs when fine-tuning models on multiple tasks simultaneously. This matters because it could enable safe multi-task adaptation without sacrificing individual task performance.
complementary multi-task approach: CPI-FT addresses interference through spatial parameter isolation while Omni-Thinker uses temporal task ordering; CPI-FT shows temporal scheduling alone is insufficient, suggesting combining both spatial isolation and BWT-guided ordering could further improve multi-task training
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Reinforcement Learning with Rubric Anchors
- Eliciting Reasoning in Language Models with Cognitive Tools
- RAGEN-2: Reasoning Collapse in Agentic RL
- Post-training makes large language models less human-like
- A Primer in Post-Training Reasoning Data: What We Know About How It Works
Original note title
multi-task rl reveals complementary entropy dynamics — structured domains systematically decrease output entropy while creative domains increase it making training order a mechanistic variable