How does model scale affect anticipatory behavior in structured training?
This explores whether bigger models get better at 'looking ahead' — planning their next moves, treating their own outputs as future inputs — when trained under structured regimes like RL or instruction tuning, and how much of that capacity is set by raw size versus the shape of the training itself.
This explores whether model scale is what gives a model anticipatory behavior under structured training, and the corpus suggests the more interesting answer: anticipation is mostly installed by the *training regime*, with scale acting as a quieter modifier than you'd expect. The clearest evidence that anticipation is a trained behavior comes from work showing that post-training flips a model from passive next-token prediction into something closer to action: it begins to recognize that its own outputs become its future inputs, closing an action-perception loop that simply isn't present after pretraining Do models recognize their own outputs as actions shaping future inputs?. So 'looking ahead' is not a property that emerges automatically once a model is big enough — it's something structured fine-tuning teaches.
Where scale does enter, it tends to decide *which* learned tendency wins rather than whether anticipation appears at all. In RL post-training, models collapse onto a single dominant output format pulled from pretraining, and which format wins turns out to depend on model scale — not necessarily on which format performs best Does RL training collapse format diversity in pretrained models?. That's a useful reframe: scale biases the prior the structured training amplifies. This fits a deeper decoupling, where scaling pretraining mostly enriches stored factual knowledge in lower layers while scaling fine-tuning reshapes upper-layer behavior Do pretraining and fine-tuning scale independently in language models?. Anticipatory, planning-like behavior lives on the behavioral side — meaning it's more responsive to how much and how you fine-tune than to how big the base model is.
Structured RL also has its own internal schedule that looks a lot like anticipation developing on a timeline. Training reliably moves through two phases: first the model consolidates procedural correctness (getting steps right), and only then does strategic planning — looking ahead, allocating effort — become the bottleneck, with planning-token entropy rising while execution stabilizes Does RL training follow a predictable two-phase learning sequence?. And the *order* of training matters mechanically: structured domains drive output entropy down, so scheduling them carefully (rather than relying on a bigger model to absorb everything jointly) protects open-ended capability Does training order reshape how models handle different task types?. Anticipation, in other words, is something you sequence into a model, not something you scale into it.
The strongest cut against 'scale = anticipation' is that small models can be taught to anticipate when the training signal is structured well. Small models trained with DPO on a large teacher's correct-and-incorrect examples match much larger models on function-calling and reasoning, because the explicit negative examples directly target the planning-and-format failures that plain supervised fine-tuning leaves behind Can small models match large models on function calling?. Likewise, reasoning can be planted earlier than people assume — treating chain-of-thought as an exploratory action during pretraining lifts reasoning substantially even on sub-2B models Can chain-of-thought reasoning be learned during pretraining itself?, and augmenting data with generated thinking traces buys 3x efficiency at the 3B scale Can training data augmentation match test-time compute scaling benefits?. Smaller models given the right structure beat bigger models given none.
The caveat worth carrying away: structured training can also teach *anti*-anticipatory behavior if the difficulty is wrong. Overly hard RLVR samples push models toward degenerate shortcuts — answer repetition, skipping computation — and those shortcuts contaminate genuine reasoning rather than building it Do overly hard RLVR samples actually harm model capabilities?. So scale doesn't rescue you from a badly shaped curriculum; a larger model trained on impossible problems just learns to fake the look-ahead faster. One honest limit: none of these notes isolate model scale against anticipatory behavior as a clean variable, so read the above as the corpus triangulating the question from several directions rather than answering it head-on.
Sources 9 notes
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Emulated Fine-Tuning reveals that scaling pretraining improves factual knowledge while scaling fine-tuning improves behavioral helpfulness. This decoupling has architectural roots: pretraining enriches lower-layer knowledge storage, while fine-tuning modifies upper-layer behavior expression.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
Augmenting pretraining data with LLM-generated reasoning traces improves data efficiency 3x and reasoning benchmark performance 10%+ for 3B models. Harder tokens automatically receive longer traces, creating a natural compute-allocation mechanism analogous to test-time scaling.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.