Can explicit goal state scaffolding at inference time transfer to autonomous tracking through training?
This explores whether the trick of handing a model an explicit goal or future-state at run time can be baked into the model through training, so it ends up tracking that goal on its own without the scaffold.
This explores whether the trick of handing a model an explicit goal or future-state at run time can be baked into the model through training, so it ends up tracking the goal on its own without the scaffold. The most direct answer in the corpus is yes — and the cleanest demonstration is TRELAWNEY, which sprinkles training data with special tokens that encapsulate future information Can embedding future information in training data improve planning?. The interesting move is that the scaffold lives only in the training corpus, not in the architecture or the inference call: the model learns goal-conditioned generation through ordinary next-token training and then plans without needing the tokens spoon-fed at run time. That's the transfer you're asking about — an explicit signal becomes an internalized habit.
But there's a recurring caution worth sitting with: a lot of what training transfers turns out to be format and protocol rather than genuine goal-understanding. Instruction tuning, for instance, mostly teaches a model the *shape* of acceptable output, not the meaning of the task — models trained on deliberately nonsensical instructions perform almost as well as those trained on correct ones Does instruction tuning teach task understanding or output format?. So if you scaffold a goal at inference time and then train it in, the honest question is whether the model learned to *pursue* the goal or merely learned to *emit text that looks goal-directed*. The distinction matters enormously for autonomy.
Where the corpus suggests something deeper than format does transfer is in the work on self-generated supervision. 'Early experience' shows agents treating the future states their own actions produce as a training signal, with no external reward — and this internalized signal gives a much stronger warm-start for later RL Can agents learn from their own actions without external rewards?. Relatedly, post-training appears to flip a model from passive next-token prediction into something that recognizes its own outputs as actions feeding its future inputs — an action-perception loop measurable as sharply lower on-policy entropy Do models recognize their own outputs as actions shaping future inputs?. That loop is arguably the substrate autonomous goal-tracking needs: a model that knows its output now constrains its situation later is already doing a primitive form of goal maintenance.
There's also a clue about *when* this transfer happens during training. RL training tends to run in two phases — first the model consolidates procedural execution, and only later does strategic planning become the bottleneck, with optimization pressure concentrating on planning tokens Does RL training follow a predictable two-phase learning sequence?. Read against your question, that implies goal-tracking isn't learned all at once: the model first learns to act correctly step-by-step, and the higher-order 'where am I heading' capacity emerges as a distinct, later acquisition. And training regime, not inference budget, is what installs it — non-reasoning models can't simply be handed more inference compute to catch up to models whose training instilled a reasoning protocol Can non-reasoning models catch up with more compute?. That's the strongest structural argument that scaffolding-at-inference and tracking-through-training are not interchangeable: the scaffold buys you behavior for one run, training buys you the protocol.
The lateral surprise here is the grounding objection. Even if you successfully transfer goal-tracking through training, there's an argument that a goal encoded purely as symbols — never checked against the world or mediated socially — can quietly drift from the value it was meant to represent Can AI systems achieve real alignment without world contact?. So 'autonomous tracking' might be achievable as a *behavioral* property while still being hollow as a *correspondence* property. The corpus, taken together, says the transfer is real and demonstrable — but invites you to ask whether what transferred is a goal, or just a very convincing routine for behaving as if there were one.
Sources 7 notes
TRELAWNEY augments training data with special tokens encapsulating future information, allowing models to learn goal-conditioned generation using standard infrastructure. Results show improved planning, algorithmic reasoning, and story generation without modifying architecture or training procedures.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Peircean semiotics reveals that symbolic goal encoding without world contact and social mediation cannot guarantee correspondence to actual values. LLMs operating in pure symbol manipulation risk divergence between stated goals and real-world outcomes.