How does token-by-token generation constrain a model's ability to plan ahead?
This explores how generating one token at a time — committing irreversibly, left to right — limits a model's capacity to plan, and what the corpus offers as workarounds.
This explores how generating one token at a time — committing irreversibly, left to right — limits a model's capacity to plan, and what the corpus offers as workarounds. The sharpest version of the constraint is architectural: an autoregressive model emits a token and can never take it back. For problems that require trying a partial answer, finding it doesn't fit, and backtracking — exactly what constraint-satisfaction solvers do for a living — this is fatal. The model lacks a "retraction primitive," so it can paint itself into corners it can't escape, which is why bolting on a symbolic solver helps: it supplies the discard-and-retry move the architecture can't perform Why does autoregressive generation fail at constraint satisfaction?. There's also a more philosophical reading of the same limit: token ordering is sequential but "atemporal" — there's no pause-and-reconsider between tokens the way human thinking spends time revising before committing Does AI text generation unfold through temporal reflection?.
But planning doesn't actually require the model to have already 'spoken' the future — it requires the future to influence the present token. Several lines in the corpus attack the problem from that angle. The cleanest is data-side: TRELAWNEY salts training data with special tokens that encode where the sequence is headed, so the model learns goal-conditioned generation and plans better — no architecture change at all Can embedding future information in training data improve planning?. This reframes the constraint as partly a training artifact rather than a hard wall.
The deeper surprise is that token-by-token output and the model's internal computation are not the same thing. Logit-lens work shows transformers can compute a correct answer in their first few layers and then actively suppress it to emit format-compliant filler — meaning the visible token stream lags behind, and even hides, reasoning the model already did Do transformers hide reasoning before producing filler tokens?. If reasoning can live below the surface, then verbalizing every step may itself be the artifact, not the substance: latent-reasoning architectures scale test-time compute by iterating hidden states instead of emitting tokens at all Can models reason without generating visible thinking tokens?. And the fact that deliberately corrupted reasoning traces train models about as well as correct ones suggests those visible chains often act as computational scaffolding rather than genuine plans being read off the page Do reasoning traces need to be semantically correct?.
The most direct assault on the left-to-right commitment is to drop it. Soft Thinking keeps the full probability distribution as a continuous "concept token," so the model holds several reasoning paths in superposition instead of gambling everything on one discrete pick — exploring alternatives without committing Can we explore multiple reasoning paths without committing to one token?. Diffusion LLMs go further: bidirectional attention lets them refine reasoning and answer positions simultaneously rather than prefix-first, so a later realization can revise an earlier slot — the retraction the autoregressive model never had Can reasoning and answers be generated separately in language models?.
Worth knowing: the constraint isn't uniform across the sequence. Only about 20% of tokens are high-entropy "forking points" where the path actually branches; the rest are largely determined Do high-entropy tokens drive reasoning model improvements?. So the real cost of irreversibility is concentrated at a handful of decisive moments — which is precisely why catching a wrong turn there, via lookahead, latent iteration, or backtracking, buys so much.
Sources 9 notes
The performance ceiling on constraint satisfaction problems is not a model-quality issue but an architectural limitation: autoregressive transformers cannot retract emitted tokens, while CSP solvers fundamentally depend on discarding invalid partial assignments. Symbolic solver integration works because it supplies what the architecture lacks.
Token ordering in LLMs follows probabilistic selection without intervening reflection or revision. Human discourse gains meaning from temporal structure—time spent thinking changes what comes next—but AI text production lacks this duration-in-reflection despite appearing sequentially composed.
TRELAWNEY augments training data with special tokens encapsulating future information, allowing models to learn goal-conditioned generation using standard infrastructure. Results show improved planning, algorithmic reasoning, and story generation without modifying architecture or training procedures.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
Training-free method replaces discrete token selection with probability-weighted concept embeddings, preserving superposition of reasoning paths. Improves accuracy up to 2.48 points while reducing tokens 22.4% via entropy-based early stopping.
ICE shows that bidirectional attention in diffusion LLMs enables in-place prompting—embedding reasoning directly in masked positions refined alongside answers. Answer confidence converges early while reasoning continues refining, allowing early-exit mechanisms to cut compute by 50% while maintaining accuracy.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.