How do recursive language models rethink where to store reasoning?
This explores how a new wave of architectures relocates reasoning out of the visible token stream — into hidden layers, continuous latent loops, masked positions, or sentence-level embeddings — and what that move buys you.
This explores how a new wave of architectures relocates reasoning out of the visible token stream — into hidden layers, continuous latent loops, masked positions, or sentence-level embeddings — and what that move buys you. The premise behind the question is that chain-of-thought reasoning, the visible step-by-step text models produce, may be the wrong place to store thinking. Several notes in the corpus quietly converge on that suspicion from different angles.
The sharpest version comes from work showing that depth-recurrent and latent-loop models scale test-time compute by iterating on hidden states rather than emitting words — architectures like Coconut and Heima suggest that verbalization is a training artifact, not a reasoning requirement Can models reason without generating visible thinking tokens?. That reframes 'where to store reasoning' as a real design choice rather than a given. Meta's Large Concept Model pushes the storage level up instead of in, reasoning over whole-sentence embeddings in a language-agnostic space and only decoding to words at the end Can reasoning happen at the sentence level instead of tokens?. Diffusion LLMs go sideways: instead of reasoning before answering, they embed reasoning directly into masked positions and refine it in place, alongside the answer, so the two co-evolve Can reasoning and answers be generated separately in language models?.
What makes this rethink feel urgent — not just clever — is a parallel set of findings that the visible reasoning trace is an unreliable place to keep anything load-bearing. Transformers can compute the correct answer in their first few layers and then actively overwrite it to produce format-compliant filler text Do transformers hide reasoning before producing filler tokens?. And the traces themselves turn out to be persuasive appearances rather than records of computation: corrupted or logically invalid steps perform nearly as well as valid ones Do reasoning traces show how models actually think?. If the words aren't where the thinking happens, storing reasoning as words is storing a performance.
The lateral payoff is that 'where' interacts with 'how well.' Models reason through semantic association, not symbolic logic, so their reasoning is tethered to training-distribution meaning regardless of the rules in front of them Do large language models reason symbolically or semantically?, and a large share of chain-of-thought errors come from local token-to-token memorization — the model leaning on the preceding tokens rather than the problem Where do memorization errors arise in chain-of-thought reasoning?. Both are failure modes of token-by-token generation specifically. Moving reasoning into latent or embedding space is partly an attempt to escape exactly that local, surface-bound dependency.
The quiet counterpoint worth carrying away: relocating reasoning doesn't relocate the bottleneck. Other work argues that collapses on hard problems are execution failures — the model knows the algorithm but can't run enough steps in text — and that tool use, not more hidden compute, breaks the ceiling Are reasoning model collapses really failures of reasoning?. So the 'where' debate sits next to an open 'whether': changing the substrate of reasoning may not help if the real limit is procedural bandwidth.
Sources 8 notes
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Meta's Large Concept Model operates on sentence embeddings rather than tokens, reasoning in a language-agnostic space before decoding to any target language. This hierarchical approach with paragraph-level planning produces more coherent output than flat token generation.
ICE shows that bidirectional attention in diffusion LLMs enables in-place prompting—embedding reasoning directly in masked positions refined alongside answers. Answer confidence converges early while reasoning continues refining, allowing early-exit mechanisms to cut compute by 50% while maintaining accuracy.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Models confined to text-only generation cannot execute multi-step procedures at scale, even when they know the underlying algorithm. Tool-enabled models solve problems beyond the supposed reasoning cliff, suggesting the bottleneck is procedural execution bandwidth.