Can reasoning happen in latent space during pretraining?
Does building iterative computation into pretraining rather than deferring reasoning to post-training actually improve how language models manipulate knowledge? And what would that tell us about where thinking happens?
Modern LLMs learn to "think" mainly through explicit text generation (CoT), which defers reasoning to post-training and under-leverages pretraining data. Ouro takes the opposite path: a family of pretrained Looped Language Models (LoopLM) that build reasoning into the pretraining phase through iterative computation in latent space, an entropy-regularized objective for learned depth allocation, and scaling to 7.7T tokens. The headline efficiency is striking — 1.4B and 2.6B Ouro models match up to 12B standard transformers (a 2–3× efficiency gain).
Two findings make this more than a parameter-efficiency trick. First, controlled experiments show the advantage stems not from increased knowledge capacity but from superior knowledge manipulation — the same facts, used better. Second, LoopLM's intermediate predictors are strongly aligned with the final predictor, so its latent reasoning traces are more faithful to the final answer than explicit CoT — a safety-relevant property, since articulated reasoning that diverges from the answer is exactly the failure mode CoT-monitoring fears.
This is the pretraining-native member of the recurrence cluster. Where How do looped transformer layers actually behave during inference? explains the mechanism and Can tiny recursive networks outperform massive language models? shows it at tiny scale post-hoc, Ouro shows the same looping pays off when baked into pretraining — and reframes the faithfulness debate, since the latent trace is structurally tied to the output.
Inquiring lines that use this note as a source 7
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Can RL create new reasoning primitives that pretraining never established?
- How does training order affect knowledge acquisition in language models?
- Can articulating latent reasoning processes improve transfer across domains?
- Can minimal training signals unlock latent reasoning capability in base models?
- Can minimal training signals unlock reasoning already latent in pretrained representations?
- Does latent density emerge during pretraining from training data familiarity?
- What latent reasoning capability do base models already possess before training?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do looped transformer layers actually behave during inference?
When language models loop their layers to improve reasoning, do they discover new computations or repeat existing ones? Understanding the internal dynamics could explain why recurrent architectures outperform simple depth scaling.
the mechanism; Ouro applies it during pretraining at scale
-
Can tiny recursive networks outperform massive language models?
Does a small network that refines its reasoning through recursion on a latent state actually generalize better than billion-parameter LLMs on hard puzzles like ARC-AGI? What makes recursion more powerful than scale?
recursion at tiny scale; Ouro at pretraining scale
-
Why is predicting latents more sample-efficient than tokens?
Explores whether learning from a network's own abstract representations requires far fewer training samples than learning from raw tokens, and what mechanism drives this efficiency gap.
both argue manipulating latents beats token-level work
-
Can models reason without generating visible thinking steps?
Do machine reasoning systems actually require verbalized chains of thought, or can they solve complex problems through hidden computation? This challenges how we measure and understand reasoning.
LoopLM's faithful latent traces are evidence that verbalized CoT is not the only reasoning substrate
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Hierarchical Reasoning Model
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Reasoning to Learn from Latent Thoughts
- Training Large Language Models to Reason in a Continuous Latent Space
- Scaling Latent Reasoning via Looped Language Models
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- A Mechanistic Analysis of Looped Reasoning Language Models
- Rethinking Thinking Tokens: LLMs as Improvement Operators
Original note title
looped language models build reasoning into pretraining via iterative latent computation — efficiency comes from knowledge manipulation not capacity