Can reasoning be learned during pretraining rather than after?

Modern LLMs learn to "think" mainly through explicit text generation (CoT), which defers reasoning to post-training and under-leverages pretraining data. Ouro takes the opposite path: a family of pretrained Looped Language Models (LoopLM) that build reasoning into the pretraining phase through iterative computation in latent space, an entropy-regularized objective for learned depth allocation, and scaling to 7.7T tokens. The headline efficiency is striking — 1.4B and 2.6B Ouro models match up to 12B standard transformers (a 2–3× efficiency gain).

Two findings make this more than a parameter-efficiency trick. First, controlled experiments show the advantage stems not from increased knowledge capacity but from superior knowledge manipulation — the same facts, used better. Second, LoopLM's intermediate predictors are strongly aligned with the final predictor, so its latent reasoning traces are more faithful to the final answer than explicit CoT — a safety-relevant property, since articulated reasoning that diverges from the answer is exactly the failure mode CoT-monitoring fears.

This is the pretraining-native member of the recurrence cluster. Where How do looped language models actually improve reasoning in depth? explains the mechanism and Can tiny recursive networks outperform massive language models? shows it at tiny scale post-hoc, Ouro shows the same looping pays off when baked into pretraining — and reframes the faithfulness debate, since the latent trace is structurally tied to the output.