Does looping layers beat adding depth in diffusion models?

Masked diffusion models (MDMs) have become a competitive alternative to autoregressive models for language, but improvement has mostly come through parameters and training tokens. LoopMDM asks how to improve along a different axis by importing the looped transformer from the AR literature: apply a shared block repeatedly, converting depth into loops at fixed parameter cost.

The specific finding is that selective looping of the early-middle layers — not the whole network — significantly improves both training efficiency and performance. Looping at train time yields a depth-scaling effect without adding parameters, while varying the loop count at inference enables flexible compute scaling. The numbers are striking: LoopMDM matches same-size MDMs with up to 3.3× fewer training FLOPs, and its final performance exceeds them on reasoning benchmarks (up to +8.5 on GSM8K). Most tellingly, it surpasses deeper non-looped MDMs trained with comparable per-step compute — so the gain is not just "more depth."

The conceptual takeaway is that reusing computation is more effective than adding depth under fixed parameter and compute budgets, at least for diffusion LMs. That this works in MDMs (where it had not been explored) extends the looped-architecture story beyond autoregression. It pairs naturally with How do looped transformer layers actually behave during inference?, which explains why reused computation helps: the loop re-applies the same stages of inference rather than computing genuinely new ones.