Can generating entire videos at once beat keyframe interpolation?

Text-to-video generation is harder than text-to-image because motion is sensitive to error and adds a temporal dimension that strains memory, compute, and data. The prevalent approach generates distant keyframes first, then fills the gaps with a cascade of temporal super-resolution models — and Lumiere identifies an inherent limitation in this: it cannot learn globally-coherent motion, because the keyframe-then-interpolate pipeline never represents the whole temporal trajectory at once.

Lumiere's response is architectural: a Space-Time U-Net that generates the entire temporal duration of the video in a single pass, incorporating both spatial and temporal down- and up-sampling modules. Generating the full clip at once — rather than stitching independently-generated keyframes — is what produces coherent motion, and it generalizes to image-to-video, inpainting, and stylized generation.

The transferable keeper is a generation principle: coherence is a property of generating the whole at once, not of stitching locally-coherent pieces. Cascades that decompose a globally-structured output into independently-generated fragments lose the global structure exactly where it matters (motion, here). This rhymes with Can iterative revision cycles match how humans actually write?: both treat a long structured artifact as something to denoise as a whole rather than assemble piecewise.