Can bounding boxes replace image encoders for document understanding?

Enterprise documents — forms, invoices, contracts, receipts — carry meaning at the intersection of text and spatial layout, and most multimodal LLMs handle this with heavy image encoders. DocLLM's design choice is to drop the image encoder entirely and use only bounding-box information to incorporate spatial structure. It captures the cross-alignment between text and spatial modalities by decomposing the classical transformer attention into a set of disentangled matrices (separating textual and spatial contributions), and pretrains with a text-segment infilling objective suited to the irregular layouts and heterogeneous content of real documents.

The keeper is the cheap-spatial-signal move: bounding boxes are a lightweight, structured stand-in for full visual encoding, and disentangled attention lets the model reason over layout without the cost and brittleness of pixel encoders. The broader claim DocLLM gestures at — that layout-aware pretraining lets language models go beyond plain-text next-token prediction to treat documents as inherently structured knowledge — points at incorporating e-books and richly-formatted corpora into pretraining without heavy preprocessing.

This sits in the multimodal/document corner of the vault as the spatial-but-not-visual design point. It contrasts with the strong-vision GUI position of Do text-based GUI agents actually work in the real world?: where GUI agents argue real deployment needs pixels, DocLLM argues that for layout-structured documents, bounding boxes recover most of the spatial signal at far lower cost.