Why does attending to own latents work better than bolted-on external memory stores?

This explores why a model attending to its own latent representations often beats a separately bolted-on memory store — and the corpus suggests the answer is mostly about translation cost and representational fit, not raw capacity. When a transformer feeds its own latents back into attention, the 'memory' already lives in the same representational space the model reasons in, so there's no boundary to cross. Can models learn working memory by attending to their own latents? shows this directly: adding a feedback loop lets a transformer attend to its own activations and grow a kind of working memory for indefinitely long inputs — with no extra weights at all. An external store, by contrast, forces the model to encode a query, search, retrieve, and re-ingest text it then has to re-understand from scratch.

There's a deeper reason latents are efficient. Why is predicting latents more sample-efficient than tokens? proves that learning at the latent level recovers compositional structure exponentially faster than learning over raw tokens, because nearby latents are far more correlated than the surface tokens they summarize. The same logic applies to memory: the model's own latents are already a compressed, structured view of the past, whereas an external store typically hands back raw text that the model must re-compress every time it reads it. Is long-context bottleneck really about memory or compute? sharpens this — the real bottleneck in long context isn't storage capacity, it's the compute to fold old context into fast internal state. A bolted-on store sidesteps storage but pays that consolidation cost over and over; internal memory pays it once.

But the corpus refuses to let 'latents always win' stand. Can state-space models match transformers at copying and retrieval? shows the opposite failure: when your memory is a single fixed-size latent state (as in state-space models), it provably cannot copy or retrieve long strings — the latent simply runs out of room, and attention over explicit context wins. And Can models store unlimited facts without growing larger? shows that for sheer factual recall, an external tool is better than internal weights: cramming facts into parameters is bounded by model size and even corrupts prior knowledge, while a lookup circuit gives unbounded recall. So the honest synthesis is: internal latents win for working memory and reasoning continuity; external stores win for large, exact, retrievable facts.

The frontier work is essentially trying to get both. Can neural memory modules scale language models beyond attention limits? (Titans) splits the job — attention as short-term latent memory, a learned neural module for long-term — and crucially the long-term memory is *learned and adaptive*, not a dumb bolt-on lookup. Can agents compress their own memory without losing critical details? does the same for agents, letting them fold their own history into structured schemas rather than dumping raw logs into a store. The recurring lesson is that the dichotomy isn't 'internal vs external' so much as 'memory the model owns and shapes' versus 'memory the model merely queries' — and even retrieval works best when the model learns *when* to reach for it, as When should language models retrieve external knowledge versus use internal knowledge? finds by framing each retrieve-or-recall decision as a learned choice.

What you didn't know you wanted to know: the very thing that makes a bolted-on store feel safe — that it stores raw text verbatim — is the source of its overhead. Where does a model store memorized paragraphs? shows that even internal verbatim memorization concentrates in a few low layers and rare-token attention heads; memory in these systems is never free-floating storage, it's a specific computational structure. The reason attending to your own latents wins is that it keeps memory *inside that structure* instead of forcing a round trip out to text and back.