Does recurrent memory or gist compression work better for ultra-long context?

This compares two ways of stretching a model past its context window: recurrent memory, which carries a compressed running state forward and filters out the irrelevant, versus gist compression, which condenses a document into coarse summaries first and pulls back detail only when a task demands it. On raw reach, recurrent memory has the more dramatic result — a fine-tuned recurrent-memory model scrolls through up to 11 million tokens by selectively filtering rather than attending to everything, succeeding at multi-hop reasoning exactly where attention-based models smear their focus onto the opening of the input and degrade Can recurrent memory scale where attention fails on ultra-long text?. Gist compression makes a different bet: a reading agent compresses passages into 'gist memories' before it even knows the question, then looks up specifics later, extending effective context 3–20× and beating retrieval baselines on long-document QA Can LLMs read long documents like humans do?. So the quick read is that recurrent memory goes further, gist goes more cheaply and interpretably.

But the more interesting finding is that the dichotomy may be false. The Titans architecture suggests the best system runs both at once — keeping attention as a precise short-term store while a separate neural memory module compresses the long tail, deciding what to keep by how 'surprising' a token is, and scaling past 2M tokens without the quadratic cost Can neural memory modules scale language models beyond attention limits?. The word doing the work there is 'complement': short-term precision and long-term compression are solving different problems, not competing for the same job.

Which matters because compression has a sharp failure mode that pure reach hides. A single-model approach that continuously merges memory, compression, and response — no external retrieval — turns out to follow an inverted-U: more reprocessing helps up to a point, then degrades *below* a no-memory baseline through misgrouping, lost context, and overfitting Can a single model replace retrieval for long-term conversation memory?. Compression is not free; squeeze too hard and you delete the thing you needed. One promising fix is to make the squeeze adaptive rather than fixed — an external trained manager that prunes aggressively for weak agents but preserves high fidelity for strong ones, matching compression to how much the consumer can actually tolerate Can external managers compress context better than frozen agents?.

Here's the part you didn't know you wanted to know: the choice between these methods might be the wrong question entirely. One line of work argues the long-context bottleneck isn't memory capacity at all but the *compute* needed to fold evicted context into the model's fast weights — a consolidation step that improves with more passes, like test-time scaling Is long-context bottleneck really about memory or compute?. And both camps inherit a deeper limit: state-space and recurrent models with fixed-size state are *provably* worse than transformers at exact copying and retrieval, because a bounded summary cannot losslessly reproduce arbitrary spans Can state-space models match transformers at copying and retrieval?. That cuts directly at compression-based memory — it's wonderful for gist, structurally bad for verbatim recall.

So the corpus answer is conditional. Need to reason across millions of tokens of mostly-irrelevant text? Filtering recurrent memory reaches furthest Can recurrent memory scale where attention fails on ultra-long text?. Need cheap, human-like skimming of long documents with detail on tap? Gist wins Can LLMs read long documents like humans do?. Need exact lookup or structured queries? Both lose to attention and retrieval Can state-space models match transformers at copying and retrieval? — and notice that even fully attentive long-context models quietly degrade on reasoning far below their stated limit, dropping from 92% to 68% accuracy with just a few thousand tokens of padding Does reasoning ability actually degrade with longer inputs?. The frontier systems hedge by combining both Can neural memory modules scale language models beyond attention limits? and by making compression adaptive Can external managers compress context better than frozen agents?.