Can neural memory modules scale language models beyond attention limits?
Can separating short-term attention from adaptive long-term memory allow models to efficiently handle context windows exceeding 2M tokens while maintaining competitive performance?
Titans (2501.00663) introduces a neural long-term memory module that addresses a fundamental contradiction in linear recurrent models: they are designed for efficiency on long contexts, but long contexts cannot be properly compressed into small fixed-size states.
The architectural insight is that attention and memory serve fundamentally different functions. Attention operates as short-term memory — accurate direct dependency modeling within the current context window, but quadratic cost limits its reach. Neural memory operates as long-term memory — compressed and persistent, memorizing data that is surprising or close to surprising tokens. The memory update mechanism considers the proportion of memory size to data surprise, resulting in adaptive memory management.
Three integration variants are proposed: memory as context (attending to memory alongside current context), memory as gating (memory modulates attention output), and memory as a layer (memory replaces some attention layers). Each variant trades off between integration depth and computational overhead.
The results establish that Titans outperform both standard Transformers (with the same context window) and modern linear recurrent models across language modeling, common-sense reasoning, genomics, and time series. Critically, Titans scale to context windows larger than 2M tokens while showing competitive performance with Transformers that use the entire context — the long-context problem is addressed without the quadratic penalty. The persistent nature of the memory module makes it a natural substrate for Can models precompute answers before users ask questions? — the memory can store precomputed inferences between interactions, and sleep-time processing can populate the memory with anticipated query-relevant information.
Since Can models reason without generating visible thinking tokens?, the Titans architecture offers a complementary path: rather than scaling reasoning depth through recurrent computation, it scales memory breadth through adaptive memorization. Both bypass the limitations of standard attention but along different architectural dimensions.
Miras unifying framework (2504.13173): The "It's All Connected" paper reconceptualizes Transformers, Titans, and modern linear recurrent models as associative memory modules that learn a mapping of keys to values using an internal objective — termed "attentional bias." The paper observes that most existing sequence models use either dot-product similarity or ℓ2 regression objectives as their attentional bias. Miras provides a general framework with four design choices: (i) associative memory architecture, (ii) attentional bias objective, (iii) retention gate, and (iv) memory learning algorithm. Forgetting mechanisms are reinterpreted as retention regularization — providing a principled basis for forget gates across architectures. Three novel sequence models — Moneta, Yaad, and Memora — go beyond existing linear RNNs while maintaining fast parallelizable training. Different Miras configurations yield models with varying strengths: some excel at language modeling, others at commonsense reasoning or recall-intensive tasks. This generalizes the Titans insight: the attention-as-short-term/memory-as-long-term distinction is one instance of a broader design space where attentional bias objective and retention mechanism can be independently varied.
Inquiring lines that read this note 145
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What design and behavioral factors drive false consciousness attribution to AI? What prevents conversational agents from taking initiative in dialogue? What structural properties of attention create systematic model biases?- How does the temporal structure of attention differ between humans and AI?
- Can better attention mechanisms close the gap between human and AI frame-activation?
- Does transformer attention architecture fundamentally prevent topic-aware memory?
- Why does attention-based drift happen automatically during generation?
- Can transformer attention patterns actually prevent topic context loss in practice?
- What does attentional state look like in a static context window?
- How do attention heads separate text retrieval from internal thought representation?
- Can targeted interventions on attention heads bridge the encoding-generation gap?
- Why do transformer attention patterns show positional and sequential bias across tasks?
- How do neural memory modules extend context length beyond attention limits?
- What attentional bias objectives compete with dot product similarity for associative memory?
- Why does attention quality degrade as context length increases?
- What neural or architectural mechanism allows selective override of frequency effects?
- Why does transformer attention architecture undermine stickiness in model behavior?
- Which attention heads are essential for maintaining factuality in sparse models?
- How does transformer attention bias toward repeated and context-prominent content?
- Do transformer architectures structurally bias models toward short-term optimization?
- How does attention sink behavior relate to internal model architecture?
- How does disentangled attention separate text from spatial reasoning?
- Why does attention concentrate on the first 25% of long input sequences?
- Does attention linearity alone explain the efficiency gains over standard transformers?
- Why do hybrid attention architectures outperform pure linear attention models?
- Why does attention excel at context retrieval but struggle with state updates?
- How do attention mechanisms fail at capturing graph structure?
- What computation remains in the attention heads that programs cannot capture?
- How does transformer attention structurally bias models toward prominent and repeated content?
- How does context collapse affect what language models can meaningfully communicate?
- How should tiny language models be architected differently than large ones?
- Why do language models fail at iterative numerical optimization despite scale?
- Do pretrained language models carry reusable computational scaffolding for length handling?
- How do parameter scaling and latent vectors interact in language models?
- Which specific capabilities must AI develop beyond current language model abilities?
- Can context compression preserve what matters without introducing bias?
- How does completion-driven KV pruning differ from attention-based cache management?
- Why do language models ignore condensed memory even when it is the only memory?
- How do memory hierarchies and compression reduce context management demands?
- How does reducing activation precision further extend context length?
- Do retrieval-augmented memory systems actually solve the compartmentalization problem?
- Can continuum memory systems prevent catastrophic forgetting in neural networks?
- How should memory consolidation timing differ across multiple timescales?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- Can fast-slow separation improve both memory and generation in language models?
- Do decoder-only models have inherent architectural limits for non-sequential information?
- How do retention gates regularize forgetting across different sequence model architectures?
- Why does recency-based recall outperform semantic similarity for episodic memory?
- What persistent memory architectures best support storing precomputed inferences across sessions?
- What makes multi-session context tracking harder than single-turn underspecification problems?
- How does context budget create tradeoffs between memory and skills?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- How do memorization and attention map onto different memory systems?
- How does separating local and global context dependencies affect long-context performance?
- Can memory primitives become first-class design objects like computation sparsity?
- Why do hybrid memory systems outperform single-tier AI architectures?
- Can offline recurrent passes replicate sleep-based memory consolidation in AI?
- Can memory-based adaptation and gradient fine-tuning operate on complementary timescales?
- How do complementary learning systems explain the need for fast and slow consolidation?
- Do long-term memory modules outperform consolidation into fast weights?
- How should memory systems split between short-term and long-term storage?
- Does recurrent memory or gist compression work better for ultra-long context?
- Can recurrent state mechanisms process longer sequences than attention-based working memory approaches?
- How do adaptive memory modules compare to feedback-based working memory for long context?
- Why does attending to own latents work better than bolted-on external memory stores?
- How do fixed recurrent states trade off copying accuracy for filtering ability?
- Can adaptive memory modules combine long-term filtering with short-term attention benefits?
- What task profiles favor recurrent filtering over scaled attention mechanisms?
- How does externalized state affect the long-context bottleneck in language models?
- How do recurrent memory systems handle ultra-long context differently than attention?
- What are the concrete efficiency gains of linear-attention state-space models?
- Can fixed-size latent states losslessly store arbitrary input context?
- Can compressed long-term memory outperform fixed-window token retention?
- How do compressed persistent memory states inside networks compare to attention for long context?
- Why do large language models follow user drift instead of maintaining topic focus?
- Can language models execute iterative numerical methods in latent space?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- What makes sparse models inefficient to train and deploy at scale?
- Why do cross-product features memorize better than dense embeddings?
- Does conditional memory reduce computation alongside conditional sparsity?
- Why do longer sequences tolerate higher sparsity than shorter ones?
- What mechanisms cause short contexts to degrade more under aggressive sparsity?
- Why do hybrid memory and compute sparsity outperform pure parameter scaling?
- Can sparse attention methods be designed specifically for multi-hop reasoning tasks?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Can attention linearity achieve similar efficiency gains as weight quantization?
- Does static per-token sparsity repeat the fixed-budget mistake at short sequences?
- Can retrofitted sparse attention ever match jointly-trained sparse attention?
- What makes sparse attention more reliable for long-context retrieval?
- How much does memorization capacity limit a model's ability to learn new information?
- Can data pruning strategies exploit the finite nature of memorization capacity?
- How can memory shift from a passive datastore to an actively trained component?
- What mechanism transfers explicit memories into parametric model weights?
- Can zero-weight drift through external memory replace parameter plasticity entirely?
- Can models consolidate context into weights during idle offline phases?
- How can a forgetting policy preserve rare knowledge while preventing over-generalization?
- What is the theoretical capacity limit before memorization saturates?
- What capacity limits does the memory model face as corpus grows?
- When does training a memory model beat RAG or fine-tuning?
- Can externalized memory and skills replace model scaling?
- Can native memory procedures acquired through training handle stale or incorrect cached information?
- Does moving memory outside model weights avoid the limitations of in-weight retention?
- Can in-weight memorization scale beyond model parameter count limits?
- How does circuit complexity limit which grammatical structures transformers can acquire?
- Can latent recurrence overcome the trainability costs of depth?
- Can latent recurrence achieve the depth that standard transformers cannot?
- How does per-token adaptive compute improve efficiency in recurrent reasoning?
- How do sleep-time and post-completion methods reduce inference latency?
- How does conditional memory compare to conditional computation for efficiency?
- Does the prediction unit shape what language models actually learn?
- How do model priors enable targeted context queries without full attention?
- Does attention bias explain grounding failure in language models?
- What other internal model decisions beyond attention could be optimized directly?
- Can next-token prediction train models to optimize for communication efficiency?
- Do attention scores predict which tokens will be pruned first?
- How do retrieved memories differ from decision-context passages for prediction?
- Does including full context always degrade memory retrieval quality in practice?
- Can a model be strong at MMLU but weak at long-horizon tasks?
- How should benchmark design account for task-dependent sparsity tolerance differences?
- How does accumulated context history degrade iteration quality in long-horizon tasks?
- How do memory relevance filters fail to prevent performance degradation?
- What computational costs does closed-loop memory refinement introduce?
- Why does LLM memory consolidation regress below no-memory baselines?
- Can episodic raw memory outperform consolidated summaries in practice?
- How does the hippocampus bind disparate elements without storing everything itself?
- Why is consolidation quality the binding constraint in neural memory systems?
- Can autoencoders act as associative memory systems like Hopfield networks?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- Can test-time scaling compound through memory consolidation into a new scaling law?
- Can sleep-time compute reduce latency demands during model inference?
- What limits the capacity of context-based fast adaptation channels?
- How much does sliding-window augmentation improve single-session modeling?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can models reason without generating visible thinking tokens?
Explores whether intermediate reasoning must be verbalized as text tokens, or if models can think in hidden continuous space. Challenges a foundational assumption about how language models scale their reasoning capabilities.
complementary architectural innovation: depth-recurrent reasoning vs. breadth-adaptive memory
-
Can models reason without generating visible thinking steps?
Do machine reasoning systems actually require verbalized chains of thought, or can they solve complex problems through hidden computation? This challenges how we measure and understand reasoning.
alternative to verbalized reasoning
-
Can models precompute answers before users ask questions?
Most LLM applications maintain persistent state across interactions. Could models use idle time between queries to precompute useful inferences about that context, reducing latency when users actually ask?
Titans' persistent neural memory is a natural substrate for sleep-time compute: the memory module can store precomputed inferences between interactions, and sleep-time processing can update the memory with anticipated query patterns; both exploit statefulness to reduce per-query cost
-
Can latent thought vectors scale language models beyond parameters?
Explores whether explicit latent thought vectors with dual-rate learning create new scaling dimensions independent of model size. This matters because it suggests alternatives to simply building larger models.
LTMs implement fast-slow dynamics at the generation level (dual-rate learning of thought vs token vectors) complementing Titans' fast-slow at the memory level (attention as short-term, neural memory as long-term)
-
Can lookup memory and computation work together better than either alone?
Mixture-of-Experts handles dynamic logic, but static knowledge might need a different mechanism. Can a hybrid approach combining conditional computation with fast lookup outperform pure sparse models?
Engram adds static O(1) N-gram lookup as a third memory primitive complementing Titans' neural memory; together they suggest memory architecture has multiple co-equal axes (neural-adaptive, static-lookup, attention) rather than a single hierarchy
-
Has memory architecture replaced parameter count as the scaling frontier?
Late-2025 research suggests the field's next major efficiency gains come from restructuring how models store and use experience rather than simply making them larger. Three convergent signals point to this shift.
Titans/Miras is one of three convergent signals (alongside ReasoningBank MaTTS and Engram U-curve) that memory-architecture restructuring has displaced parameter count as the active scaling axis
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Titans: Learning to Memorize at Test Time
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
- Language Models Need Sleep
- Repeat After Me: Transformers are Better than State Space Models at Copying
- Memorization and Knowledge Injection in Gated LLMs
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Jamba: A Hybrid Transformer-Mamba Language Model
Original note title
neural memory modules that adaptively memorize surprising tokens complement attention as long-term vs short-term memory — scaling to 2M+ context