Metis: Memory Foundation Model

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training.

Introduction. In recent years, large foundation models have achieved rapid development, demonstrating significant performance across many aspects, such as language modeling [1, 2], code generation [3–5], and complex reasoning [6–8]. This provides a solid foundation for constructing AI agents, which enables them to handle more complex tasks. Beyond the reasoning capabilities of foundation models, memory is another critical capability of AI agents, responsible for retaining past information and leveraging it to support future inference [9, 10]. In most previous works, memory is implemented However, external memory suffers from several limitations presented in Figure 1. First, external memory is decoupled from backbones with separated targets and processing stages [11, 13]. External memory typically aims to construct an informative context as input, and backbones only perform conditional language modeling over the constructed context. Therefore, external memory may not provide the most useful information to support the backbone inference, and the backbone may not utilize the memory optimally.

Discussion / Conclusion. In this paper, we introduce memory foundation models and provide formal definitions of native memory based on the memory state and memory procedures. Based on this formulation, we propose Metis, the first prototype of memory foundation models. We introduce Metis blocks composed of local memory blocks and hyper memory blocks, enabling the model to maintain compact dense memory states across interaction steps and to update them according to the current input and generated response. We further construct a memory-specific dataset from public benchmarks and design a mid-training framework with memory reconstruction, memory operation, and regularization objectives. Our experiments verify the effectiveness of Metis and analyze its behavior from multiple perspectives. Despite these promising results, Metis is still an early step toward memory foundation models.