Know It, Act on It: Investigating Memory Utilization in LLM Personalization

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapyrelated preferences, where failures to act carry the greatest real-world stakes 1.

Introduction. Large Language Model (LLM) agents are rapidly evolving from stateless tools into personalized longterm companions capable of sustained interaction with users over days, weeks, and months (OpenAI, 2025b; Meta, 2025). A core capability driving this transition is memory: the ability to store, organize, and retrieve information from past interactions (Zhang et al., 2025b). Both industry and academia are actively equipping agents with increasingly sophisticated memory architectures, from commercial systems such as Chat- GPT’s persistent memory (OpenAI, 2024c) and Claude’s memory feature (Anthropic, 2025) to open-source frameworks such as Mem0 (Chhikara et al., 2025), Letta/MemGPT (Packer et al., 2024), and knowledge-graph-based systems like Zep (Rasmussen et al., 2025) and HippoRAG (Gutiérrez et al., 2025). These systems have demonstrated strong performance on benchmarks that test factual recall, multi-hop reasoning, and long-range understanding (Wu et al., 2025; Hu et al., 2026a). But remembering information is only the first step.

Discussion / Conclusion. This paper examines whether memory-augmented LLMs and long-context baselines can translate recalled preferences into appropriate behavior. Evaluating 16 systems across 1,000 preferences and three expression levels, we find that agents often achieve high accuracy on the Know test but perform poorly on the Act test. This gap varies by expression strength and preference type, with health and emotional domains proving especially challenging. Failure attribution further indicates that errors arise from retrieval, comprehension, and application, with comprehension failures dominating overall. These findings suggest that future personalization systems should move beyond simply storing and recalling user information, and instead focus on ensuring that personal memories are selectively retrieved, correctly interpreted, and reliably translated into context-appropriate behavior.