Can agents learn preferences by watching rather than asking?
Explores whether multimodal agents can build accurate preference models through continuous observation of user behavior, without explicit instruction, by organizing memory around entities and separating concrete events from derived knowledge.
M3-Agent (2508.09736) proposes a multimodal agent framework where long-term memory is organized as an entity-centric graph, with two types of memory generated from continuous video-stream perception:
Episodic memory records concrete events: "Alice takes the coffee and says, 'I can't go without this in the morning.'" Semantic memory derives general knowledge: "Alice prefers to drink coffee in the morning." Information about the same entity — face, voice, textual knowledge — is connected in graph format, incrementally established as the agent extracts and integrates semantic memory.
The architecture runs two parallel processes: (1) memorization, which continuously perceives real-time multimodal inputs to construct and update long-term memory; and (2) control, which interprets external instructions, reasons over stored memory, and executes tasks. This dual-process design means the agent can hand you coffee without asking "coffee or tea?" — it has already formed a memory of your preferences through observation.
The entity-centric graph structure is the key architectural choice. Unlike flat memory stores or conversation-history retrieval, entity-centric organization enables cross-modal association: a person's face links to their voice links to their preferences. This mirrors how Does abstract preference knowledge outperform specific interaction recall? — but M3-Agent captures both episodic and semantic layers and connects them through entity nodes rather than discarding one.
The dual episodic/semantic distinction also echoes the hierarchical knowledge source in Can reasoning systems maintain memory across retrieval cycles?, where ComoRAG builds veridical, semantic, and episodic layers — but M3-Agent applies this to continuous multimodal perception rather than text retrieval.
Since How should agents decide what memories to keep?, M3-Agent's memorization process operates as continuous implicit memory — always running, always extracting, rather than waiting for explicit recognition of importance.
Inquiring lines that read this note 50
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can reward models capture diverse human preferences without excluding minority populations?- Does learning community preferences as training rewards operationalize prediction without participation?
- Can curiosity rewards about user type complement general social motivation frameworks?
- How should preference channels from historical sessions inform unified policy learning?
- Does sequential structure within sessions complement cross-session preference channels?
- Why do abstract semantic memories outperform specific interaction histories for journey discovery?
- How do implicit signals like clicks capture preference more reliably than explicit ratings?
- How should systems learn what each meeting participant actually cares about?
- Can abstract preference summaries substitute for specific user interaction history?
- How do entity graphs connect faces, voices, and preferences across modalities?
- Can elicited user responses measure true preferences or just elicitation artifacts?
- How should historical preferences be weighted when users change their stated intent?
- Can users detect and correct an AI's mental model of their preferences?
- How does active learning reduce queries needed for user preference inference?
- Can input-only training encode user preferences without task-specific labels?
- Can rich environment feedback replace human preference labels entirely?
- Can light human signals steer already-learned behavior without preference labels?
- What explicit concept annotations would improve cross-concept preference reasoning?
- Can curiosity-driven dialogue incrementally discover user interest journeys in real time?
- Can users articulate what they want before AI helps them discover it?
- Why might text-only interfaces underestimate agent preference elicitation capabilities?
- How can agents detect whether users are willing to follow their topic guidance?
- When should agents accommodate user preferences over their own goals?
- Can agents balance goal-driven proactivity with user preference alignment?
- How can insert-expansion techniques help users discover their own preferences?
- What multi-turn reward structures would encourage active intent discovery?
- How can agents learn user preferences during conversation without pre-calibration?
- Can agents learn user intent from unlabeled video without text labels?
- Can multimodal architectures successfully integrate vision without replicating past failures?
- Can curiosity-driven personalization work better than pre-conversation preference elicitation?
- Does semantic memory improve AI personalization more than episodic memory?
- When does combining episodic and semantic memory reduce personalization performance?
- What distinguishes genuine user preferences from similar-user preferences in sparse data?
- Why does semantic memory abstraction outperform raw episodic recall for personalization?
- Should abstract preference knowledge replace specific interaction recall in personalization?
- Why does abstract preference knowledge outperform specific interaction recall in personalization?
- Can multimodal agents use entity-centric graphs within this three-axis framework?
- Can relationship dynamics between user and agent be tracked as distinct memory?
- Do memory architectures genuinely close the gap between knowing and acting on preferences?
- Can agents learn from their own experience without fine-tuning through episodic memory?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can three axes replace the short-term long-term memory split?
Does breaking agent memory into forms, functions, and dynamics provide a clearer framework than the traditional short-term/long-term distinction? This matters because current agent-memory literature lacks a unified vocabulary, making comparison between systems nearly impossible.
M3-Agent's episodic/semantic split is a specific instantiation along the *functions* axis; its parallel memorization+control processes are an instantiation of the *dynamics* axis (formation operator runs continuously, retrieval operator is goal-triggered)
-
Can brain memory systems explain how LLMs should store knowledge?
This explores whether the brain's three-tier memory architecture—neocortex, hippocampus, and prefrontal cortex—maps onto transformer weights, external knowledge stores, and agentic state. Understanding this mapping could reveal which AI memory problems each tier solves and which it cannot.
M3-Agent's entity-centric graph functions as a hippocampal-style index that binds disparate elements (face, voice, knowledge) of an entity across modalities — the AI analog of how the hippocampus binds elements of an episode across cortical regions
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Preference Discerning with LLM-Enhanced Generative Retrieval
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
- Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time
- On Generative Agents in Recommendation
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- User-Centric Conversational Recommendation with Multi-Aspect User Modeling
Original note title
multimodal agents require entity-centric memory graphs that separate episodic events from semantic knowledge — parallel memorization and control processes mirror human cognitive architecture