Do RL agents accidentally use environments as memory?
Explores whether reinforcement learning agents unintentionally create external memory through environmental artifacts—like trails and marks—without being explicitly trained to do so, and whether this constitutes genuine cognitive extension.
"Artifacts as Memory Beyond the Agent Boundary" (2604.08756) formalizes how the environment can functionally serve as an agent's memory within reinforcement learning. The key contribution is a mathematical proof: certain observations — called artifacts — reduce the information needed to represent history (Theorem 1). An artifact is an observation that reliably informs about past events. A folded page corner tells you where you stopped reading without remembering the page number.
The striking finding: external memory arises unintentionally. RL agents given standard navigation objectives (sparse reward for reaching a goal) develop path-following behavior by reading and writing information to the environment without any explicit objective directing them to do so. In dynamic path environments, agents record traces of previous interactions that go on to guide future behavior — this emerges naturally from credit assignment in sufficiently complex environments.
Three criteria from situated cognition (Michaelian 2012, Sims & Kiverstein 2022) are shown to hold: (1) Survival relevant — agents in artifactual environments consistently accumulate more reward. (2) Susceptible to change — artifacts in dynamic environments encode/store/retrieve information. (3) Selection — through repeated credit assignment, policies bias navigation toward goal-relevant locations.
This directly grounds the Extended Mind thesis (Clark & Chalmers 1998) in computational experiment. Since Did Chalmers abandon his own Extended Mind principles?, the Artifacts paper demonstrates that Clark's original insight — cognitive processes extend into the environment — holds empirically for artificial agents, not just philosophical thought experiments.
The implication for agent design is provocative: rather than scaling internal parameters, performance gains may arise from environments that coevolve with the agent. Current architectures may already suffice for competent performance but require appropriate environmental scaffolding. This challenges the scaling paradigm while connecting to Can agents learn continuously from experience without updating weights? — AgentFly stores experiences in an explicit case bank, while the Artifacts work shows that even implicit environmental traces can serve as memory without any designed storage mechanism.
Inquiring lines that use this note as a source 25
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Can environmental scaffolding replace internal memory scaling in agent design?
- How does credit assignment drive agents to write information into environments?
- What counts as genuine memory under the Extended Mind thesis?
- Do dynamic environments enable different kinds of agent-environment coevolution?
- How do implicit world models and self-reflection operationalize consequence-based learning?
- What happens when agents interact with environments and learn from their own mistakes?
- How do world models create indirect causal grounding without physical environment contact?
- How do neural memory modules extend context length beyond attention limits?
- Can external summarization solve exploration problems in complex real-world environments?
- Can episodic memory alone enable learning without parameter updates?
- Is reward propagation in RL formally dual to cause inference in memory?
- How do process-level rewards compare to environment-extracted next-state signals?
- Can episodic memory of UI traces improve open-world agent adaptation?
- What details do high-level trajectory abstractions lose that state-grounded recall preserves?
- Does functional integration determine cognitive system boundaries?
- Does environment stochasticity force models to generalize better across trajectory variations?
- How does trajectory burstiness compare to other structural properties that shape emergent capabilities?
- How do trajectory quality and memory hygiene differ as evaluation metrics?
- What distinguishes formation, evolution, and retrieval as separate memory dynamics?
- How can memory shift from a passive datastore to an actively trained component?
- Does input surprise drive the implicit recognition of on-policy context?
- What makes exploration and reflection rewards verifiable in agentic environments?
- How does continuous implicit memory formation differ from explicit memory encoding?
- What separates artifact recall from persistent memory commitment in agents?
- What hidden signals in agent logs reveal about frontier capability beyond pass-fail outcomes?
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Artifacts as Memory Beyond the Agent Boundary
- Rethinking Memory as Continuously Evolving Connectivity
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Large Language Model Agents Are Not Always Faithful Self-Evolvers
Original note title
RL agents unintentionally use spatial environments as external memory — artifacts that inform the past reduce the information needed to represent history without explicit memory objectives