Do RL agents accidentally use environments as memory?
Explores whether reinforcement learning agents unintentionally create external memory through environmental artifacts—like trails and marks—without being explicitly trained to do so, and whether this constitutes genuine cognitive extension.
"Artifacts as Memory Beyond the Agent Boundary" (2604.08756) formalizes how the environment can functionally serve as an agent's memory within reinforcement learning. The key contribution is a mathematical proof: certain observations — called artifacts — reduce the information needed to represent history (Theorem 1). An artifact is an observation that reliably informs about past events. A folded page corner tells you where you stopped reading without remembering the page number.
The striking finding: external memory arises unintentionally. RL agents given standard navigation objectives (sparse reward for reaching a goal) develop path-following behavior by reading and writing information to the environment without any explicit objective directing them to do so. In dynamic path environments, agents record traces of previous interactions that go on to guide future behavior — this emerges naturally from credit assignment in sufficiently complex environments.
Three criteria from situated cognition (Michaelian 2012, Sims & Kiverstein 2022) are shown to hold: (1) Survival relevant — agents in artifactual environments consistently accumulate more reward. (2) Susceptible to change — artifacts in dynamic environments encode/store/retrieve information. (3) Selection — through repeated credit assignment, policies bias navigation toward goal-relevant locations.
This directly grounds the Extended Mind thesis (Clark & Chalmers 1998) in computational experiment. Since Did Chalmers abandon his own Extended Mind principles?, the Artifacts paper demonstrates that Clark's original insight — cognitive processes extend into the environment — holds empirically for artificial agents, not just philosophical thought experiments.
The implication for agent design is provocative: rather than scaling internal parameters, performance gains may arise from environments that coevolve with the agent. Current architectures may already suffice for competent performance but require appropriate environmental scaffolding. This challenges the scaling paradigm while connecting to Can agents learn continuously from experience without updating weights? — AgentFly stores experiences in an explicit case bank, while the Artifacts work shows that even implicit environmental traces can serve as memory without any designed storage mechanism.
Inquiring lines that read this note 45
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What makes agent memory systems durable and reusable across sessions?- Can environmental scaffolding replace internal memory scaling in agent design?
- Can episodic memory of UI traces improve open-world agent adaptation?
- What distinguishes formation, evolution, and retrieval as separate memory dynamics?
- How does structured environment-side state reduce multi-turn agent failure better than transcript replay?
- What makes agent-scale memory persist beyond individual sessions or people?
- How does credit assignment drive agents to write information into environments?
- What makes exploration and reflection rewards verifiable in agentic environments?
- Does intentionally varying environment properties isolate causal effects on agent performance?
- How do agent actions change state that reward procedures later read?
- Can an agent change reward-path state through actions during evaluation?
- Can simple intrinsic reward signals emerge as effective drivers of complex capability in agents?
- What counts as genuine memory under the Extended Mind thesis?
- Does functional integration determine cognitive system boundaries?
- How do implicit world models and self-reflection operationalize consequence-based learning?
- Does input surprise drive the implicit recognition of on-policy context?
- What happens when agents interact with environments and learn from their own mistakes?
- Can simulation fidelity limit what agents learn from trained world models?
- Why do environments authored once encode only what their builders imagined?
- Can agents learn beyond the boundaries of their training data curators?
- How do world models create indirect causal grounding without physical environment contact?
- Is reward propagation in RL formally dual to cause inference in memory?
- Can episodic memory alone enable learning without parameter updates?
- How can memory shift from a passive datastore to an actively trained component?
- What determines whether accumulated state generalizes spuriously across continual learning domains?
- What details do high-level trajectory abstractions lose that state-grounded recall preserves?
- How does continuous implicit memory formation differ from explicit memory encoding?
- What separates artifact recall from persistent memory commitment in agents?
- Can agents learn from their own experience without fine-tuning through episodic memory?
- What shapes of memory help frozen agents improve without retraining?
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Artifacts as Memory Beyond the Agent Boundary
- Rethinking Memory as Continuously Evolving Connectivity
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
Original note title
RL agents unintentionally use spatial environments as external memory — artifacts that inform the past reduce the information needed to represent history without explicit memory objectives