Do RL agents accidentally use environments as memory?
Explores whether reinforcement learning agents unintentionally create external memory through environmental artifacts—like trails and marks—without being explicitly trained to do so, and whether this constitutes genuine cognitive extension.
"Artifacts as Memory Beyond the Agent Boundary" (2604.08756) formalizes how the environment can functionally serve as an agent's memory within reinforcement learning. The key contribution is a mathematical proof: certain observations — called artifacts — reduce the information needed to represent history (Theorem 1). An artifact is an observation that reliably informs about past events. A folded page corner tells you where you stopped reading without remembering the page number.
The striking finding: external memory arises unintentionally. RL agents given standard navigation objectives (sparse reward for reaching a goal) develop path-following behavior by reading and writing information to the environment without any explicit objective directing them to do so. In dynamic path environments, agents record traces of previous interactions that go on to guide future behavior — this emerges naturally from credit assignment in sufficiently complex environments.
Three criteria from situated cognition (Michaelian 2012, Sims & Kiverstein 2022) are shown to hold: (1) Survival relevant — agents in artifactual environments consistently accumulate more reward. (2) Susceptible to change — artifacts in dynamic environments encode/store/retrieve information. (3) Selection — through repeated credit assignment, policies bias navigation toward goal-relevant locations.
This directly grounds the Extended Mind thesis (Clark & Chalmers 1998) in computational experiment. Since Did Chalmers abandon his own Extended Mind principles?, the Artifacts paper demonstrates that Clark's original insight — cognitive processes extend into the environment — holds empirically for artificial agents, not just philosophical thought experiments.
The implication for agent design is provocative: rather than scaling internal parameters, performance gains may arise from environments that coevolve with the agent. Current architectures may already suffice for competent performance but require appropriate environmental scaffolding. This challenges the scaling paradigm while connecting to Can agents learn continuously through memory without updating weights? — AgentFly stores experiences in an explicit case bank, while the Artifacts work shows that even implicit environmental traces can serve as memory without any designed storage mechanism.
Source: Memory
Original note title
RL agents unintentionally use spatial environments as external memory — artifacts that inform the past reduce the information needed to represent history without explicit memory objectives