How should agents decide what memories to keep?

Agent memory management — how to transfer information between the LLM's context window and external storage — decomposes into two fundamentally different paths that parallel the human explicit/implicit memory distinction:

Explicit memory (hot path): The agent autonomously recognizes important information during conversation and decides to remember it via tool calling. This mirrors human conscious storage (episodic and semantic memory). The advantage: context-sensitive importance assessment — the agent can judge what matters based on the current conversational state. The challenge: implementing robust importance recognition is hard. What counts as "important enough to remember" depends on the user, the task, and future needs that can't be predicted.

Implicit memory (background): Memory management is programmatically defined at specific trigger points:

• After a session — batch process the entire conversation post-session
• At periodic intervals — transfer session data to long-term memory on a schedule (for long-running conversations)
• After every turn — real-time updates (highest fidelity, highest cost)

The CoALA vs Letta taxonomy debate reveals a deeper design question about working memory. CoALA treats working memory as a single category. Letta splits it into message buffer (recent messages from current conversation) and core memory (specific information the agent self-manages, like user's birthday). This split matters because core memory is agent-curated while the message buffer is conversation-driven — they have different update mechanisms and different information types.

Neither taxonomy cleanly maps human memory types to agent implementations:

• CoALA's semantic memory ≈ Letta's archival memory (explicitly stored knowledge in external DB)
• But CoALA's procedural/episodic memory ≠ Letta's recall memory (raw conversation history)
• CoALA doesn't include raw conversation history in long-term memory at all

The six core components of agent memory management — generation, storage, retrieval, integration, updating, and deletion (forgetting) — each face the explicit/implicit design choice independently. You might generate memories explicitly (hot-path recognition) but delete them implicitly (TTL-based expiration). The design space is combinatorial.

Since Can AI agents learn when they have something worth saying?, the inner thoughts mechanism could serve as the importance recognition layer for explicit hot-path memory — the agent's continuous covert thoughts identify what's worth remembering, solving the "what matters" problem.

Source: Memory