How should we actually evaluate agent memory systems?

Agent memory has quietly become a full data-management system — it stores, extracts, retrieves, routes, updates, consolidates, and governs the lifecycle of information across long-horizon execution. But evaluation never caught up: benchmarks still score the whole apparatus by end-to-end task success (F1, BLEU), which treats memory as a monolithic black box. This paper argues that framing is what hides the questions that actually matter in production — operational cost, the architectural trade-offs between memory modules, and robustness when the stored knowledge keeps changing.

The move is to decompose memory into four core modules — representation and storage, extraction, retrieval and routing, and maintenance — and evaluate 12 systems (Mem0, Letta, Zep, A-MEM, MemoryBank, etc.) module by module across 11 datasets. This is the memory analogue of what the vault already argues about working context: since How should agent memory split across time scales?, granularity and module boundaries are themselves the design surface, not just the content stored. And it grounds the practical claim that since Is agent memory a storage problem or a connectivity problem?, measuring "did the task succeed" tells you nothing about which module failed. A task-success score cannot distinguish a bad extractor from a bad router — which means it cannot guide design. The data-management lens restores that diagnostic resolution.