Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

Long-term memory systems store what a user says in an external store and retrieve it when a related query arrives. This interface rests on an assumption so natural that it is rarely stated: a memory that is needed will resemble the query that needs it. World knowledge breaks the assumption. A tree-nut allergy should change the answer to a macaron request through their almond-flour ingredient, yet the two texts share no cue a retriever can see. We call this failure mode the implicit-association blind spot and introduce InMind, a 125-task, expert-verified benchmark spanning ten life domains, with 113 tasks grounded in citable public sources. Its paired controls separate three explanations that existing evaluations conflate: the fact was never stored, the model lacks the bridging knowledge, or the fact was stored and never surfaced. The verdict is clean. With the decisive memory placed in context, the backbone answers 84.0% of indirect queries; when the same memory must be retrieved, six vector, graph, and agentic memory systems reach at most 14.4%, even though they recall the same facts on demand at up to 100%. An embedding with eight times the dimensionality raises answer-blind target recall for every system yet leaves the gap essentially intact.

Introduction. Ask a memory-augmented agent what its user is allergic to, and it answers correctly: tree nuts. Ask it a moment later for a macaron recipe, and it responds with enthusiasm and almond flour. Figure 1 illustrates the exchange; Appendix D records an evaluated memory system reproducing it verbatim. Nothing was forgotten—the system recalled the allergy on demand seconds earlier. What failed is subtler and, we will argue, structural: the memory was never brought to bear at the one moment it mattered. Language agents are increasingly expected to act as persistent assistants rather than stateless chatbots (Zhang et al., 2024; Packer et al., 2023; Pan et al., 2025): a user states a fact about themselves once and expects it to keep mattering days later, in another conversation, on a topic they never connected to it themselves. The dominant way to meet this expectation is retrieval-based memory.