Can memory primitives become first-class design objects like computation sparsity?
This explores whether memory — storage, lookup, consolidation — can be designed as its own architectural axis, the way conditional computation (MoE, sparsity) already is, rather than treated as a passive byproduct of attention.
This explores whether memory can be promoted to a first-class design knob — something you allocate and tune deliberately — instead of being whatever the context window happens to hold. The corpus answers with a fairly clear yes, and the sharpest evidence is the one paper that treats the two as siblings: Engram pairs an O(1) N-gram lookup with Mixture-of-Experts routing and finds a *U-shaped scaling law* where splitting your parameter budget between lookup-memory and compute-routing beats spending it all on either Can lookup memory and computation work together better than either alone?. That framing is the heart of your question: memory isn't a fallback for when compute runs out, it's a complementary axis you co-design alongside sparsity, and the optimum lives in the balance.
Once you start treating memory as a designed object, you find architectures that give it its own machinery. Titans splits the model into short-term attention (quadratic, expensive) and a separate neural memory module that decides *what's worth storing* by prioritizing surprising tokens — a deliberate memory subsystem that scales past 2M tokens without the quadratic penalty Can neural memory modules scale language models beyond attention limits?. At the agent level, DeepAgent's 'memory folding' does the same thing one level up: it consolidates interaction history into typed schemas (episodic, working, tool) so memory becomes a structured, queryable resource rather than an ever-growing transcript Can agents compress their own memory without losing critical details?. And the Thread Inference Model treats working memory as something you actively prune — rule-based KV-cache eviction inside recursive subtask trees — letting one model reason past its context limit even while discarding 90% of the cache Can recursive subtask trees overcome context window limits?.
The most interesting twist is a paper that argues the real constraint was never memory capacity at all — it's the *compute* needed to fold evicted context into the model's fast weights, a consolidation step that improves with more passes like a test-time scaling law Is long-context bottleneck really about memory or compute?. Read alongside Engram, this completes the symmetry: memory and computation aren't just complementary, they trade into each other. Storing something cheaply later costs compute to consolidate; spending compute now saves you from having to remember.
There's also a quieter lesson from the sparsity side of the corpus, which suggests memory-as-design-object may be partly *emergent* rather than fully engineered. Networks learn dense activations for familiar data and default to sparse ones for the unfamiliar, without anyone training that behavior in Is representational sparsity learned or intrinsic to neural networks? — and that learned sparsity is concrete enough to exploit as a tool, e.g. ordering few-shot examples from sparse-hard to dense-easy for free gains Can representation sparsity order few-shot demonstrations effectively?. The implication for memory: the primitive you want to make first-class may already exist as a measurable signal inside the model, waiting to be allocated rather than invented.
The thing you might not have expected to learn: making memory a first-class object doesn't just add capacity — it can *replace structure*. The Thread Inference Model shows a single model with well-designed working memory standing in for an entire multi-agent system, and Titans shows a memory module letting one transformer outperform RNNs at long range. Designed memory isn't a storage upgrade; it's an architectural simplifier.
Sources 7 notes
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
The Thread Inference Model demonstrates that reasoning structured as recursive subtask trees with rule-based KV cache pruning sustains accurate reasoning beyond context limits, even when manipulating 90% of the cache. This enables single models to replace multi-agent systems by handling full recursive reasoning internally.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
Sparsity-Guided Curriculum In-Context Learning uses last-layer activation sparsity to order demonstrations from sparse (harder) to dense (easier), yielding considerable performance improvements. This approach requires no external difficulty labels and works across diverse in-context learning tasks.