Can encoder models compete with large language models for retrieval?
As decoder-only LLMs dominate NLP, do modernized encoders like BERT still offer efficiency and quality advantages for retrieval, classification, and embedding tasks at scale?
The narrative around large language models makes it easy to assume decoder-only generation superseded everything. ModernBERT is a corrective: for the non-generative half of NLP — retrieval, reranking, classification, and the embedding backbones that feed RAG — encoder-only models are still the right tool, because they carry modest inference cost and process document corpora at scale in a single forward pass rather than token by token. The problem was that BERT had seen almost no Pareto improvement since 2019 while decoder LLMs absorbed years of architecture and data-scaling advances.
The insight is that those advances transfer back: an encoder trained on 2 trillion tokens with a native 8192 sequence length and modern optimizations (rotary embeddings, FlashAttention-style efficiency) sets new state of the art on classification and both single- and multi-vector (ColBERT-style) retrieval, while running short-context inputs roughly twice as fast as DeBERTaV3 with best-in-class memory efficiency. This matters directly to the retrieval-and-routing layer of agent memory — since How should we actually evaluate agent memory systems?, the embedding model is the retrieval module, and its speed and long-context quality set the ceiling on what the whole memory system can afford to search. The broader lesson: architectural progress is not a one-way migration to decoders. The efficient-encoder branch is undervalued precisely because it does the unglamorous work that generative demos never showcase.
Inquiring lines that read this note 3
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should systems decide whether to retrieve or reason alone? Why do embedding systems fail to capture task-relevant relationships? Do language models learn genuine understanding or just surface patterns?Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How should we actually evaluate agent memory systems?
Current benchmarks score agent memory by task success alone, hiding critical design questions about cost, trade-offs, and robustness. What would evaluation reveal if we decomposed memory into its core data-management stages?
the retrieval module this encoder powers
-
Can state-space models match transformers at copying and retrieval?
Explores whether the efficiency gains of state-space models come at a fundamental cost in their ability to copy strings and retrieve exact information from context, compared to transformers.
related architecture-choice reasoning about what different backbones can afford
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- Context Embeddings for Efficient Answer Generation in RAG
- Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
- On the Theoretical Limitations of Embedding-Based Retrieval
- FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions
- Long-context LLMs Struggle with Long In-context Learning
- Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Original note title
encoder-only models remain the efficient retrieval workhorse — ModernBERT ports LLM-era training to BERT and beats decoder models on the size-per-quality tradeoff