Can small transformers trained on similarity maps replace dense retrievers entirely?

This explores whether a small transformer reading token-to-token similarity maps can do the entire retrieval job on its own. The clearest piece of evidence the corpus has on this is also the one that complicates the premise: the system in Can verification separate structural near-misses from topical matches? does train a small Transformer on token-token similarity maps, and it reliably catches "structural near-misses" that compressed-vector methods like MaxSim late interaction wave through. But it does this as a second stage — pooled-cosine dense recall runs first to pull candidates, and the small transformer then scrutinizes the full interaction pattern. It's a verifier downstream of recall, not a replacement for it.

Why not just let the small model do everything? Because the two stages are good at opposite things. A dense retriever's job is cheap breadth — scan everything, lose detail. The reason it loses detail is structural, not fixable by tuning: Where do retrieval systems fail and why? points out that embeddings measure association rather than relevance, and that embedding dimension mathematically caps how many distinct documents a vector space can even represent. A similarity-map transformer escapes that ceiling precisely because it works on uncompressed token interactions — but running that over a whole corpus instead of a recall shortlist would be ruinously expensive. So the architecture isn't a compromise; each stage covers the other's blind spot.

The corpus is also full of cautionary tales about "replace retrieval entirely" claims in general. Can long-context LLMs replace retrieval-augmented generation systems? shows long-context models can absorb RAG's job for semantic lookup — and then fail outright on structured, relational queries. Can a single model replace retrieval for long-term conversation memory? folds retrieval into a single generating model and gets an inverted-U: it beats baselines for a while, then degrades below even a no-memory baseline as reprocessing compounds errors. The pattern repeats: collapsing a two-part system into one model trades a known bottleneck for a fragile failure mode.

There's a deeper reframe worth noticing. Across these notes, the winning move is rarely "better similarity" — it's *adding a different signal on top of similarity*. Can visual similarity alone guide robot object retrieval? keeps visual retrieval but reranks by whether an action is physically executable; the verifier note keeps cosine recall but reranks by structural match. Retrieval becomes recall-plus-judgment, and the small transformer is the judgment layer. That it can be small is itself encouraging: Does depth matter more than width for tiny language models? shows sub-billion-parameter models punch well above their size when built deep-and-thin, which is exactly the regime a per-candidate verifier lives in.

So the thing you didn't know you wanted to know: the question's word "entirely" is the part the corpus quietly rejects. A small transformer on similarity maps doesn't make the dense retriever obsolete — it turns the dense retriever into a fast, lossy first pass and supplies the precise second look the vectors can't, which is a more capable system than either piece alone.