Can describing images in text improve zero-shot recognition?

SignRAG performs road sign recognition without training a sign-recognition model. The pipeline is: a vision-language model produces a textual description of the sign image, that description is used to retrieve similar known sign designs from a vector database, and an LLM reasons over the candidates to identify which one matches. The architecture treats recognition as a retrieval-and-reason task rather than a classification task.

The methodological move worth keeping is the description-as-bridge step. Instead of computing image embeddings directly and retrieving by visual similarity (which is brittle when images differ in lighting, angle, and resolution), the VLM converts the image into a structured textual description that is far more robust to those variations. Retrieval then happens in text space against a database of known sign descriptions, which sidesteps the fragility of cross-domain visual embedding similarity. This is the visual analogue of Why do queries and documents occupy different embedding spaces? — both bridge a representational gap by passing through a text intermediate.

The general pattern — VLM description, text-space retrieval, LLM reasoning — generalizes well beyond road signs to any recognition task where the target vocabulary is closed and well-documented but visual variation in queries is high. It is a way of getting zero-shot transfer that depends on the VLM and LLM rather than on any task-specific training, and the key insight is that natural-language description is a better bridge between noisy queries and clean references than direct visual embedding. The same pattern of describing-then-retrieving anchors Can you adapt retrieval models without accessing target data? in the language-only setting.