Can multimodal knowledge graphs answer questions that flat retrieval cannot?

Long documents like books mix text and figures across hundreds of pages, and flat chunk-based retrieval can find local matches but cannot answer questions that require synthesizing entities across the whole work. MegaRAG builds a multimodal knowledge graph as preprocessing: it extracts entities and relations from both prose and visuals, organizes them hierarchically, and uses the graph during retrieval and generation. This means a question about how a character in chapter 1 relates to an event in chapter 18 traverses a graph rather than searching across chunks that may or may not co-retrieve.

The hierarchy matters as much as the multimodality. A flat knowledge graph over a book is unwieldy; the hierarchical structure gives the system levels of abstraction so it can answer high-level questions ("what is the book about") at one level and detailed questions ("what happened on page 273") at another, without rebuilding the graph for each query. Multimodal extraction means that figures, diagrams, and images become first-class graph nodes connected to the text that references them, which supports answering questions about visual content in a way text-only RAG cannot.

The architectural cost is upfront: building a hierarchical multimodal knowledge graph for a book is expensive compared to embedding chunks. The payoff is that the graph is reusable across many queries and supports a class of question — global, cross-chapter, multimodal — that flat retrieval simply cannot answer. The principle generalizes to any long-form multimodal corpus where global synthesis is the normal mode of querying. Can community detection enable RAG systems to answer global corpus questions? applies the same upfront-graph trade-off to text-only corpora; MegaRAG extends it to multimodal long-form.

Source: 12 types of RAG