Do hierarchical retrieval architectures outperform flat ones on complex queries?
Explores whether separating query planning from answer synthesis into distinct architectural components improves performance on multi-hop retrieval tasks compared to unified single-pass approaches.
HierSearch separates two functions that flat retrieval architectures conflate: deciding what to search for (query planning) and deciding what the answer is (answer synthesis). The finding is that these functions interfere with each other when combined, and separating them improves multi-hop query performance.
The interference mechanism: in a flat architecture, the model must simultaneously track what it is looking for, what it has found, and how the findings combine into an answer. Multi-hop queries require multiple retrieval rounds with intermediate synthesis steps — each round's findings must inform the next round's query while also contributing to the final answer. When one model component handles all of this, it loses coherence across the chain. The hierarchical architecture assigns query planning to one component and answer synthesis to another, letting each specialize.
This has implications beyond deep research. The same interference between planning and execution is well-documented in agent design: models that plan and execute simultaneously produce worse plans and worse execution than models where these are separated. HierSearch is the retrieval-specific confirmation of a general architectural principle.
The structural finding also has a connection to How do readers track segments, purposes, and salience together? — that is the cognitive architecture problem HierSearch solves at the system level. The discourse-level problem (tracking segments + purposes + salient objects in parallel) is equivalent to the retrieval-level problem (tracking query intent + retrieved evidence + synthesis state in parallel). Architecturally separating these reduces the tracking burden.
LogicRAG extends the hierarchical principle by making the query planning step structurally explicit: it decomposes the query into a directed acyclic graph (DAG) of subproblems at inference time, then resolves them in topological order. Where HierSearch separates planning from synthesis at the system level, LogicRAG implements the planning step as a structured dependency graph at the query level. The result: query-adaptive logic structures without corpus pre-processing cost. See Can query-time graph construction replace pre-built knowledge graphs?.
Inquiring lines that read this note 100
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does intelligent routing among smaller models outperform training larger models?- What makes query complexity a better routing signal than response quality?
- Can hierarchical vector routing reduce context overhead while maintaining tool coverage?
- Can embedding-cluster routing outperform a single frontier model?
- How do hierarchical architectures improve multi-hop query performance?
- How does ranking-aligned summarization compare to aspect-controlled generation methods?
- Does parallel retrieval outperform sequential search chains at test time?
- What makes proactive tool retrieval better than single-round semantic matching?
- How does hierarchical query planning versus flat prompting affect multi-source retrieval?
- Why does selective context retrieval outperform including all historical information?
- Do single-step retrieval systems with sophisticated synthesis qualify as deep research?
- What makes web retrieval more effective than static knowledge bases?
- How does query planning as a separate step improve multi-hop retrieval coherence?
- How do hierarchical query planning architectures improve multi-hop retrieval?
- Can step-level rewards improve training of agentic retrieval systems?
- Should production CRS systems combine multiple retrieval strategies in a hybrid approach?
- Can parallel retrieval chains avoid the context consumption problem?
- Why do question types determine retrieval and decomposition strategy in QA?
- What limits exist on retrieval budget during inference?
- Can retrieval strategies drive both draft refinement and new research question generation?
- How do retrieval heads interact with layer-level separation of knowledge and reasoning?
- Can tree search improve question generation the way it improves reasoning?
- How does reflection-based query refinement differ from single-pass retrieval strategies?
- Do expansion-reflection loops and chain-of-retrieval approaches solve the same problem?
- Why do deep research agents outperform retrieval augmented generation systems?
- How should retrieval and verification tasks be separated architecturally?
- Can adaptive per-step decisions outperform uniform retrieval policies across different reasoning tasks?
- What role does document reranking play alongside decisions about whether to retrieve?
- How do hierarchical research architectures improve multi-hop query accuracy?
- How should retrieval systems handle multi-hop reasoning and iterative information needs?
- What would instruction-following retrieval enable that query-only systems cannot?
- How does temporal grounding in retrieval compare to architectural approaches?
- How does query decomposition reduce retrieval costs at inference?
- How do hierarchical research architectures handle multi-hop queries better?
- Why does query routing matter for retrieval-augmented systems?
- How should retrieval and reasoning be integrated architecturally?
- Can relevance judgments based on query resemblance miss essential factual connections?
- Do different network layers specialize in retrieval versus reasoning tasks?
- Can retrieval improve multi-step reasoning by triggering at each uncertainty?
- Can task-aware ranking replace similarity scoring in other RAG systems?
- What makes reranking during retrieval better than catching failures at plan time?
- Does the parallel versus sequential trade-off appear in retrieval-augmented generation systems?
- Can generator feedback backpropagate through the entire retrieval pipeline?
- Does uncertainty trigger retrieval better than fixed-interval tool calls?
- Can stateless multi-step retrieval capture evidence integration as well as dynamic memory?
- Does retrieval iteration improve accuracy only after strong first-stage ranking?
- How should we allocate compute between reasoning and retrieval iterations?
- How does test-time search budget efficiency benefit from hierarchical architectures?
- What mathematical limits constrain embedding-based retrieval systems?
- Can embedding-based retrieval alone solve the causal relevance problem?
- What makes retrieval augmentation more effective than simply increasing embedding size?
- When should relational graph traversal replace vector embedding retrieval?
- Can explicit linkers replace vector similarity for multi-step question answering?
- How can knowledge graphs improve over pure embedding retrieval?
- Can re-ranking and advanced chunking fix embedding retrieval failures?
- Do graph databases outperform embeddings for relational retrieval tasks?
- How does description-based bridging compare to affordance-aware reranking for retrieval?
- What makes graph databases better than embeddings for relational queries?
- Can graph databases outperform embeddings when queries demand relational traversal?
- How does structure-aware retrieval routing differ from existing graph-versus-vector RAG tradeoffs?
- Why does community detection in knowledge graphs outperform pure retrieval or pure summarization?
- Can hierarchical entity extraction from books enable both textual and visual reasoning?
- How do community-based summaries differ from retrieval-based traversal in knowledge graph RAG?
- What makes hierarchical community summaries useful for exploration without a specific question?
- How does map-reduce over communities compare to flat multi-hop retrieval architectures?
- Can inference-time query decomposition replace pre-built knowledge graph structures?
- How does hypergraph accumulation differ from single-pass graph retrieval?
- Which knowledge structure types best fit different query types?
- Why does Personalized PageRank naturally discover concepts multiple hops from query seeds?
- Can query-time logic graphs match the efficiency of pre-built knowledge graph indexing?
- How do hierarchical knowledge graphs solve similar multimodal retrieval problems in books?
- Can graph-based retrieval with knowledge graphs scale to multi-hop reasoning?
- How do taxonomy-based retrieval scaffolds improve model performance at inference time?
- Can knowledge graph structure be exploited for efficient multi-hop retrieval?
- Can knowledge graphs built at inference time outperform pre-built retrieval augmented generation?
- Why do hierarchical architectures better implement the deep research definition?
- How do hierarchical architectures separate planning from retrieval differently than flat ones?
- How does retrieval-augmented generation extract structured properties from domain descriptions?
- What documents improve answers beyond surface query similarity?
- How does retrieval-augmented generation create topically redundant content patterns?
- Why does single-round retrieval fail on multi-step tasks across different domains?
- When do queries fail to capture relevance patterns effectively?
- Does retrieval quality depend more on access structure or write gating?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does search budget scale like reasoning tokens for answer quality?
Explores whether the test-time scaling law that applies to reasoning tokens also governs search-based retrieval in agentic systems. Understanding this relationship could reshape how we allocate inference compute between thinking and searching.
extends: hierarchical architecture makes the search budget more efficient by reducing interference loss
-
How do readers track segments, purposes, and salience together?
Can discourse processing actually happen in parallel rather than sequentially? This matters because understanding how readers coordinate multiple layers of meaning at once reveals where AI systems break down in comprehension.
connects: HierSearch solves at system architecture level the same parallel-tracking problem that discourse processing requires at the cognitive level
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Chain-of-Retrieval Augmented Generation
- RAG-R1 : Incentivize the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
- You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- Deep Research: A Systematic Survey
- A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Original note title
hierarchical research architectures that separate query planning from answer synthesis outperform flat architectures on multi-hop queries