Domain-specific Question Answering with Hybrid Search

With the increasing adoption of Large Language Models (LLMs) in enterprise settings, ensuring accurate and reliable question-answering systems remains a critical challenge. Building upon our previous work on domain-specific question answering about Adobe products (Sharma et al. 2024), which established a retrieval-aware framework with self-supervised training, we now present a production-ready, generalizable architecture alongside a comprehensive evaluation methodology. Our core contribution is a flexible, scalable framework built on Elasticsearch that can be adapted for any LLM-based question-answering system. This framework seamlessly integrates hybrid retrieval mechanisms, combining dense and sparse search with boost matching, while maintaining production-grade performance requirements. While we demonstrate its effectiveness using our organization’s domain-specific data, the architecture is designed to be domain-agnostic and can be readily deployed for other enterprise applications.

Developing QA systems tailored to specialized domains necessitates techniques that effectively capture domain specific knowledge and terminology. Sharma et al. (2024) proposed a framework for compiling domain-specific QA datasets and introduced retrieval-aware fine-tuning of language models, which reduces hallucinations and enhances contextual grounding. Eppalapally et al. (2024) introduced a model for query rewriting to take the domain of interest into account. Additionally, hybrid retrieval methods that combine dense and sparse techniques have shown promise in domain-specific QA.

Context Awareness Limitations

The current system lacks crucial contextual awareness in several dimensions:

1. User Location Context: The system cannot differentiate between users on different pages or sections of the product (e.g., homepage versus editor), leading to potentially misaligned responses. This becomes particularly evident in answers that reference generic locations like “open homepage” without considering the user’s current position in the product journey.