Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey

To address this, researchers have explored diverse methods to enhance LLMs by integrating domain-specific knowledge. In this survey, we provide a comprehensive overview of these methods, which we categorize into four key approaches: dynamic knowledge injection, static knowledge embedding, modular adapters, and prompt optimization. Each approach offers unique mechanisms to equip LLMs with domain expertise, balancing trade-offs between flexibility, scalability, and efficiency. We discuss how these methods enable LLMs to tackle specialized tasks, compare their advantages and disadvantages, evaluate domain-specific LLMs against general LLMs, and highlight the challenges and opportunities in this emerging field.

Table 2: Guidance on selecting knowledge injection paradigms based on training cost, inference speed, and limitations. Dynamic knowledge injection integrates external knowledge at runtime, offering flexibility and adaptability to new information without increasing training cost. However, it requires an effective retrieval module, and the inference speed depends heavily on retrieval performance, which can slow down the overall process. Static knowledge embedding embeds domain expertise during pretraining or fine-tuning, requiring large-scale domain-specific data and significant training resources, including GPUs and time. While it incurs no additional inference cost, its limitations lie in the potential risks of catastrophic forgetting and its inability to adapt to evolving information. Modular adapters serve as a middle ground, allowing plug-and-play components to enhance domain-specific capabilities with minimal training data. Only a small subset of parameters needs to be trained, which reduces training costs, and inference speed is largely unaffected. However, the quality of training data significantly impacts the performance of this method. Prompt Optimization, on the other hand, avoids retraining entirely by leveraging carefully crafted inputs. It has no impact on inference speed but relies on extensive human effort to find optimal prompts. This method is limited in its ability to utilize new knowledge and primarily activates pre-existing knowledge.

3.4 Prompt Optimization Unlike previous approaches, prompt optimization does not retrieve knowledge from external sources. Instead, it focuses on fully leveraging or guiding the LLM to utilize its internal, pre-existing knowledge. The process can be formalized as: y = M 􀀀 [p, x]; θ  , where p represents a textual prompt containing implicit domain knowledge or specific instructions. Prompt optimization offers significant advantages, including eliminating dependency on external domain knowledge bases and avoiding training. However, it also presents challenges, as designing effective prompts can be both complex and time-consuming. Additionally, long prompts may reduce the available context window, potentially affecting model efficiency and performance.