Chatbots in Knowledge-Intensive Contexts: Comparing Intent and LLM-Based Systems

we conducted a user study comparing an LLM-based CA to an intent-based system regarding interaction efficiency, user experience, workload, and usability. This revealed that LLM-based CAs exhibited better user experience, task completion rate, usability, and perceived performance than intent-based systems, suggesting that switching NLP techniques should be investigated further.

A knowledge management system (KMS) differs from an information system (IS) because users are expected to share and retrieve knowledge. One of these systems, cognitive assistants (CA), is a form of KMS that can capture and share knowledge among workers through conversational interactions [12]. The prevailing conversational technique for CAs is intent-based natural language processing (NLP) [26]. However, the rigidity of this technique can result in frequent conversation breakdowns [22]. Furthermore, intent-based systems are resource-intensive to create and maintain as the developer must define all possible user intents, how the assistant should respond, and what functions or data it might need [20]. LLM-based systems could help alleviate some of these constraints as they are quick to deploy, and their superior NLP capabilities can be used to develop more flexible, robust conversational interactions.