Challenges of Large Language Models for Mental Health Counseling

large language models (LLMs) capable of understanding and generating human-like text may be used in supporting or providing psychological counseling. However, the application of LLMs in the mental health domain raises concerns regarding the accuracy, effectiveness, and reliability of the information provided. This paper investigates the major challenges associated with the development of LLMs for psychological counseling, including model hallucination, interpretability, bias, privacy, and clinical effectiveness. We explore potential solutions to these challenges that are practical and applicable to the current paradigm of AI. From our experience in developing and deploying LLMs for mental health, AI holds a great promise for improving mental health care, if we can carefully navigate and overcome pitfalls of LLMs.

These AI models have the potential to assist therapists in the daily provision of mental health services, through content suggestion and patient management [8, 9]. These efforts tend to focus on mental health issues that are not life-threatening and rather requires counseling. In this role, AI can help providers scale the delivery of mental health services and reduce patient costs, thus helping to address the global shortage of counselors and therapists. Additionally, several applications have been developed that place an LLM model in the role of digital counselor [10, 11, 12]. The primary challenge to all of these proposed uses is model accuracy and reliability,

Several mental health applications for use by individuals and institutions incorporate LLMs into their architecture. They can be divided into two broad categories: 1) user facing counseling and therapy; and 2) therapist assistants. Among user facing applications, we find some that provide an immersive conversation experience directly with the underlying model (e.g., [10, 11]), others that offer a combination of open-ended conversation with the model and rule-based elements (e.g. [23]), and finally, those that rely on the LLM primarily to understand and categorize the user’s message input, so as to better connect them with a “real” human therapist working for the service [24, 9]. This last category of user facing apps may overlap with therapist assistant apps, whose generated content never directly reaches the patient. Rather, the model outputs are sent to the mental health service providers as recommendations or suggested answers, sometimes acting as a “co-pilot.”

5 major challenges for building, training, and deploying LLM for mental health counseling: 1) Model hallucination, which impacts all LLMs regardless of application; 2) Model interpretability, which is crucial for human understanding, wider acceptance, and model improvement; 3) Privacy and regulatory concerns, notably arising from the use of patient electronic health records (EHR); 4) Clinical methodology and effectiveness; and 5) Bias arising from current LLM paradigms and limited data sources.

there are two primary forms of hallucination: 1) Intrinsic hallucination, where the model’s response is contradictory to the dialogue history or external sources of knowledge; and 2) Extrinsic hallucination, where the response cannot be readily verified against either the dialogue history or external knowledge sentences. In other words, extrinsic hallucinations are impossible to verify with the given inputs.

For example, if the model mis-references earlier parts of the conversation, or misstates a fact or even a belief of the user, this will negatively impact the user’s trust and immersion in the session. These factors as essential to a positive experience with the service, which is a necessary condition for its therapeutic effect. Therefore, ensuring that the model produces consistent responses is essential to user acceptance, and by extension, clinical efficacy.

the inability to maintain an internally consistent dialogue with the user.

Developers of language models and providers of counseling services must adopt transparent practices, openly discussing the limitations, biases, and potential risks associated with these models in therapy.

Human-in-the-loop deployment: Incorporating human reviewers or therapists in real time or post-hoc validation processes can help identify and rectify model hallucination instances, ensuring the provision of accurate and reliable responses. This may in some cases by a regulatory requirement, as the GDPR requires that any AI system that affects any natural person’s legal rights must involve human supervision [35].

However, language models lack the capacity for genuine empathy, often resulting in responses devoid of emotional understanding and tailored guidance. Limitation of LLM’s understanding has been studied directly on users for eating disorders [50] and depression and anxiety [51]. We may consider limited scopes (e.g., arising from data, architecture, training schemes, and others) inherent in the current generation of LLMs, as a key way to improve their capabilities.

LLMs such as ChatGPT may struggle to demonstrate emotional intelligence, resulting in inappropriate responses and an inability to understand nuanced emotional expressions [52]. This limitation hampers their ability to provide sensitive and empathetic counseling. The empathic capabilities of language models are limited, and they may fail to understand the nuanced emotions and experiences shared by individuals seeking counseling. This can undermine the therapeutic alliance and hinder the provision of appropriate emotional support,

Unlike human counselors, LLMs lack the ability to process nonverbal cues and body language, which are essential for effective counseling. This deficiency limits their ability to provide appropriate nonverbal support to patients, potentially hindering the counseling process. Similarly, AI is mostly focused on texts and audio responses. And human clients may seek nonverbal cues to form authentic connections. A lack of multimodal human-AI interactions may reduce the effectiveness of the client-counselor relationship [52].

LLMs may provide misguided or inappropriate advice due to limited understanding of unique patient circumstances, leading to potential harm for vulnerable individuals. Especially, the best practices of mental health counseling are evolving with evidence-based research. For example, the current prevailing approach in mental health counseling is to be less prescriptive and advisory. But using a foundation models like GPT-4 [6] would often result in highly specific advice. The advice could be unwarranted and unwanted, especially since LLMs may struggle to obtain, much less understand the concept of, informed consent. As a result,