Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting

In the era of Large Language Models, we believe it is the right time to develop AI assistance for computational psychotherapy. We study the task of cognitive distortion detection and propose the Diagnosis of Thought (DoT) prompting. DoT performs diagnosis on the patient’s speech via three stages: subjectivity assessment to separate the facts and the thoughts; contrastive reasoning to elicit the reasoning processes supporting and contradicting the thoughts; and schema analysis to summarize the cognition schemas. The generated diagnosis rationales through the three stages are essential for assisting the professionals. Experiments demonstrate that DoT obtains significant improvements over ChatGPT for cognitive distortion detection, while generating high-quality rationales approved by human experts.

However, existing works mostly take shallow attempts in a heuristic manner, e.g., analyzing emotions and generating comforting responses.

There is still a significant gap for such systems to contribute to real professional psychotherapy, which requires deep studies of the patient’s thinking patterns, the establishment of cognition models, and the methods to reconstruct the cognition models.

professionals perform nuanced diagnosis over the patient’s speech, we propose the Diagnosis of Thought (DoT) prompting. In DoT, we diagnose the patient’s speech through three stages: (1) subjectivity assessment, (2) contrastive reasoning, and (3) schema analysis. In subjectivity assessment, we distinguish the patient’s subjective thoughts from the objective facts; In contrastive reasoning, we elicit the reasoning processes supporting and opposing the patient’s thoughts; Finally, in schema analysis, we summarize the underlying thought schema and map it to the cognitive distortion types. We conduct comprehensive experiments using the recent top-performing LLMs. In zero-shot setting, DoT obtains over 10% and 15% relative improvements for distortion assessment and classification, respectively, on ChatGPT.

Patients with mental disorders, such as depression or anxiety, tend to form negative thoughts very rapidly and unconsciously, leading to negative emotions which further strengthen their overall negative views and beliefs about the world.

in a typical CBT process, the first core step is to identify those maladaptive negative thoughts and summarize their underlying schemas, formally known as cognitive distortions. There are generally 10-20 common, well studied types of cognitive distortions.

In the real psychotherapy process, there’s a significant amount of textual information including therapeutic conversations and diaries, etc. Such information is often long, highly fragmented, and disorganized, containing multiple types of distortions beyond the toy examples in Table 1. The task of cognitive distortion detection aims to automatically detect the distortion types given such textual information from the patients, in order to assist therapists to enhance their efficiency and productivity. Meanwhile, such detectors can also potentially serve as self-assisting tools for the patients to diagnose their thoughts and conduct CBT practice, upon meeting the robustness and safety requirements.

Formally, cognitive distortion detection consists of two steps: 1) Distortion assessment to predict whether the given speech contains cognitive distortions, as a binary classification problem; and 2) Distortion classification to predict the specific distortion types, as a multiclass classification problem.

Subjectivity Assessment. The patient’s speech consists of a mixture of reality (objective facts) and interpretations/opinions (subjective thoughts). In order to perform deep analysis of distorted thinking, we first need to find out which parts of the speech are objective facts and which parts are subjective thoughts. After such an assessment, we summarize the objective facts into the situations as the evidence base to diagnose the subjective thoughts.

Contrastive Reasoning. This stage aims to discover how the patient ascertains the veracity of their subjective thoughts. Based on the situation, we deduct the reasoning processes that supports and contradicts the patient’s thoughts respectively. By contrasting two different interpretations based on the same situation, we can identify the thought schemas more clearly.

Schema Analysis. This stage aims to study why the patient forms the specific reasoning process. The term "schema" refers to the cognitive structures that organize our knowledge, beliefs, and expectations. Understanding what schemas a patient is relying on can reveal much about their cognitive mode and distortions.

Dataset and experimental settings

We experiment on the cognitive distortion detection dataset proposed by Shreevastava and Foltz (2021), which is annotated by experts based on the Therapist QA dataset2. The dataset consists of 2,531 examples of patient speech annotated with ten common types of cognitive distortions, as specified in Appendix A. 63.1% of the examples have cognitive distortions, which are annotated with the two dominant ones.

we instruct the experts to choose between: 1) Comprehensive. (Correct and comprehensive.); 2) Partially good. (Reasonable but not comprehensive) 3) Invalid. (Not reasonable.) Table 4 shows the evaluation results on 100 examples for DoT over ChatGPT and GPT-4.

Theory-wise, we are eager to explore to what extent the LLM can simulate the human cognitive functions, so as to determine the role that language plays in the overall human cognition.

We believe that there exists substantial potential for leveraging LLMs within numerous facets of mental health support that are currently under-explored. Our findings, thus, not only illuminate a path towards more efficient therapy methods but also open doors for future investigations to push the boundaries of AI’s role in mental health treatment.