A Computational Framework for Behavioral Assessment of LLM Therapists

we propose BOLT, a novel computational framework to study the conversational behavior of LLMs when employed as therapists. We develop an in-context learning method to quantitatively measure the behavior of LLMs based on 13 different psychotherapy techniques including reflections, questions, solutions, normalizing, and psycho education.

these LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions, which is against typical recommendations. At the same time, unlike low-quality therapy, LLMs reflect significantly more upon clients’ needs and strengths.

LLM therapists are currently not fully consistent with high-quality care,

First, we use client-human therapist conversations from two existing public datasets of high-quality therapy conversations (Pérez-Rosas et al., 2019; Malhotra et al., 2022) to simulate conversations between simulated clients and LLMs. Next, to assess the conversational behavior of therapists and clients in these conversations, we develop a prompting-based strategy that uses psychotherapy-based definitions and in-context examples to identify the psychotherapy techniques underlying all utterances. The techniques, which are operationalized based on those established by mental health experts (Lee et al., 2019; Cao et al., 2019), include REFLECTIONS, QUESTIONS, SOLUTIONS, NORMALIZING, and PSYCHOEDUCATION for therapists and BEHAVIOR CHANGE, SELF-DISCLOSURE OF AFFECT OR EXPERIENCE, and GAINING INSIGHTS for clients (Appendix Tables 6 and 7). We show that our proposed method achieves a macro-f1 of 57.7% on the 13- class multi-label therapist-behavior-identification problem, 13.6% greater than the next-best performing GPT- 4-based baseline.

We hypothesize that the popular Reinforcement Learning with Human Feedback (RLHF) alignment (Bai et al., 2022; Ouyang et al., 2022) may promote some of these behaviors. One of the key goals of RLHF is to help users solve their tasks and offer advice, which may lead LLM therapists to overly focus on problem-solving.

six different ways of reflecting differentiating between (1) NEEDS, (2) EMOTIONS, (3) VALUES, (4) CONSEQUENCES, (5) CONFLICTS, and (6) STRENGTHS (Appendix Table 6)