Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLMs

Robust therapeutic relationships between counselors and clients are fundamental to counseling effectiveness. The assessment of therapeutic alliance is well-established in traditional face-to-face therapy but may not directly translate to text-based settings. With millions of individuals seeking support through online text based counseling, understanding the relationship in such contexts is crucial.

In this paper, we present an automatic approach using large language models (LLMs) to understand the development of therapeutic alliance in text-based counseling. We adapt a theoretically grounded framework specifically to the context of online text-based counseling and develop comprehensive guidelines for characterizing the alliance. We collect a comprehensive counseling dataset and conduct multiple expert evaluations on a subset based on this framework. Our LLM-based approach, combined with guidelines and simultaneous extraction of supportive evidence underlying its predictions, demonstrates effectiveness in identifying the therapeutic alliance. Through further LLM-based evaluations on additional conversations, our findings underscore the challenges counselors face in cultivating strong online relationships with clients. Furthermore, we demonstrate the potential of LLM-based feedback mechanisms to enhance counselors’ ability to build relationships, supported by a small-scale proof-of-concept.

In psychological counseling, a positive relationship between counselors and clients is fundamental for achieving effective therapeutic outcomes (Tichenor and Hill, 1989; Horvath and Symonds, 1991; Knaevelsrud and Maercker, 2006). The robust therapeutic alliance signifies the cooperative relationship between counselors and clients, characterized by their shared therapeutic goals and their ability to engage together, within the context of an affective bond or positive attachment (Constantino et al., 2002). The absence of regular and timely assessment and feedback may impede counselors in gradually nurturing of a solid rapport with clients, potentially leading to client dropout. Traditionally, counselors rely on professional supervisors for feedback, a process that is often labor-intensive and time-consuming, resulting in delayed responses. Natural Language Processing (NLP) systems capable of understanding the alliance could empower counselors with prompt and insightful feedback to enhance their practice. However, the current understanding of the alliance is primarily limited to traditional face-to-face, speechbased counseling (Martinez et al., 2019; Goldberg et al., 2020; Lin et al., 2022), due to the scarcity of resources and methods tailored to text-based interactions.

Also, while previous NLP research has focused on the behaviors of individual participants, either counselors’ strategies (Cao et al., 2019; Gibson et al., 2016; Chiu et al., 2024) or clients’ reactions (Tanana et al., 2015; Li et al., 2022, 2023), the mutual cognitive and affective agreement they reach through multi-turn interactions is the key factor influencing conversation outcomes (Rector et al., 1999; Watson and Geller, 2005).

In this paper, we present an effective automatic approach using Large Language Models (LLMs) to understand the establishment of therapeutic alliances in online text-based counseling (Wei et al., 2022a). We propose a conceptual framework with detailed guidelines to characterize the alliance in conversations, adapting theories and scales of therapeutic relationships from face-to-face therapy to text-only interactions. The framework includes three dimensions: goal-setting consensus, approaches to goals, and cultivation of affective bonds (Figure 1). The guidelines facilitate identifying observed elements in textual conversations corresponding to each framework component.

We then collect a large-scale text-based counseling dataset from an online platform. Using our proposed framework and guidelines, trained experts annotate a subset of sessions with high inter-rater reliability. We employ prompt tuning to enable LLMs to apply these guidelines in understanding the alliance within texts. Additionally, we use the Chain-of-Thought (CoT) process (Wei et al., 2022b) to help models identify supportive evidence for their evaluations (as shown in the reason part of Figure 1). Experimental findings show that integrating precise guidelines and CoT significantly enhances LLMs’ ability to understand the alliance, ensuring consistency and alignment with experts.

We use the best-performing model on the remaining unannotated sessions to show a positive correlation between the alliance and favorable counseling outcomes. Our findings highlight that counselors, including experienced ones, may struggle to build deeper connections as counseling progresses. This underscores the need for evaluation and feedback mechanisms to enhance counseling effectiveness. Our small-scale proof-of-concept demonstrates that LLM-based feedback can offer counselors insights to better understand their alliances with clients and improve their relationship-building skills.

There is mutual understanding about what participants are trying to accomplish in

therapy.

In psychology research, the preeminent definition of therapeutic alliance, as introduced by Bordin (1979), emphasizes interactive and collaborative elements in counselor-client relationship in the context of a positive affective attachment (Constantino et al., 2002). This concept consists of three core components – counselors and clients’ mutual agreement on the targets of counseling (Goal), abilities to engage in the tasks of counseling (Approach), as well as the cultivation of emotional connections (Affective Bond) (Bordin, 1979).

We adopt the Observer-rated Short version of Working Alliance Inventory (WAI-O-S) (Tichenor and Hill, 1989) to measure the alliance. This inventory comprises 12 designed questions, with each alliance dimension measured by four questions. Each question is rated ranging from 1 to 5 points. Its reliability and validity has undergone thorough and comprehensive verification in various psychotherapy types (Santirso et al., 2018; Ribeiro et al., 2021). Table 1 presents the dimensions along with questions that shape the alliance.

Goal. In counseling, goals are important for facilitating changes in clients’ thoughts, feelings, and actions. They provide direction for both counselors and clients during their sessions. Clear agreement on goals increases adherence and leads to better outcomes. However, at the beginning of counseling, there can be a lack of clarity about clients’ issues and differences in goals between clients and counselors. To address this, counselors should engage in deeper discussions with clients to establish mutually endorsed and valued objectives.

Approach. In addition to the agreement on goals, the strength of the working alliance also depends on the participants’ clear and mutual understanding as well as acceptance on the tasks that their shared goals impose upon them (Bordin, 1983).

Tasks are usually assigned by counselors based on their counseling styles, personal experiences and predispositions. However, clients may not fully understand the interconnections between the assigned tasks and the overarching goals. Moreover, clients may perceive that the demands of tasks exceed their abilities. In such cases, counselors need to skillfully adapt to their clients by offering alternative or modified tasks, thereby empowering clients to actively and effectively engage.

Affective Bond. Apart from cognitive collaboration, emotional connections play a crucial role in shaping the therapeutic alliance. The concept of affective bonds embraces the complex network of positive personal attachments between counselors and clients, including issues such as mutual trust, liking, acceptance, and confidence (Horvath and Marx, 1990). As clients perceive that counselors genuinely care about and appreciate them, a sense of security is established, fostering a greater willingness to delve into deeper self-disclosure during counseling, particularly in discussing their negative behaviors and thoughts. Moreover, clients’ confidence in counselors’ capabilities to facilitate positive changes make them more inclined to accept counselors’ guidance and actively participate in the tasks assigned by the counselors.

The client and counselor are working on mutually agreed upon goals. Q2

The client and counselor have same ideas about what the client’s real problems are. Q3

The client and counselor have established a good understanding of the changes that

would be good for the client.

There is agreement about the steps taken to help improve the client’s situation. Q5

There is agreement about the usefulness of the current activity in therapy (i.e., the

client is seeing new ways to look at his/her problem).

There is agreement on what is important for the client to work on. Q7

The client believes that the way they are working with his/her problem is correct. Q8

There is a mutual liking between the client and counselor. Q9

The client feels confident in the counselor’s ability to help the client. Q10

The client feels that the counselor appreciates him/her as a person. Q11

There is mutual trust between the client and counselor.

Long-Term Communications ̸= Stronger Alliance.

We divide clients’ counseling sessions into three phases—early, middle, and late—and compare the counselor-client relationship across these stages. Our findings indicate that the relationship does not significantly deepen over time. Specifically, there is only a marginal increase in affective connections, while agreement on counseling goals and approaches remains constant. Further analysis shows that nearly 50% of client-counselor pairs experience either a decline or no change in the strength of the therapeutic alliance, with less than 3% improving by at least one level within our framework. This emphasize the challenges counselors face in enhancing relationship-building skills without adequate feedback.

Better Counseling Outcomes are More Likely Based on Robust Alliance. Psychology research underscores the pivotal role of a robust alliance in counseling outcomes (Horvath and Greenberg, 1994; Falkenström et al., 2014). Here, we utilize clients’ self-reported ratings on the Outcome Rating Scale (ORS) (Miller et al., 2003; Bringhurst et al., 2006) to gauge the effectiveness of each counseling session. The ORS evaluates various aspects including clients’ individual physical and mental well-being, interpersonal relationships, social role functioning and overall well-being, with scores ranging from 0 to 100 for each aspect. Pearson correlation analyses between total working alliance scores and ORS dimensions show significant correlations (r ≈ 0.30, p < 0.001). This indicates that a stronger working alliance may be associated with more favorable conversation outcomes. Additional details are provided in Appendix D.2.

Counselors’ Common Behaviors Shape the Therapeutic Alliance. We conduct further natural language analysis to investigate common counselor behaviors that influence the establishment of therapeutic relationships, based on GPT-4- generated explanations.

By analyzing and comparing counselors’ performance in sessions with both poor and strong alliances, we have gained valuable insights. Counselors who struggle to establish a robust alliance often overlook their clients’ negative emotions and resistance during counseling. Their guiding approach typically manifests in two problematic ways: either they respond passively to clients’ statements without exploring core issues, or they overstep boundaries by excessively directing clients, thus compromising their autonomy. Additionally, the feedback provided tends to be vague and generalized, lacking the personalized and specific solutions that address each client’s unique needs.

These findings highlight the necessity for counselors to cultivate deeper empathy and consistently monitor clients’ responses and behaviors throughout the counseling process. Counselors must find a balance between offering guidance and respecting clients’ autonomy. It is crucial for them to provide tailored advice that deeply considers each client’s circumstances to foster a positive therapeutic alliance. Implications for Feedback using LLMs’ Explanations. These results highlight the significance of the working alliance in online text-based counseling, supported by clients’ positive outcomes and psychological theories. However, even experienced counselors may face challenges in fostering deep connections in long-term sessions. To address this, we use GPT-4’ explanations generated via CoT to provide constructive feedback to counselors.

Counselors E and H, who struggled with relationship-building, were given 10 sessions evaluated by LLM with explanations as feedback. They assessed the feedback on: (1) enhancing their understanding of the client alliance, (2) identifying improvement directions, and (3) willingness to adjust strategies based on the feedback. Each aspect was rated from 1 to 5, with 3 as neutral. The average scores were 3.43, 3.49, and 3.74, respectively, indicating the efficacy of LLM-based feedback in helping counselors deepen their client alliances and refine their connection-building approach. Further details are in Appendix D.3.

In future work, we will integrate LLM-based real-time evaluation and feedback on the working alliance into actual counseling sessions to facilitate counselors cultivate deeper therapeutic connections with their clients.