Measuring Alliance and Symptom Severity in Psychotherapy Transcripts Using Bert Topic Modeling

After decades of psychotherapy outcome research, the effectiveness of psychotherapy has been convincingly shown in numerous clinical and naturalistic studies and meta-analyses across a broad range of different conditions (for an overview, see Lambert, 2021). More recently, the focus has shifted to the processes and mechanisms of change that drive the effectiveness of psychotherapy (Lutz et al., 2021). At the core, process research tries to identify ingredients and mechanisms that either allow psychological interventions to work or increase their desired effects. Several processes have been identified, most prominently the alliance, but also many others (see Crits-Christoph et al., 2021). The effects and clinical utility of these processes have been shown in a landmark meta-analysis by Norcross and Lambert (2019). However, limitations prevail, such as unknown directions of the process–outcome relationship (e.g., is the alliance driving improvements in outcome or the other way around?) and difficulties to distinguish patient and therapist effects. Another limitation is a lack of research on what could be called second-order processes. Second-order processes relate to processes as processes relate to treatment outcome: They are the ingredients and mechanisms that allow processes to be effective. As such, for example, they aim to answer the important question ‘What ingredients and mechanisms does it take for the intervention to improve the alliance?’ just as processes aim to answer the question ‘What ingredients and mechanisms does it take for the intervention to improve outcome?’. Second-order processes are important, because often, it is far from self-evident how to foster therapeutic processes (Norcross & Lambert, 2019). A final limitation of process research lies in the fact that it is often focused on questionnaire data for large analyses as qualitative assessments can be very time-consuming. This leads to several problems: First, questionnaire data must be filled by patients, which can be both time-consuming and burdensome. Second, questionnaires are better suited for a top-down approach, since they must be relatively short so that a specific construct needs to be pre-selected. If we want to conduct a more bottom-up (i.e., data-driven) analysis, they fall short due to a lack of items. Third, patients may not quite understand the items the way they were intended (e.g., Hawkins et al., 2017). Instead, using Natural Language Processing (NLP), we can use therapy transcripts to create features that predict relevant constructs. This allows for a more data-driven analysis, employing the patient’s own words across the whole session without resorting to session-wise questionnaires with few items that may become burdensome for the patient. We aim to exemplify this approach by using the session content to predict treatment progress (i.e., symptom severity) and a fundamental mechanism (i.e., alliance) building on prior work (e.g., Aafjes-van Doorn et al., 2020; Burkhardt et al., 2022; Goldberg et al., 2020; Ryu et al., 2023). By analyzing transcript features we aim to identify first-order processes that correlate with outcome and second-order processes that correlate with the alliance. We hope that this approach may enrich process research by providing new opportunities to analyze different processes and their respective interactions.

The value of session-wise assessments of symptom severity and alliance can also be viewed through the lens of routine outcome monitoring (ROM) and patient-focused research (Castonguay et al., 2013). Based on research that has proven clinical intuition to be inaccurate (Ægisdóttir et al., 2006), data-driven prognostic models have been developed to predict treatment outcome for specific patients. These models rely on ROM by measuring symptom severity on a session-to-session level since symptom severity is a robust predictor of therapy outcome (e.g., Lorenzo-Luaces et al., 2020). By employing ROM and giving psychometric feedback to the therapist, patient-focused research seeks to increase therapists’ awareness of their patient’s progress or lack thereof. Feedback can be more or less statistically and methodologically sophisticated, from raw outcome scores to expected treatment response (ETR) curves with additional clinical support tools (e.g., Lutz et al., 2022; Whipple et al., 2003). In several meta-analyses feedback has shown additional treatment effects beyond the mere intervention, with further benefits regarding treatment duration and dropout rates (for an overview, see Lutz et al., 2021). Even though the exact mechanisms of feedback remain unknown, it is hypothesized that it may work by increasing therapists’ awareness of clients in risk of deterioration and supporting the therapists with tools to counter these developments early on (e.g., Delgadillo et al., 2018). Also, it remains unknown whether the effects of ROM feedback could possibly also apply to some degree to alliance feedback as the alliance is one of the most investigated psychotherapy processes and a strong indicator of good outcome (r = .28; Flückiger et al., 2018). According to Bordin’s (1979) pantheoretical definition, the working alliance entails three components: Consensus on therapeutic goals, agreement on the tasks that lead to these goals, and the affective bond between therapist and patient. Similar to symptom severity, a within-person and between-person distinction can be made. Accordingly, the alliance can be conceptualized both as a general factor that mediates the effects of baseline characteristics on outcome and an idiosyncratic process that evolves over the course of therapy (Zilcha-Mano & Fisher, 2022). This distinction has proven fruitful as the alliance has been measured on a session-to-session level to predict symptom change over the course of therapy (e.g., Rubel et al., 2017).

With advancements in artificial intelligence and machine learning, researchers now wield powerful tools for analyzing big data (O’Leary, 2013), leading to new approaches in psychotherapy research (Aafjes-van Doorn et al., 2021). However, methodological challenges remain, such as heterogeneity of cross-validation strategies, a broad range of different algorithms without any consensus which one might perform best for a given problem, and limited model interpretability, especially with more complex algorithms. Addressing these challenges may involve employing more robust nested k-fold cross-validation procedures (Tibshirani et al., 2021) and an eXplainable Artificial Intelligence (XAI) framework (Holzinger et al., 2022).

Machine learning approaches from the area of NLP allow for a quantification and a subsequent statistical analysis of language. This applies for unsupervised algorithms, which can be used for clustering (e.g., PCA, DBSCAN), allowing for complexity reduction in large data sets. One promising approach is topic modeling (Kherwa & Bansal, 2018). Topic modeling is a set of statistical and probabilistic techniques to detect clusters of related words (“topics”) in a collection of documents. Popular techniques entail Latent Semantic Analysis (LSA; Deerwester et al., 1990), Non-negative Matrix Factorization (NMF; Paatero & Tapper, 1994) and the Latent Dirichlet Allocation (LDA; Blei et al., 2003). These techniques are based on bag-of-words-embeddings that measure the frequency of words in the respective document but ignore relevant information such as word order, textual context, or semantic similarities. Yet, they have been successfully implemented in many areas of research (Kherwa & Bansal, 2018) including psychotherapy research. Atkins et al. (2012) conducted topic modeling on a corpus from a couples therapy trial showing two important benefits: Topic modeling can be used to predict relevant target variables, such as symptom load or alliance. Also, it can be used to analyze therapeutic processes by looking at the upcoming topics. Further, Atkins et al. (2014) successfully predicted the treatment fidelity in sessions of motivational interviewing with labeled topic modeling, a topic modeling approach to classify documents according to their respective label. Imel et al. (2015) showed that topic modeling creates highly relevant topics in individual therapies. Using labeled topic modeling, they were able to discriminate different therapy approaches. Atzil-Slonim et al. (2021) found that topics accurately reflected the psychotherapeutic process. They were able to cluster topics in superordinate themes that comprised a broad perspective of therapy-related topics. Further, they could predict good outcomes and the occurrence of alliance ruptures in a logistic regression while simultaneously identifying associated topics.

Recently, the rise of deep learning allowed for the development of neural topic modeling, which shows superior performance to the “classic” approaches mentioned above (Miao et al., 2016). This is due to their more complex and accurate embeddings taking into account word similarities, word order, and textual context. Lin et al. (2022) compared several neural topic modeling approaches showing their potential to monitor the therapeutic process. More recently, deep learning transformer embeddings have shown even better performance (Vaswani et al., 2017), leading to the development of a topic modeling approach based on transformer embeddings (BERTopic; Grootendorst, 2022).

This study aims to overcome some of the aforementioned limitations by using BERTopic topic modeling in a nested cross-validation procedure to compare several promising machine learning algorithms to predict symptom severity and alliance. Moreover, we want to employ XAI to identify both important topics as well as important superordinate topic themes for each of the predictions. This will allow us to gain insight into session processes that drive change in symptom severity and alliance. Further, we would like to provide a proof of concept for future patient-focused and routine outcome monitoring research: By using NLP and XAI, feedback may be provided solely based on the session transcript, highlighting the driving processes for the prediction.

Several limitations of this analysis come to mind. First, we conducted very little hyperparameter tuning for BERTopic (varying document size for five and ten words and varying the number of topics between 150, 200, and 250 words). More tuning on other parameters could have been conducted (e.g., n-gram range, language model). Also, some topics were very homogeneous, because they consisted of different declinations of the same word (word flexion is common in German). This led to similar topics that were very homogeneous at the same time (e.g., topic write1: written, write, wrote, writing and topic write2: write, write, write [different German declinations]). Since BERTopic does not employ lemmatization like LDA, this has led to these homogeneous topics. However, BERTopic modeling still showed better performance than LDA, likely due to the superior embeddings. Second, even more specific themes could have been created which would have allowed for a more fine-grained analysis with better performance. Maybe that would also be a reason for the superior performance of the uni- and bigram model, which originally contained over 400,000 features. Third, the data set is still rather small, and it included on average only 4.5 transcripts per patient. Larger data sets would have allowed more stable results across different test sets and could possibly have led to better performance due to more training data. Fourth, model performance was not good enough to replace questionnaire data, and remained especially unsatisfactory for the alliance (r = .20). Since the uni- and bigram model showed better performance both for symptom severity and alliance, future research may be directed towards these features. Yet, the gains come with the cost of reduced interpretability, as uni- and bigrams are more difficult to comprehend than topics. However, considering vastly better performance, this could be addressed by clustering the features into more accessible superordinate categories.

Despite these limitations, we were able to use BERTopic, a state-of-the-art topic modeling approach to predict symptom severity and alliance. Topics were created separately for patient and therapist speech. We employed several machine-learning algorithms (Elastic Net, XGBoost, SVR/SVC, RF, MERF, GPBoost, and SuperLearner that competed in a nested cross-validation procedure for the best performance. This allowed us to use all available data as training data and to assess the performance across ten test sets. As a sensitivity analysis, we conducted the cross-validation with patient-level splits to evaluate model accuracy for new patients. Further, we compared the best results to a smaller topic model of only 50 topics¹, an LDA-based topic model and a uni- and bigram model. Finally, even though we employed complex machine-learning algorithms, using an XAI framework allowed us to assess feature importance across all topic and themes.

Future directions may entail the use of topic synchrony features (e.g., Aafjes-van Doorn et al., 2020) for the prediction of the alliance. For example, differences in frequency of patient and therapist topics may be calculated as new features for better accuracy. Also, the approach could be used to provide feedback to therapists after each session by giving predictions of symptom severity and alliance, as well features that drive these predictions. For example, the therapist may receive a prediction of lower alliance scores for the last session because of a strong focus on income- and health-related themes. This could prompt the therapist to check in with the patient about their experience of the session. In general, topic modeling and similar NLP approaches could in the future be implemented as a feedback tool for a broad set of treatment-relevant variables (i.e., outcome, processes, and higher-order processes), supporting therapists to spot subtle negative trends and counter them early on (Lutz et al., 2022).