Using video-annotation software to identify interactions in group therapies for schizophrenia: assessing reliability and associations with outcomes

Participants were recruited from out-patient mental health services between 2012 and 2014. Participants were aged between 18 and 65 years, had a diagnosis of schizophrenia, scored 18 or above on the negative subscale of the Positive and Negative Syndrome Scale (PANSS) [21], were willing to participate and were able to provide written informed consent.

Interactions were assessed from participants with clinically relevant changes in negative symptoms, i.e. those who met the criteria of being an ‘improver’, and participants who met the criteria for being a ‘no-changer’. Improvers were participants who had a clinically significant reduction on the PANSS negative symptoms subscale from the baseline to end of treatment assessment phase, defined as a clinically relevant reduction of at least 20% [22]. No-changers were participants who had either no change in the PANSS negative symptoms subscale, a reduction of just one point from baseline to end of treatment assessment phase, or an improvement of just one point from baseline to end of treatment assessment phase.

Participants were excluded if they had an insufficient command of English, a physical disability that impaired participation in groups, did not attend at least one session within the first and last five sessions of the group, and did not meet the criteria of an ‘improver’ or ‘no-changer’. Participants who did not have a 20% negative symptom reduction in the Clinical Assessment Interview for Negative Symptoms (CAINS) [2], a more specific measure of negative symptoms, were also excluded from the ‘improver’ category. Participants who demonstrated a reduction of more than 10% in negative symptoms measured by the CAINS were excluded from the ‘no changer’ category.

All analyses were conducted in STATA/SE version 12.0 (StatCorp. 2011), where p < 0.05 was taken to indicate statistical significance and p < 0.10 was taken to indicate a statistical trend. Frequency ratings, indicating the presence of interactive behaviors, were calculated as sample population percentage scores for each improver and no-changer. The sample population percentage was calculated by dividing the total number of statements rated ‘yes’ (i.e. present) for each IGMIPS category, by the total number of statements made by each participant. Scores were calculated for statements made in the first and last sessions of attendance, each of which were 90 min long.

Rater agreement between SO and SA was calculated to ensure sufficient coding reliability: this was calculated as the percentage of categories coded by both SO and SA as ‘present’. An alternative statistical test of inter-rater reliability that can account for any rater agreement that occurs due to chance, such as Cohen’s Kappa, was considered. However, the type of data measured within the IGMIPS categories varied across the different IGMIPS categories; including binary outcome data (yes/no for the ‘frequency of interactive behavior’ ratings), continuous outcome data (for the ‘significance’ ratings) and nominal outcome data (for the ‘who’ ratings). Therefore no statistical test of inter-rater reliability that could account for this variation was deemed suitable. Furthermore, the actual percentage agreement was deemed as more informative than a statistical test of inter-rater reliability, which relies on the prevalence of the measure (i.e. the number of statements rated per person).

The relationship between the frequency of interactive behavior and improver-status was explored through a series of binary logistic regression analyses. Separate analyses were conducted for each category outlined in the IGMIPS. Furthermore, separate analyses were conducted for statements coded from the first and last sessions. Significance ratings were adjusted for.

Of the 81 participants recruited to the nine therapy groups included in this study, 17 participants met the inclusion criteria. Nine participants met the criteria for ‘improvers’ and eight participants met the criteria for ‘no-changers’. The characteristics of these participants are outlined in Table 2.

In total, 1275 individual statements were coded in accordance with the categories outlined in the IGMIPS. Eighty-four of these statements (7%) were excluded from the analysis as they were rated as ‘less than 50% audible’ - i.e. less than 50% of the statement could be annotated due to inaudibility of the statement. The frequency of interactive behaviors are summarized in Table 3.

Across all IGMIPS categories, there was 96% agreement between the two independent ratings made by SO and SA. There was 92% agreement for all frequency categories rated either ‘yes’ or ‘no’. In addition, there was 98, 99 and 95% agreement for significance, locational and who items respectively. See Table 4 for a summary of the percentage agreement for each of the categories.

Findings from the binary logistic regression analyses, which explored the relationship between the frequency of interactive behaviors and improver-status, are summarized in Table 5.

There was a significant association between self-initiated statements and improver-status in the first session of attendance (95% CI = 1.00 to 1.13, p < 0.05); where a higher frequency of self-initiated statements in the first session of attendance was associated with being an ‘improver’ at the end of treatment. For statements occurring in the last session of attendance, there was a statistical trend between the frequency of self-initiated statements and improver-status (OR = 1.08, 95% CI = 0.99 to 1.18, p = 0.083). There was also a trend for an association between the frequency of questions (95% CI = 1.00 to 1.14, p = 0.067) and improver-status, and giving advice (95% CI = 1.00 to 2.07, p = 0.067) and improver-status; where a higher frequency of statements involving questions or giving advice in the first session of attendance was associated with being an ‘improver’ at the end of treatment.

There was no statistically significant evidence for a relationship between these variables in the last session of attendance. There was no statistically significant evidence for any of the other IGMIPS categories and improver status.

In line with our first hypothesis, the overall agreement between the two researchers was high. This supports the feasibility of combining a group behavioral coding scale, such as the IGMIPS, with ELAN video-annotation software. There was minimal disagreement between raters on coding the IGMIPS categories, further supporting its feasibility in measuring a range of individual group member interactions.

There was also evidence that group interactions, measured on a moment-to-moment basis using video-annotation software, were predictive of negative symptom outcomes. A higher frequency of self-initiated statements in the first session of attendance was associated with a clinically significant improvement in negative symptoms following participation in group therapy. There was also evidence for a statistical trend between more questions and advice giving in the first session of attendance, and improved odds of clinical improvements in negative symptoms.

This study had a number of strengths. First, stringent measures were taken to ensure that observer-rated group interactions were measured reliably, with minimal rater-bias. Participant statements were transcribed by two trained researchers to ensure minimal errors in annotating individual statements. Interactions were measured from group sessions that were recorded in high-definition with two microphone sources, for optimal visual and audio quality. Furthermore, coding reliability was assessed from a randomly selected proportion of statements coded by an independent researcher. Second, assessments of negative symptoms on 15 of the 17 participants were conducted by blinded researchers as part of a randomized controlled trial. Third, interactions were measured across multiple group psychological therapies, which have varying therapeutic orientations. Fourth, the impact of group interactions was explored in participants with clinically relevant outcomes, selected from a large pool of participants who attended multiple groups for schizophrenia.

One limitation is multiple testing. Given that separate logistic regression analyses were conducted for each of the IGMIPS categories, the chances of finding a false positive were high. A further limitation was the small sample included in the logistic regression analyses. This meant that these analyses lacked the power to detect a significant change in the odds of being an improver or no-changer following group attendance. Hence it is not possible to conclude with confidence that an effect of the frequency of interactions on improver status did not exist where there was no statistical evidence for an effect. Despite careful planning, there was also inevitably a degree of opportunism, such that data included in this study depended on the availability and quality of data that could be obtained from the NESS trial. Furthermore, the generalizability of the findings is arguably questionable, since eight of the nine groups from which data were collected were BPT groups. Given that therapist behavior wasn’t rated, it is not known how the therapeutic orientation of the intervention affected group interactions.

The results from this study suggest that modern video-recording devices and ELAN video-annotation software can be used to identify moment-to-moment group interactions [4]. Researchers can improve the feasibility of this approach by focusing their resources on the interactive categories identified as being most important in this study. Annotating the verbally spoken statements from the video-recorded sessions was notably the most time-consuming process. To date, clinical research has focused on advancing technologies that automatically annotate nonverbal interactive behaviors, for example 3D motion detection [23] or motion energy analysis [24]. Until recently, these approaches have only been possible to use in laboratory or restricted spaces, and therefore lack ecological validity. We therefore propose that future research would benefit more from exploring whether automatic voice recognition and transcription technologies [25] can be used with the approach described in this study. In particular, whether prosodic features of speech shown to be linked to negative symptoms of schizophrenia [26] can be used to improve the accuracy and feasibility of assessing important group interactions. Nonetheless, the research conditions, in particular the video recording equipment, did not appear to impact the clinical integrity of the treatment. Whilst this supports the feasibility of using this approach in common clinical practice, future research is needed to test the potential ease of transferability of the method described.

The results from this study support the hypothesis that interactions between group members in group therapies are linked to improvements in negative symptoms [3]. Negative symptoms are difficult to treat with conventional psychological and pharmacological medications and are linked with poor quality of life and impaired social functioning [27‐29]. Hence our findings, which give insight into what type of group interactions are linked to clinically relevant improvements in this symptom domain, have important clinical implications. For example, clinicians may want to consider emphasizing activities aimed at promoting interactions related to initiating statements, asking questions or giving advice in group therapies.

The finding that self-initiation, advice giving and question asking are associated with improvements in negative symptoms is in line with research within the field of conversational analysis in schizophrenia. Within this literature, studies have found that ‘proactive’ communicative behaviors are associated with outcomes [7, 8, 23]. For example, in doctor-patient consultations, proactive gestures and asking questions have been linked with improved clinical decision making and treatment adherence [7]. Based on this literature and the results from this study, future research should explore the clinical impact of actively enhancing these types of interactions in the treatment of schizophrenia.

The method described may also be useful in identifying beneficial group interactions from the very first session of therapy. In accordance with research on group therapeutic processes [30, 31], individualized psychotherapy [32] and pharmacological treatment [33], the findings from this study highlight the importance of an initial positive response to therapy. Future research should therefore explore the impact of promoting beneficial group interactions from the very first session. In doing so, baseline participant characteristics shown to be related to clinical outcomes in schizophrenia, for example cognitive performance [34], should also be measured and accounted for.