Use of Fidelity Assessments to Train Clinicians in the CBT for PTSD Program for Clients With Serious Mental Illness

Significant progress has been made over the past 25 years in the development of effective interventions for the treatment of posttraumatic stress disorder (PTSD), with the strongest evidence supporting the efficacy of prolonged exposure therapy and cognitive restructuring (1). However, most research on the treatment of PTSD has been conducted in general population samples or with individuals exposed to specific types of traumas (for example, combat and rape). Furthermore, it has been generally accepted that active psychosis and suicidal ideation are contraindications to the treatment of PTSD (2). Because of concerns about the clinical fragility of people with serious mental illnesses, little attention has been paid until recently to the treatment of PTSD among these individuals (3), despite their high rates of trauma exposure (4,5) and co-occurring PTSD (6).

To address this problem, two clinical research groups have adapted principles of treating PTSD in the general population to persons with serious mental illness. Frueh and colleagues (7,8) have developed and pilot-tested an exposure therapy approach for individuals with a psychotic disorder. Mueser and colleagues (9) developed a 12- to 16-week individual cognitive-behavioral therapy (CBT) program for clients with serious mental illness and PTSD. The program includes breathing retraining and education about PTSD, with a primary focus on teaching cognitive restructuring to address core beliefs related to trauma that are thought to underlie PTSD symptoms (10). Two pilot studies (11,12) and one randomized controlled trial (13) have supported the feasibility and efficacy of this program.

Although one of the aforementioned controlled trials demonstrated clinical benefits of the CBT for PTSD program, the clinicians were mainly doctoral-level therapists who were specifically hired and trained for the study (13). This raises the question of whether the program can be successfully implemented in more of a real-world context in which publicly funded services are typically provided by clinicians who have less extensive formal training in the delivery of psychotherapy (14). Most public mental health services for clients with serious mental illness are provided by master's-level therapists, and thus there is a need to determine whether the CBT for PTSD program can be delivered with good fidelity by these clinicians.

There is strong evidence that frontline clinicians can be successfully trained to deliver CBT (15–19). Previous studies suggest that the best way to train clinicians at community agencies is through a review of the treatment model in a didactic seminar, followed by closely supervised case work (8,20,21). In this study, we evaluated whether frontline clinicians treating an ethnically diverse, urban population could be trained in the CBT for PTSD program via training seminars and by provision of session-by-session feedback with a standardized fidelity scale. We report data on therapist fidelity to the CBT for PTSD model, clinician satisfaction with the training program, client retention in the program, and treatment outcomes for clients who participated as training cases.

Methods

The study protocol, informed consent, and all study-related materials were reviewed and approved by the institutional review boards at Dartmouth Medical School and the University of Medicine and Dentistry of New Jersey. The program for training clinicians in the CBT for PTSD model was developed as part of a National Institute of Mental Health grant to evaluate the model when delivered by frontline clinicians in a randomized controlled trial. Although the costs of weekly supervision and expert consultation (described below) were covered by the grant, clinicians billed clients' insurance plans for the CBT sessions.

Therapist participants

The trained therapists were frontline clinicians employed by a large, public-sector mental health provider serving primarily inner-city clients with low incomes. All therapists who expressed interest in the training and were approved by their administrators participated. To maintain continuity in the usual care clients were receiving, their current treating clinician or case manager did not provide the PTSD treatment. However, because the clinicians who treated PTSD were members of clients' treatment teams, the CBT program was fully integrated with other mental health services that clients were receiving.

PTSD treatment clients

Clients with serious mental illness who were recruited from the same large, urban, mental health agency that employed the trainee therapists served as training cases. Clients were enrolled in partial day hospital programs or outpatient treatment programs. As part of the agency's commitment to better identifying and treating clients with comorbid PTSD, screening was introduced into the standard intake procedure. Clients were first screened with a brief trauma history questionnaire. If they reported experiencing a traumatic event in their lifetime, a self-report PTSD screening measure was administered (see below). If they met criteria for a probable diagnosis of PTSD, they were asked whether they wanted more information about the treatment study. All participants gave written informed consent and were offered a $30 honorarium for participation in the assessments.

Clients eligible for the study had a primary chart diagnosis of major depression, bipolar disorder, schizophrenia, or schizoaffective disorder; met State of New Jersey criteria for serious mental illness, including functional disability of a minimum duration of two years; had a current diagnosis of PTSD that was confirmed through a clinical interview; were age 18 or older and capable of providing informed consent; had no psychiatric hospitalizations or suicide attempts in the prior two months and were not dependent on substances in the prior three months; and had sufficient fluency in English to participate in the treatment and complete the assessments.

Measures

CBT for PTSD Fidelity Scale.

The format of this scale was based on the Problem-Solving Treatment Adherence and Competence Scale developed by Hegel and colleagues (22), a seven-item, 5-point Likert scale designed to evaluate the ability of family medicine residents to implement a manualized problem-solving intervention for depression in primary care. Like that scale, the CBT for PTSD Fidelity Scale was developed to provide a combined measure of adherence to and competence in implementing the CBT for PTSD model, which together are conceptualized as “fidelity” to the program. The scale includes 17 items, each rated on a Likert scale (range 1–5), that assess the quality of teaching specific components of the CBT program (for example, psychoeducation, breathing retraining, and cognitive restructuring), structuring of the session (for example, agenda setting and homework review), the interpersonal effectiveness of the therapist, and the overall session quality. Ratings of 1 or 2 represent unacceptable fidelity, ratings of 3 represent minimally acceptable fidelity, and ratings of 4 or 5 represent good or excellent fidelity. The scale was used to monitor therapist fidelity in the first controlled trial of the CBT program (13). [A copy of the scale is available online as a data supplement to this article.]

In this study, after training on use of the scale, the intraclass correlation coefficient for five raters across all scale items, based on 30 sessions rated by all five raters, was .82, indicating good interrater reliability. Cronbach's alpha for the early sessions of the CBT for PTSD program (sessions 1–3), including the 12 items that were rated for all of these sessions (agenda setting, homework review, use of educational materials, psychoeducation, assignment of homework, trauma focus, manual adherence, teaching effectiveness, interpersonal effectiveness, pace of session, reduction of client stress, and overall session quality), was .84, indicating satisfactory internal reliability. Similarly, Cronbach's alpha for the later sessions of the program (sessions 4–16), including the 12 items that were rated for these sessions (agenda setting, homework review, cognitive restructuring, development of action plans, assignment of homework, trauma focus, manual adherence, teaching effectiveness, interpersonal effectiveness, pace of session, reduction of client stress, and overall session quality) was .90, indicating high internal reliability.

Clinician satisfaction.

Two years after the initial training conference, therapists were asked to rate their satisfaction with the supervision and the written fidelity ratings in terms of perceived helpfulness to the clinician (0, did not help; 1, helped; and 2, helped very much), as well as their overall training experience (0, unsatisfactory; 1, satisfactory; and 2, excellent).

Trauma and PTSD screening.

An abbreviated 16-item version of the Traumatic Life Events Questionnaire (TLEQ) (23) was used to screen lifetime trauma history for all clients at the agency. This version of the TLEQ has been successfully used in previous research on the CBT for PTSD program to screen for trauma exposure in this population (13). It assesses the presence of the types of trauma experienced by the participant (for example, “Has anyone threatened to kill you or seriously hurt you?”).

The PTSD Checklist (PCL) (24) was used to screen and identify individuals with probable PTSD. The PCL includes one question for each DSM-IV PTSD symptom, requiring the respondent to rate the severity of each symptom over the past month on a 5-point (range 1–5) Likert scale (range of possible scores 17–85). The PCL has good test-retest reliability and convergent validity in samples of persons with serious mental illness (25). A total score of 45 or greater on the PCL was used to identify cases of probable PTSD (24).

Symptom monitoring.

The severity of PTSD and of depressive symptoms was monitored over the course of the CBT program following the procedures outlined in the treatment manual (9). PTSD symptoms were assessed using the PCL. Depressive symptoms were monitored with the Beck Depression Inventory (BDI-II) (26). The BDI-II is a commonly used self-rating scale that contains 21 items, each rated on a 4-point (range 0–3) Likert scale (possible score range 0–63). Scores of 0 to 10 define the normal range, scores of 11 to 16 define mild depression, scores of 17 to 30 define moderate depression, and scores of 31 and higher define severe depression (27).

The self-report symptom assessments were completed by the client at the beginning of the first CBT session, at every third session thereafter, and at the last session. The results of these assessments were reviewed by the therapist with the client immediately after they were completed; the therapist often worked with the client to address distressing symptoms during the treatment session. For example, if a client reported a high level of distress on the PCL item “Repeated, disturbing memories, thoughts, or images of the stressful experience,” the therapist might suggest helping the person use cognitive restructuring to address those feelings.

The CBT for PTSD Program

The program is provided over 12 to 16 weekly sessions that typically require about six months to complete, guided by three main therapeutic aims (9): to provide education to help clients conceptualize their trauma-related symptoms as common, learned responses to a traumatic event; to teach clients a breathing retraining skill to help them manage and reduce anxiety; and to teach clients the skill of cognitive restructuring to enable them to cope with and change their upsetting emotions through identifying and challenging maladaptive thoughts and beliefs which are often related to their traumatic experiences. Homework assignments are developed at the end of each session to promote generalization to daily living of skills learned in therapy. The program includes client handouts and worksheets and provides clinicians with both clear structure and goals, while also permitting flexibility in tailoring the material to clients' personal experiences and current circumstances. The program also provides guidelines for dealing with common challenges related to serious mental illness, such as affective instability, cognitive impairment, and psychotic symptoms.

Procedures

Methods for training clinicians included didactic presentations, directed reading, videotapes of treatment cases, role plays, supervision, and fidelity-based feedback on practice cases. After an initial two-day on-site training conference, each clinician treated a minimum of one client (practice case) in the CBT program. Sessions were audio-recorded and reviewed by clinical supervisors or expert consultants for quality and fidelity. A one-day follow-up training was conducted one year after the initial training. On-site weekly group supervision was provided by one of two clinical supervisors, with one of three expert consultants usually joining by telephone. Clinic administrators ensured that time was reserved in clinicians' schedules for the CBT sessions, supervision, and attendance at the training conferences.

The two supervisors were doctoral-level clinical psychologists. One supervisor (PTY) had formerly been a member of the clinical staff at one of the sites, where he had received training in the CBT program and had subsequently supervised clinicians at that site in implementing the program for a pilot study (11). The second supervisor (SMD) participated in a subsequent training in the CBT program and treated one patient in the program while receiving weekly supervision from one of the consultants before supervising the frontline clinicians for this project.

The two on-site supervisors and one of the consultants (JDG) were trained in the use of the CBT for PTSD Fidelity Scale by the other two consultants who developed the scale (KTM and SDR). The two on-site site supervisors and the consultant together reviewed items on the scale and then discussed ratings over a series of jointly rated sessions. Finally, interrater reliability of ratings was assessed, as previously described in the CBT for PTSD Fidelity Scale section.

Narrative and quantitative feedback on the CBT for PTSD Fidelity Scale were sent via e-mail to each therapist trainee by one of the supervisors or consultants usually within a week after a session was rated and in most cases before the subsequent session with that client. Group supervision involved a review of the progress of each of the therapists' clients, including new data from monitoring of PTSD and depression symptoms when available, discussion of challenges and potential solutions, and occasional role plays to practice or demonstrate skills for implementing the program and addressing challenges. Supervision did not involve formal review of the feedback to the therapists on the CBT for PTSD Fidelity Scale, although therapists had the opportunity to ask questions about the feedback if they chose. The on-site supervisor led the weekly group supervision sessions, with the consultants providing additional feedback as needed.

During group supervision sessions, the supervisors also collected the progress notes, BDI-II and PCL forms completed by clients, and audio-recorded therapy sessions.

All clinicians completed at least one practice case with a client in the CBT for PTSD program. If a client dropped out before completing the program, then the clinician treated another client as a practice case. A criterion level of skill in treating clients was established with the fidelity scale in order to certify therapists as competent at providing the CBT for PTSD program. Because cognitive restructuring is introduced in the fourth session, which is assumed to be the most critical therapeutic method in the program for improving PTSD (9), certification in the program was defined as achieving an average overall fidelity rating of at least 3.5 (between adequate and good) on the fourth through last sessions of a completed case. Clinicians whose average score with their first case fell below this criterion treated a second client as a practice case, which was monitored with feedback provided by using the same procedures as with the first case.

Results

Therapist and client characteristics

A total of 25 frontline clinicians were trained to provide the CBT for PTSD program. Two clinicians dropped out of the training. Key characteristics of the therapists trained are presented in Table 1. The treated clients' characteristics are summarized in Table 2.

Therapist fidelity ratings

A total of 285 CBT sessions from the 25 clinicians were rated for fidelity, with a mean±SD of 11.40±6.73 sessions per clinician (number of rated sessions ranged from two to 30 per clinician; mode=14, median=10.5). Among the 23 clinicians who completed the training, 21 (91%) met the certification criterion for competence in the CBT for PTSD program with their first completed practice case, and the remaining two clinicians met the criterion with their second practice case. Fidelity ratings are summarized in Table 3. Fidelity ratings averaged 3.6 or above for almost all components of treatment, indicating that clinicians provided the program with good to excellent fidelity across most of its components.

Therapist satisfaction ratings

Satisfaction ratings are summarized in Table 1. On average, clinicians reported that the on-site group supervision was very helpful. The written fidelity feedback was rated between helpful and very helpful, with the mean closer to the latter. They all rated the training experience as excellent.

Client retention in treatment

To ensure that clients would receive at least three sessions of cognitive restructuring, we defined treatment dropout a priori as failure to complete at least half of the minimum 12 sessions specified in the CBT program (12). Using this criterion, eight of 34 clients (24%) enrolled in the program dropped out of treatment. A variety of reasons for dropout were identified, including ongoing litigation related to trauma that exacerbated symptoms, medical problems, and leaving or being transferred out of the program treatment center. Among the remaining 26 clients, 24 completed the 12- to 16-session CBT program. One client completed eight sessions but left early in order to enter a substance abuse rehabilitation program. Another client left after ten sessions by mutual agreement with the therapist that she had gained maximum benefit from the program. No differences in demographic or clinical characteristics at baseline were found between clients who completed the program and those who dropped out (Table 1).

Treatment outcomes

The effectiveness of the CBT program for PTSD was assessed by evaluating improvement in client self-reported PTSD and depressive symptoms on the PCL and BDI-II, respectively, administered at the first session and every third session thereafter. The mean scores for the PCL and BDI-II for clients who completed the program are shown in Figure 1. Participants' PCL scores did not significantly differ between the screening and the first session (58.46±15.85 and 57.48±11.09, respectively).

To evaluate statistical change, we fit random intercept and random slope models for the BDI-II and PCL outcomes (28). Results on both instruments indicated a significant decline in symptom severity over time, with an average rate of decline per assessment of 4.05 points on the BDI-II and 5.02 points on the PCL (p<.001 for both). Random slopes (variation of individual slopes around average) were marginally significant for both the BDI-II (p=.08) and the PCL (p=.09), indicating a trend toward significant variation among the clients in their rate of improvement in PTSD and depressive symptoms over time.

Discussion

The results indicate that frontline clinicians can be trained to deliver the CBT for PTSD program to a criterion level of competence. Among the 23 clinicians who participated in the training, 21 (91%) achieved certification in the CBT for PTSD program with their first practice case, and the other two clinicians achieved it with their second case. Thus, after an initial training and with weekly supervision meetings and fidelity-based written feedback, high levels of competence in the CBT for PTSD treatment model were attained by all the clinicians working with typical clients with serious mental illness—for most, with a single practice case.

The clients who participated as training cases demonstrated substantial and clinically significant improvements in the severity of symptoms of PTSD and depression when treated by primarily master's-level clinicians who were learning the CBT for PTSD program. The average score on the PCL did not decline significantly from the initial screening (58.46) to the first CBT session approximately three weeks later (57.48), but it then fell steadily every three sessions to a final score of 33 at the last treatment session, below the cutoff score of 45 used to screen for PTSD. The average score on the BDI-II fell from 38 at the first treatment session, considered to indicate severe depression, to 8 at the last session, which is in the normal range (27). The clients who served as practice cases were receiving other standard services for their disorders, and minimal exclusion criteria were required for clients to participate in the program. Thus these clients were similar to other clients with serious mental illness and PTSD at the agency, suggesting that the skills for treating these clients may generalize to the broader population of clients for whom the program was developed.

The clinical improvements in the clients are similar to the reductions in severity of PTSD and depression symptoms observed in the initial pilot study and the randomized controlled trial of the CBT for PTSD program (12,13). These results provide external validation for the effectiveness of the training program and certification process by demonstrating clinical improvements based on independent ratings of symptom severity by the clients themselves. The findings are also encouraging because they suggest that implementing the training program for frontline clinicians was associated with immediate improvements in PTSD and depressive symptoms for the first clients with serious mental illness who received the intervention. Thus there did not appear to be a significant learning curve for clinicians to develop competence in the program and provide clinically effective treatment.

The dropout rate of clients in the CBT for PTSD program was 26%. This is similar to the rates of dropout reported in two previous open clinical trials of this program for clients with serious mental illness—14% in the New Hampshire study (12) and 26% in the New Jersey trial (11)—and to the rate of 19% in the New Hampshire-Vermont randomized controlled trial (13).

A slightly higher dropout rate (35%) was reported in a small open trial of an exposure therapy-based program for PTSD for people with schizophrenia or schizoaffective disorder (29). In addition, a meta-analysis of data from 34 randomized controlled trials of CBT for psychosis reported that seven studies (21%) had dropout rates higher than 25% (30). These findings suggest that the rate of dropout in this study is comparable to that in other studies of the CBT for PTSD program and in studies of CBT for psychosis.

In addition to the success of the training model in teaching the CBT for PTSD program to frontline clinicians and the positive clinical outcomes for their clients, the therapists reported high levels of satisfaction with the training. All of the clinicians rated the training experience as excellent, 92% found the supervision very helpful, and 83% reported that the written fidelity feedback was very helpful. Furthermore, only two of the 25 clinicians (8%) dropped out of the training, providing some additional evidence of therapist satisfaction with the training experience.

Two aspects of the training experience may have contributed to the clinicians' satisfaction with the process. First, the session-by-session feedback, including quantitative and narrative input based on the fidelity scale, may have provided timely reinforcement to clinicians for effective implementation of the treatment model and recommendations for change that enabled them to modify and improve their skills over the course of therapy. Second, the routine monitoring of PTSD and depressive symptoms during the treatment program, which indicated an average reduction every three sessions of about 5 points for PTSD symptoms on the PCL and 4 points for depressive symptoms on the BDI-II, may have provided therapists with concrete evidence of their clinical effectiveness, contributing to feelings of personal and professional satisfaction.

The findings suggest that the CBT for PTSD program and the approach to training clinicians in the model have promise for addressing the lack of access to evidence-based practices for treating PTSD in the population with serious mental illness (3). The fact that a large number of clinicians working at five different sites with an ethnically and racially heterogeneous population of clients with serious mental illness could be successfully trained in the model provides additional support for the potential generalizability of the treatment model and training approach. It should be noted that the CBT for PTSD Fidelity Scale was primarily used in this study as a tool to facilitate the training of frontline clinicians and to ensure that they met a minimum level of competence in delivering the program before treating clients in a controlled evaluation of the program. Future research using the scale in this project will address whether certified therapists maintain acceptable levels of fidelity after the initial training period and will explore the potential relationship between therapist fidelity to the CBT model and client outcomes.

Some study limitations should be noted. First, the therapists were volunteers and may not be representative of all clinicians at the sites. Second, we do not know whether the therapists continued to use the model with fidelity once intensive supervision ended. Third, although client self-reports indicated significant improvements in PTSD and depressive symptoms over the course of treatment, assessments by independent evaluators were not conducted nor were outcomes of the treated clients compared with those of a control group.

Conclusions

Despite these limitations, the findings support the feasibility and utility of the model employed for training frontline clinicians to provide the CBT for PTSD program to clients with serious mental illness. Most of the clinicians were able to learn and demonstrate competence in the treatment program with a single practice client. Furthermore, clients receiving treatment from clinicians being trained in the model demonstrated significant improvements in PTSD symptoms and depression. Providing clinicians with regular feedback regarding their adherence to the principles of the intervention may be a useful approach for training clinicians in a new practice.

Figures and Tables