The Family Psychoeducation Fidelity Scale: Psychometric Properties

Five clinical sites, randomly assigned to implement FPE with supports, represented health trusts in urban and rural areas in Norway. Three of the sites were community mental health centers, one was a combined inpatient and outpatient site for assessment of persons with first—episode psychosis, and one was a child and adolescent outpatient clinic. Only one site (the first episode psychosis site) was using the structured FPE format at baseline.

Prior to initiating implementation of FPE, the research team provided to staff from all the study sites (experimental and control) a four-day workshop to introduce both the multi- and single-family FPE format. Each site sent at least two clinicians to the workshop. The research team developed an online toolkit and distributed it to all sites. This toolkit included a description of the evidence-based practice (key points from Norwegian guideline regarding FPE evidence) (Helsedirektoratet 2013), translated version of single and multifamily groups FPE manuals (McFarlane 2004; Miklowitz et al. 2010) and presentations from the workshop. The sites were offered clinical supervision focusing on cases by an FPE clinical expert weekly (first 6 months) to monthly (last 6 months) during the first year, all in a group format by video and telephone). In addition, the five experimental study sites was offered (on site) supervision on implementation and quality improvement (focus on FPE fidelity scale content) by a separate trained expert in implementation supervision every 2 week for 6 months and then monthly for 12 months.

The fidelity assessors were not involved as staff or in supervision in any FPE activity at the sites where they completed an assessment. The 15 fidelity assessors (who included psychiatrists, psychologists, mental health nurses and others with experiences as researchers and/or clinicians in treatment of psychosis) had been trained in doing the fidelity assessment of FPE. They subsequently completed ratings at each site at baseline and after 6, 12 and 18 months. To enhance reliability, two assessors completed each review, conducting a daylong site visit to gather information from the sources specified in the fidelity rating manual (including interview and reviewing written program material). During the site visit, the two assessors independently rated the items on the FPE fidelity scale. After completing their independent ratings, the fidelity assessors compared ratings, identified items on which their ratings disagreed, and reached a consensus rating through discussion. The fidelity raters participated in joint workshops that were held after each round of assessment discussing experiences and results for the fidelity assessment.

The 14-item FPE Fidelity Scale (Dixon et al. 2001; McHugo et al. 2007) rates current behavior and organizational activities on 5-point, behaviorally-anchored items, ranging from 1 = not implemented to 5 = fully implemented, with a rating of 4.0 defined as adequate fidelity. For this study, we dropped the item on prodromal signs because prodromal patients were not included, using the remaining 13 items as the FPE fidelity scale.

The fidelity assessors also completed a survey seven months after the last fidelity assessment, answering questions on the feasibility of administering and scoring the fidelity scale. Questions addressed ease of finding information, making ratings, using various aspects of the scale, and the usefulness of different sources of information and the instructions.

After every fidelity review, we calculated the independent site-level fidelity scores for both fidelity assessors completing the review. The site-level fidelity score is defined as the sum of the item ratings divided by the number of items (that is, 13). To evaluate interrater reliability of the site fidelity ratings, we used the intraclass correlation coefficient (ICC) (McGraw and Wong 1996), based on a one-way random effects analysis of variance model for agreement between the two fidelity assessors on the FPE fidelity scale. We calculated a single ICC, based on 20 paired ratings for the five sites across four assessments.

We used consensus ratings in all subsequent analyses. To examine internal consistency of the FPE scale, we used Cronbach’s alpha, calculating an alpha coefficient for each time period. We examined the item distributions and site scores at 18 months, (mean, standard deviations, and distribution of scores) for full (rating = 5), adequate (= 4), and poor (= 1–3) fidelity.

Finally, we examined longitudinal patterns of fidelity both graphically and statistically. We examined sensitivity for change over time in fidelity using a one-way ANOVA repeated measures design with pairwise post hoc tests with Bonferroni correction for changes between baseline and each of the three follow-up assessments. We tabled frequency distributions of site fidelity scores over time, with specific attention to achievement of high fidelity (\(\ge \) 4.0). Change over time was estimated by calculating the standardized mean difference effect size (Cohen’s d_z) for within-subjects design (Lakens 2013). We examined feasibility using descriptive statistics and paired-sample t-tests for FPE item differences. All data analyses were done using SPSS version 25 (https://​www.​ibm.​com/​analytics/​us/​en/​spss/​spss-statistics-version/​).

Over all items and time periods, exact agreement between assessors on items was good, averaging 88% (see, Table A in the Appendix). The mean exact agreement declined from 95% at baseline to 75–88% thereafter. High agreement at baseline may have been spurious due to many ratings of 1 reflecting a lack of FPE implementation. At the item level, mean agreement between assessors exceeded 80% on ten items and was under 80% on three items: 1 (Family Intervention Coordinator), 13 (Stagewise Provision of Services) and 14 (Assertive Engagement and Outreach).

Two fidelity assessors rated the FPE fidelity scales on four occasions at each of the 5 participating sites (100% completion rate). The intraclass correlation measuring interrater reliability (assuming two assessors) was excellent (0.98). For all subsequent analyses, we reported the findings based on consensus ratings.

Internal consistency (Cronbach’s alpha) ranged from moderate to high: 0.96 (baseline), 0.79 (6 months), 0.97 (12 months), and 0.60 (18 months).

As shown in Table 1, the mean item scores for the five sites at 18 months ranged from 3.40 (Item 1: Family Intervention Coordinator and Item 14: Assertive Engagement and Outreach to 4.80 (Item 2: Session Frequency, and Item 8: Coping Strategies). Ratings significantly increased between baseline and 18 months on several items; Long-term FPE, Psychoeducational Curriculum, Structured Problem Solving and Stage-wise Provision of Services. Notably, by 18 months, ten of the items reached a mean score of 4.0 or above, which is the benchmark for adequate fidelity. Fidelity reviewers used the entire rating scale from 1 to 5 for all 13 items, suggesting that the rating scale captured the observed variability in actual practice.

We visually inspected the graph of change across the 18-month period for the five sites, as shown in Fig. 1. At baseline, the mean site-level fidelity rating for the total scale was 2.35, suggesting that some implementation of family implementation was occurring at baseline, but nonetheless resulting in overall very low fidelity. By 6 months, mean fidelity had increased to 3.71, a mean increase of 1.35, but not significant, (t =  − 3.08, p = 0.22). At 12 months, the level of fidelity declined to 2.98, which was not significantly different from baseline (t =  − 0.53, p = 0.62) At 18 months, fidelity increased to 4.11, a mean increase of 1.75 from baseline, (t =  − 2.55, p = 0.38), thereby exceeding the benchmark for good fidelity. The standardized mean difference effect size (Cohen’s d_z) was 1.14.

We also examined change over time looking at the percentage of sites attaining adequate fidelity (4.0 or higher) at each time period. At baseline, one site (20%) had already achieved adequate fidelity to FPE at baseline. By 12 months, one additional site had achieved adequate fidelity, and by 18 months, three sites (60%) achieved adequate fidelity.

The 15 FPE fidelity assessors rated feasibility of the fidelity review process based on fidelity reviews both for sites receiving implementation support and seven control sites that did not. On average, fidelity assessors completed fidelity reviews at 11 sites across the four points in time. For most items assessors reported that both finding information and making ratings were easy. One exception was that, assessors found it difficult to find information on the quality of practitioner-consumer-family alliance. The interviews with clinicians were the most useful sources of information, while interviews with leaders and written procedures were less useful. They reported that the format of the fidelity scale was clearly set out and that the instructions were clear.

The strength of the study was implementation in a national, public-funded, “real world” clinical system serving nearly all patients with psychotic disorders in each health trust’s catchment area. Limitations included the small number of study sites for generalizability, the therapist self-report bias and the lack of interviews with patients and their families, and the absence of direct observation of FPE sessions which affects the FPE fidelity scale validity.

The current study is one of few (Kealey et al. 2015; McHugo et al. 2007) investigating both use and psychometric properties of the FPE fidelity scale. The FPE fidelity scale has good psychometric properties and feasibility for evaluating the implementation of FPE programs. We conclude that the FPE scale is feasible for evaluating clinical services where structured family interventions are offered. In future revisions of the FPE fidelity scale collection of observational fidelity data should be considered. Nevertheless, a larger study could provide more robust conclusions and investigate the predictive validity of the FPE fidelity scale on long-term outcome.