Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: a Delphi study

Fidelity is the most commonly reported measure of implementation outcomes[29]. The US National Institutes of Health Behavior Change Consortium has described five aspects of fidelity as follows: study design, provider training, treatment delivery, treatment receipt, and treatment enactment[30]. Although all of these aspects of fidelity can ultimately influence implementation and outcomes of a program, evaluation of treatment delivery is often prioritized to verify if an intervention is being provided as intended[31]. As described by Perepletchikova and Kazdin[32], fidelity in treatment delivery can be further specified as provider adherence (i.e., utilization of specified procedures and engagement in specific tasks and activities) and treatment differentiation (i.e., care that is reflective of critical dimensions of CCC rather than specialty mental healthcare). We chose to emphasize adherence and differentiation over other aspects of delivery, such as provider competence (i.e., skillfulness in delivering treatment), given the multitude of factors that might impact the skill set of BHPs and to avoid blurring fidelity assessment and individual performance appraisal.

Having identified adherence as the most relevant focus for our investigation of BHP fidelity, our conceptual framework for this study was guided by Carroll et al.[33], who described intervention adherence as the “bottom line” of implementation fidelity. Carroll et al.[33] suggest that adherence mediates the impact of an intervention on outcomes, with adherence reflecting how much of an intervention’s content, or active ingredient, is delivered. Estimates of adherence are quantifiable, based on the frequency, duration, or coverage of the intervention content. Furthermore, some intervention content may be prioritized over other content if it is considered ‘essential,’ or required for the intervention to produce the expected outcome. This conceptualization of fidelity guided the specific study approach described below.

The modified Delphi approach refers to a structured, iterative process of collecting and summarizing opinions from content experts with the primary goal of consensus building[34‐36]. This approach is often used to address problems of clinical practice when there is incomplete knowledge on a subject[37, 38]. The modified Delphi process polls a panel of experts who provide feedback about an evolving set of statements during several rounds of data collection. As applied to the current instrument development study, the Delphi method helps ensure content validity and an appropriate item pool that sufficiently represents behaviors of BHPs in CCC. We employed three rounds of data collection with the aim of reaching at least 80% consensus among participants on each item[34]. Participants were asked to re-rate items that had less than 80% consensus after round one, providing a process by which participants could anonymously agree or disagree with other experts. An advantage to this approach is that it provides anonymity, thereby decreasing the likelihood that some participants will modify their responses based on the opinions of highly influential experts[35].

Purposeful sampling was used to recruit VA and non-VA experts in CCC known to the study investigators as expert clinicians, administrative leads, or content experts based on previously published research. Although panel sizes for Delphi studies vary considerably depending on the goal of the project and availability of experts[36], we invited 33 experts to participate with the goal of recruiting at least 20 experts through three rounds of data collection. In round one, 88% (n = 29) of experts contacted returned completed surveys. In round two, 86% (n = 25) of round one participants returned completed surveys. In round three, 100% (n = 25) of round two participants returned completed surveys yielding a 91% response rate across rounds. Thus, the overall response rate for those participating in the entire study was 76%.

Guided by the fidelity framework[33] and related literature[19, 32] noted above, initial survey items were generated and refined over a six-month period by the study team. Items were developed to represent the critical components of the CCC model enacted by BHPs based on a qualitative review of published scientific literature[1, 6‐8, 12, 13, 24‐26, 39‐41], unpublished clinical practice guidelines from clinics using a CCC platform, clinical and administrative experience of the research team, and informal polling of CCC experts known to the research team but not asked to be part of the Delphi study[19, 42]. The CCC-related publications reviewed were in-depth descriptions of CCC model aims, procedures, and components because empirical studies relating CCC components to clinical outcomes have not been conducted. We therefore translated these descriptions of CCC models into a pool of items representing discrete BHP behaviors so that expert opinion could be used to evaluate the relevance of each behavior within the conceptual domain of BHP protocol adherence. To ensure breadth in the development of the items, several aspects were considered. First, we attempted to identify components that reflected either BHP behavior related to the delivery of clinical services to patients or related to collaboration with the primary care team. Second, items were classified as representing either structural elements of care (the framework for CCC service delivery) or process elements of care (the methods or procedures related to CCC service delivery[19]). Third, subgroups of the final set of items were classified by content domains representing core components of BHP practice as determined by the research team: clinical scope and interventions; practice and session management; referral management and care continuity; consultation, collaboration, and interprofessional communication. Fourth, items were further classified by the study team as essential, compatible, or prohibited when working in CCC[31]. Essential behaviors were those considered to be highly reflective of the CCC model and required for good practice, even if that behavior did not necessarily need to happen in every encounter. Compatible behaviors were those that were acceptable when working in a CCC model, but are not required or specific to CCC. Prohibited items were those behaviors that should be clearly avoided when working in a CCC model because they were inconsistent with CCC.

The round one survey included a brief introduction to the purposes and goals of the study followed by a definition of CCC to help orient participants to the general model of behavioral health in primary care. As noted previously, nomenclature and definitions of CCC-type models of care vary. Thus, the definition of CCC provided to participants was selected from the VA Center for Integrated Healthcare Clinical Operations Manual[43]:

‘Co-located, Collaborative Care services are offered by an embedded behavioral health provider. This approach involves providing services to primary care patients in a collaborative framework within primary care teams. Behavioral health visits are brief (generally 20–30 minutes), limited in number (1–6 visits with an average of between 2 and 3), and are provided in the primary care practice area, structured so that the patient views meeting with the behavioral health provider as a routine primary care service and medical providers are supported across a broad scope of behavioral health concerns.’

Note that we did not provide any additional review of published CCC literature because the study investigators felt this information was sufficiently captured in the content of the item pool. Participants were asked to classify the 56 items in the round one survey that reflected BHP behaviors related to the CCC model as essential, compatible, or prohibited when working in CCC. Open-ended questions followed each item so that participants could suggest modifications to the item and provide a rationale for suggested changes. Additionally, several open-ended items at the end of the survey requested information on what BHP behaviors should be added to the item pool. Finally, participants were asked to report professional background characteristics, such as their primary occupational setting.

The round three survey was modified to reflect the results from round two. First, participants were asked to comment on and indicate their agreement (yes/no) with a subsequently revised definition of CCC. We provided an additional opportunity to modify the CCC definition and label because qualitative feedback in round two pointed to concerns that ‘co-located, collaborative care’ was an unclear term. Therefore, we asked respondents to select their preferred label for the model described. Second, participants were asked to rate for a final time the remaining 13 items that did not reach consensus after round two. Third, participants were presented the 26 items that met consensus after round two for informational purposes only. An example of how items were presented to participants across rounds is provided in Figure 1.

All communication between the research team and participants was conducted by email. The study PI (GPB) and Co-I (JSF), both clinical research psychologists at the VA Center for Integrated Healthcare were primary contacts for the study and initiated all email correspondence. All emails were sent to individuals, rather than the full group of participants, from the investigators’ personal VA email accounts to maintain privacy. Potential participants first received an email briefly introducing the purpose of the study and nature of their participation, including the goal of building consensus with other experts by completing up to three surveys. Attached to this email was a document that included informed consent information and the round one survey. Note that we sought and were granted a waiver of written informed consent from our Institutional Review Board because of the low-risk nature of the study and the goal of increasing feasibility by reducing response burden to the greatest extent possible. Experts were asked to read the consent document, contact the study team with any questions or concerns as necessary, and continue on to complete the survey to indicate their agreement to participate. To encourage participation, up to three personalized email reminders were sent per round. When participants returned their completed survey by email, they were provided with a thank you reply along with information indicating that they would be contacted within several weeks with the round two survey comprised of an updated survey reflective of round one findings. Similar information was provided to participants when they completed round two surveys in anticipation of round three. Those participants who ultimately did not complete the round one survey were not contacted to complete the round two survey. All participants who completed round two subsequently completed round three. The conduct of this study was approved by the VA Western New York Healthcare System Research & Development Committee and Institutional Review Board.

As shown in Table 1, self-reported professional background among participants that completed all three rounds (n = 25) confirmed that the sample met the goal of reaching clinical, administrative, and research experts with extensive knowledge of CCC. Participants were primarily doctoral-level psychologists (92%), current or former BHPs in CCC (96%), and held an administrative leadership position related to the implementation of CCC in their institution (68%). Experts’ primary occupational setting was VA (40%), DOD (28%), or other (32%), including academic or community health centers. Many participants had authored or co-authored a peer-reviewed publication that was an empirical investigation related to CCC (46%) or a descriptive paper regarding CCC models (55%). In examining the characteristics of the non-responders, seven of eight non-responders were psychologists and the remaining non-responder was a physician. Two non-responders had primary affiliations with the DOD, three non-responders were affiliated with VA, and three non-responders were affiliated with academic health centers suggesting adequate representation across settings.

Based on participant feedback, the original definition of CCC presented in survey round one was expanded considerably, with 84% of participants agreeing on the final definition presented below. Modifications to the original definition are bolded:

‘Co-located, Collaborative Care (CCC) services are offered by an embedded Behavioral Health Provider. This approach involves providing patient-centered behavioral health services to primary care patients in a collaborative framework within primary care teams that utilize a shared medical record. Behavioral health appointments are usually brief (30 minutes or less), typically limited in number (1–6 visits with an average of between 2 and 3 per care episode), and are provided in the primary care practice area. These appointments are structured so that the patient views meeting with the Behavioral Health Provider as a routine primary care service. Primary Care Providers maintain responsibility for patient care decisions, but are supported by the Behavioral Health Providers who emphasize patient self management across a broad scope of concerns, including common mental health diagnoses, physical health issues, and prevention. CCC Behavioral Health Providers may also coordinate, or blend, their services with other forms of primary care based behavioral health ( e.g., depression Care Management).’

Although modifications to the definition led to agreement about the general features of the model, the term ‘co-located, collaborative care’ was not well-agreed upon. Participants ultimately preferred the term ‘primary care behavioral health’ (41.7%) over ‘co-located, collaborative care’ (16.7%), ‘integrated primary care’ (29.2%), or other terms (‘collaborative care’ or ‘integrated behavioral health’ 12.5%).

Table 2 displays the original item wording, component domains, classification applied by the study team, final item wording, and the consensus rating following round three. Following round one of the Delphi process, 19 survey items met consensus, leaving 37 items to be rated again in round two. All but one item (item 35) was modified for the round two survey based on qualitative feedback received. Most changes to items were focused on improving clarity, eliminating terms or acronyms that were used exclusively within VA settings, or adding qualifiers (such as ‘typically’) in describing certain behaviors. Other changes were more substantive, such as modifying item 18’s emphasis on the content of progress note documentation to the time spent on documentation. Additionally, two new items suggested by participants in round one were constructed by the study team to address providing feedback to primary care teams about initiating or modifying psychotropic medications (item 57) and participation in clinical pathways for common health conditions (item 58) for a total of 39 items to be rated in round two.

At the end of round two, 26 of the 39 items met consensus, leaving only 13 items for round three. Eight of the 13 items in round three were again modified based on qualitative feedback, but no additional items were added to the survey. At the end of round three, 54 items (93%) from the 58 total items across rounds met sufficient consensus to be included in the final survey. The four items that failed to reach consensus (items 14, 33, 34, 49) had final rankings that ranged from 68.0% to 79.2% agreement and covered a range of behaviors. Our ability to explain the lack of consensus for these items was limited due to our small sample size. However, when we inspected response patterns by primary occupational setting, VA respondents tended to consider item 14 (use of educational handouts) to be more essential (40.0%) than respondents from DOD (28.6%) or other settings (12.5%). For item 33 (use of open-ended questions), VA respondents tended to classify this behavior as prohibited (30.0%) more frequently than participants from DOD (0.0%) or other settings (12.5%). For item 34 (seeing patients for 6 sessions or less), VA participants (90.0%) and DOD participants (85.7%) tended to rate this item as essential more frequently than participants from other settings (50.0%). Finally, for item 49 (having appointments with couples/families), participants from other settings were more likely to view this behavior as essential (62.5%) compared those participants from VA (20.0%) or DOD (14.3%).

The present study is the first step towards establishing the PPAQ as a useful fidelity assessment tool of the CCC model. The high participation rate among a panel of experts and high level of consensus required (≥80%) help support the PPAQ content validity. However, this study does have some limitations. Foremost, results of this study are based primarily on expert opinion because there is little empirical evidence regarding which components of CCC, operationalized as BHP behaviors, are predictive of clinical outcomes. Thus, this study represents a first step in providing a tool for future dismantling studies that can evaluate empirically which aspects of protocol adherence (i.e., combinations of items within or across essential, compatible, or prohibited categories) are related to patient outcomes. The PPAQ was also developed to focus only on the adherence of BHPs to the CCC model and does not assess fidelity at a system- or patient-level[29]. We made this choice for several reasons. Contextual factors in the conceptual framework of fidelity we adopted[33] are proposed to impact outcomes only indirectly, as moderators, through provider adherence. Therefore, it seemed logical to first quantify provider intervention adherence because of its direct and proximal association with clinical outcomes. Due to our interests in expanding the evidence base for integrated healthcare, we also prioritized the development of a measure of adherence that could address the increasing pressure for clinical outcome data demonstrating the effectiveness of specific formulations of CCC. Additionally, as noted previously, we aimed to develop an efficient and easy-to-use measure to guide implementation of CCC using BHP self-report.

Whereas we included BHPs in the development of the PPAQ, more formal study of provider reactions to and acceptability of the measure is warranted. Future studies will need to consider the degree to which social desirability plays a role in shaping their responses. For example, BHPs may feel they need to report high levels of fidelity by either over-reporting essential behaviors and/or under-reporting prohibited behaviors if they believe that their report is linked to individual performance evaluation. Therefore, if the PPAQ was being used for training and program development, it would be important to provide education to BHPs regarding the purpose of fidelity assessment as a learning opportunity and to remain supportive when introducing the PPAQ to establish a norm of routine fidelity assessment. When used in future wider-scale research on patient outcomes, rather than as a tool for training, the nature of the PPAQ as a self-administered instrument may be beneficial for reducing the impact of social desirability[45], particularly if the research design allows for administration over the internet or other approaches that preserve respondent anonymity.

Although this study demonstrates that the PPAQ items are content valid, future research needs to be conducted to establish the reliability and other aspects of validity. For example, although the PPAQ was designed to measure BHP behaviors in CCC, we did not examine whether the PPAQ is able to discriminate CCC from other models of integrated healthcare, such as care management[46, 47] or to predict CCC outcomes. While necessary, this type of work may be difficult to conduct due to the small number of BHPs working in these settings, the difficulty of identifying and accessing providers, and the large sample size necessary to conduct the traditional factor analysis that typically is used when validating assessment measures. However, an investigation to examine the reliability and validity of the PPAQ with BHPs is currently underway by the research team.