Background
The practice of regularly evaluating one’s mental and emotional states in search of patterns or trends is a common informal practice that has long been found to facilitate increased social and emotional improvement [
1,
2]. When formally conducted in clinical settings, the utilization of Patient-Reported Outcome Measures (PROMs) and their results to drive clinical decision making is known as Measurement-Based Care (MBC) [
3]. MBC has been investigated for its added clinical value and utility, and research findings repeatedly indicate the multitude of benefits to psychiatric patients and clinicians alike [
4‐
8].
Though research is supportive of MBC as an evidence-based practice (EBP) for mental and behavioral healthcare, the literature also suggests significant barriers to successful implementation [
4,
9‐
12]. Providers are more than capable of administering and interpreting psychosocial assessments; however, MBC has proven challenging to integrate into care [
13‐
16]. Previous studies have found that less than 20% of clinicians or providers regularly administer MBC to their patients despite the availability of a wide range of validated measures that reliably reflects changes in symptom severity [
17,
18]. In addition, over 60% of clinicians have never used a Measurement Feedback System (MFS) [
19,
20], a web- and mobile-based electronic platform that assists administration, scoring, and interpretation of multiple measures to facilitate MBC. Clinicians have regularly identified barriers to routine MBC implementation (e.g., lengthy assessments), which affects time and patient commitment, and personal endorsement of symptoms [
21‐
23].
There are hundreds of dissemination and implementation frameworks intended to guide the integration of an EBP across settings [
5,
24,
25]. To enhance the integration of MBC, this study narrows on the Practical, Robust Implementation and Sustainability Model (PRISM) [
26] as the guiding framework. The PRISM model builds off pre-existing models (i.e., Diffusion of Innovations, the Chronic Care Model; Model for Improvement) and merges with the RE-AIM framework [
27] to act as a critical tool to translate research into practice. The model itself involves several domains: intervention (e.g., new EBP), recipients (individuals who receive or implement the EBP), external environment (e.g., the continuing pandemic), and implementation (e.g., actual use of the new technique) and sustainability infrastructure (e.g., supports put in place by a hospital to facilitate and maintain the new procedure). These major domains provide structure for the broader context and identify critical stakeholders who inform program implementation and dissemination. The program can then be evaluated within the RE-AIM framework to evaluate Reach (e.g., equitably implementing treatment to target population), Effectiveness (e.g., impact of intervention in improving clinical outcomes), Adoption (e.g., recipient’s commitment level/status), Implementation (e.g., fidelity to intervention delivery), and Maintenance (e.g., long-term intervention sustainability). Previous research integrating PRISM has shown the most successful implementation occurs when three or more domains (e.g., intervention, recipients, external environment) are considered along with at least one element within these domains (e.g., intervention (strength of the evidence base), recipient (management support and communication), or implementation (adaptable protocols and procedures) [
28‐
32]. PRISM promotes identifying and documenting key factors or leverage points at multiple levels of internal and external influence for implementation success.
In a healthcare setting, it is critical to examine the organizational and structural barriers in conjunction with psychological barriers to integrate an effective EBP such as MBC. There are many approaches to understand ambivalence to MBC that include surveying or interviewing stakeholders involved in the intervention-organizational perspective and recipients-organizational characteristics domains within the PRISM implementation model [
26]. From this context, MBC (intervention) is received by patients (recipients) and implemented by clinicians and staff (recipients) in mental healthcare settings within the broader context of a global pandemic (external environment). At the same time, a sustainability infrastructure needs to be created to ensure systematic support and ongoing consultation by the healthcare setting to facilitate utilization of MBC. It is necessary to take multiple approaches to understand an organization’s current infrastructure supporting an EPB. As such, we sought to better understand MBC supports currently embedded within the organization to identify changes that can be made to improve adoption by the recipients. In addition to the team’s expertise functioning within this institution, one promising method to disentangle the organizational perspective is using the Brainwriting Premortem Method (BPM) to hear directly from those responsible for delivering and assisting MBC [
33]. The BPM was designed to identify failures and limitations a-priori to ensure better implementation and dissemination of EBPs. BPM uses a group brainstorming approach with a flexible interviewing style to receive input from multiple participants at once. This method supports participants’ ability to share ideas simultaneously and to agree or counter other ideas, thus ensuring more rapid and efficient data collection when compared to traditional focus groups. Utilizing the BPM can assist in better MBC integration with downstream improvements in health system performance and ultimately health outcomes. This method has been successfully utilized as an important tool guiding implementation; specifically, to inform emotion regulation prevention program development with college students [
34], improve care coordination for rural veterans [
35], to inform the scale-up of a nursing intervention [
36], improving electronic health records with user-centered design [
37], inform adoption of technology in higher education settings in South Africa [
38], and assess suicide-specific training for mental health providers of active-duty military personnel experiences [
39].
Present study
It is apparent that standardized training is necessary for successful implementation of MBC [
40,
41], though which aspects are critical for successful and relevant training have yet to be identified. To date, there are no studies that have implemented BPM focus groups in psychiatric settings, which include multidisciplinary healthcare professionals, to improve the effectiveness of MBC implementation. Instead, the literature delineates a plethora of barriers with intermittent success stories. As such, this study is focused on identifying the obstacles as well as facilitators to efficient implementation prior to any formalized training. This study applies the PRISM framework to evaluate the recipients, intervention, and external environment of MBC to create an implementation and sustainability infrastructure which ensures effective use of MBC and informs training efforts. Using a virtual adaptation of the BPM, the current study conducted focus groups to assess the understanding and opinions of MBC implementation by clinicians and staff, who have been involved in incorporating MBC into their usual care. Furthermore, the utility and feasibility of a virtual setting for conducting the focus groups was evaluated. Together, using the mixed method design, data was organized within the PRISM framework to inform current MBC implementation and build a plan for sustainability.
Discussion
Measurement-Based Care (MBC) is an evidence-based practice that has demonstrated capability to enhance mental and behavioral health care when appropriately implemented. This study utilized the Brainwriting Premortem Method (BPM) [
33] to collect data on the current status, opinions, and clinicians’ understanding and attitudes of MBC in an adult outpatient psychiatry setting to understand barriers and facilitators of MBC within the PRISM framework [
26]. Virtual focus groups were intended to engage stakeholders and adjust the implementation process in a setting where rapid implementation was conducted without formalized training. The researchers aimed to receive feedback from multidisciplinary professionals, including clinicians and staff, to further the development of MBC training and systematically re-implement MBC at a departmental level. A virtual BPM was born out of the limitations of COVID-19, and this study provides a unique opportunity to determine the successes and limitations of translating this method to a virtual platform while maintaining protective pandemic precautions. The BPM was successful in engaging participants from each focus group and retrieved meaningful information from all participants. Participants’ active engagement in focus groups resulted in 291 individual codes from clinicians and 97 from staff, based on the similarity of their original responses. There were several benefits of virtual focus groups. First, the virtual BPM allowed participants to share their ideas without a heavy time burden, considering travel was not required to participate [
44,
45]. Although unexpected technical difficulties or disabilities interfered with full anonymity, participants still provided rich and descriptive responses to each question. Considering that the primary purpose of the BPM was to facilitate an open discussion in an environment that can mitigate psychological burden when participants disagree with others’ opinions, this virtual BPM was successful in gathering various responses, opinions, and attitudes on MBC from different clinicians and staff, considering the wide range of response topics and themes arose. Additionally, clinic leadership personnel were not part of the focus groups, and participants were assured anonymity, especially to their leaders, with the goal of providing as much openness as possible during the focus groups. For the researchers, the use of Google Docs for the virtual BPM reduced the time burden by decreasing the manual labor of transcribing focus group participants’ responses. Despite instances when participants could not type (e.g., network disconnections, disabilities), most provided clear responses in sentences, phrases, or words, which facilitated easy data organization once the focus groups were completed.
On the other hand, there were limitations of virtual BPM. This method was restricted in its ability to retrieve the same amount of information from each participant. Although participants answered most questions, this study did not provide strict guidelines for participation (e.g., each participant should answer all questions, each participant can only answer in less than 3 sentences for each question). This flexibility may have resulted in a difference in response quantity per participant, since their engagement level (i.e., number of verbal/written responses) varied.
When the responses were divided into barrier, facilitator, and protocol categories, the ratio differed by clinicians and staff. Although barriers accounted for a significant portion of clinician responses, there was more of a balance between barriers and facilitators noted for the clinicians as compared to the staff responses. Despite these barriers, as recipients of MBC, clinicians appear aligned with the “shared goal and cooperation” of implementing MBC, implying that the clinicians were tuned in to both barriers and facilitators of conducting and incorporating MBC into their practice and interventions. These results also align with the quantitative analysis about the further follow-up questionnaire surveying clinician attitudes and satisfaction with MBC, which showed clinician’s overall positive attitudes toward MBC, actual commitment in implementation, and their willingness to continuously incorporate MBC into their practice. On the other hand, staff reported significantly fewer facilitators than barriers. Clinicians intervene directly with patients in a formal care setting, whereas staff may receive more informal feedback from patients. Additionally, clinicians may be biased towards appreciating facilitators as they have more time to develop a relationship with patients as compared to staff and experience the external environmental expectations that they integrate MBC as a part of their standard care. Despite the lack of “readiness” for implementing this intervention illustrated by the lack of formalized MBC training at this institution, clinicians were knowledgeable or motivated to incorporate MBC based on their general attitudes toward MBC [
46]. Clinician’s positive MBC attitudes were supported via quantitative analysis reflecting clinician’s understanding of the benefits of MBC and motivation for continuous MBC incorporation into their practice. Conversely, staff appeared to perceive more patient barriers related to their experiences providing a general explanation of MBC, guiding patients on the MFS including step by step procedures (i.e., help patients log-in, facilitate the completion of PROMs), and problem-solving when patients have difficulties (e.g., technical issues) in completing the measurements. As such, clinicians may see the positive effects of utilizing MBC with their patients that the staff does not, since staff are not directly involved in that aspect of care. It is possible that staff were unable to recognize the rationale or benefits of MBC as an EBP that ultimately improves intervention outcomes, as MBC was quickly introduced without formal training. This is hypothesized as a likely reason that staff expressed significantly more barriers than clinicians.
Furthermore, differences were observed between clinicians and staff responses according to the focus area (i.e., patient, clinician, staff, technology, institution) of barriers/facilitators. Clinicians reported patient-related barriers the most, followed by clinician-related and technology-related barriers. This data could reflect the clinician’s primary focus (i.e., patients) and their perception of primary obstacles when incorporating MBC in their usual care. These results are aligned with previous studies that investigated the barriers to MBC implementation on a different level, which showed multiple existing challenges on a patient and clinician level, though organizational and system level barriers were also identified [
12]. Alternatively, staff reported technology-related barriers most frequently, followed by patient-related barriers, which could reflect the fact that staff often problem-solve both patients’ and clinicians’ technological issues. Regarding facilitators, clinicians reported patient-related facilitators and clinician-related facilitators (e.g., patients’ positive attitude on MBC, clinicians’ recognition of useful PROMs in their practice). Similarly, staff reported staff-related facilitators most often followed by institution-related facilitators. This data shows how each recipient group perceives different barriers or facilitators for implementing MBC, and provides insights considering that no known prior studies have identified these implementation barriers/facilitators from staff members who are directly facilitating patient completion of PROMs regularly. Table
6 summarizes the critical factors for implementation of MBC divided by each PRISM domain. These results implicate the need for further MBC training and re-implementation of MBC (i.e., ‘adaptable protocols and procedure’ elements in the implementation and sustainability infrastructure domain) in this adult outpatient psychiatry setting. By better understanding the recipient perspective, this data helps to inform holes in implementation and critical adoption barriers to increase fidelity and patient uptake. Previous research on MBC implementation was primarily focused on the training of clinicians, since these groups interact with patients the most [
47]. However, in larger institutions, collaborative work among all recipients and solving the issues that each group faces could be the key to successful MBC incorporation to the usual care. Furthermore, the overall results demonstrate the need for organizational and system level changes to the external environment that can facilitate training which could include all recipients of MBC.
Table 6
Deconstructing the PRISM framework for MBC in adult ambulatory psychiatry based on the results of the virtual BPM
External Environment
| Adult ambulatory psychiatry - Department of Psychiatry and Behavioral Medicine’s (PBM) outpatient clinic in a US regional hospital | Clinicians | Agents of administration of measurement-based care (MBC) - Includes MD, NP, PhD, and LCSW |
Staff | Agents assisting implementation of MBC |
Patients | Receiving MBC by clinicians and provided help by staff |
Intervention
| Measurement-Based Care - Evidence-based practice (EBP) - Regularly evaluating patients’ mental and emotional state to facilitate treatment and inform clinical decision making - Utilizes patient-reported outcome measures (PROMs) | Clinicians | Implement MBC into their clinical practice - Utilize MBC to guide, track, and monitor treatment and patient’s symptoms - Review the results and have discussion with the patients - Incorporate PROMs results and inform clinical decision making |
Staff | Assisting administration - Prompt and guide patients to complete their PROMs - Problem-solve technical issues |
Patients | Participate in MBC - Complete PROMs regularly - Review their PROMs’ results and participate in collaborative evaluation with the clinician - Embrace information maximizing patient-centered care |
Implementation & Sustainability Infrastructure
| Focus Groups & Trainings - The present study conducted focus groups as a precursor to help develop and implement appropriate standardized MBC trainings - Brainwriting premortem method was utilized - Both clinicians and staff focus groups were conducted | Clinicians | Clinician focus groups resulted in 291 individual codes - Similar number of barriers and facilitators were identified - Main negative themes indicated clinician’s difficulty with patients (e.g., non-adherence), time burden, skepticism on PROMs’ usefulness, and lack of designated staff when utilizing MBC - Main positive themes indicated clinicians’ positive attitude toward MBC implementation into their practice |
Staff | Staff focus groups resulted in 97 codes - More barriers (67%) were identified than facilitators (24.7%) regarding MBC - Staff raised technology/virtual visit difficulties, patients’ negative attitude towards MBC and MFS, as well as chart integration issues |
Reach & Effectiveness
| Focus group results indicated current limitations and future directions for improved reach and effectiveness | Both clinicians and staff | Suggestions - Need for designated MBC/technology staff - Address patient non-adherence - Address ways to utilize MBC with certain type of patients (e.g., how to address high suicide risk patients, substance use/elderly patients are less likely to use MBC) - Plan for easy chart integration and visualization - Increase overall positive attitude and understanding of patients toward MBC |
Adoption
| Adopt MBC to adult ambulatory psychiatry | Clinician | Perceive the need for systematic and standardized training to better implement MBC into their clinical practice |
Staff | More resistant to implementing MBC policies and appear to have more negative perceptions of MBC and identify greater barriers to adoption |
Implementation
| Implement MBC to adult ambulatory psychiatry | Both clinicians and staff | Although MBC is already implemented, results of the BPM suggest that specific training is warranted and desirable for both staff and clinicians |
Maintenance
| Sustain the intervention (i.e., MBC) and systematically institutionalize MBC | All recipients of MBC | Implement routine trainings and assessments for implementation failures - Ensure systematic support and ongoing consultation by hospital to facilitate continuous utilization of MBC |
Further analyses that identified the main themes from each focus group provided concrete themes for adapting the implementation process. Contrary to initial analyses, clinicians reported more negative themes than positive themes when individual responses were aggregated based on the specific content. Based on the review of negative themes, clinicians expressed patient-specific barriers, such as patient non-adherence, patients’ lack of understanding of MBC, and difficulty in utilizing MBC with certain types of patients. These results align with previous research identifying patients’ non-adherence issues [
48] that could be resolved with clinician training on the ways to deliver and communicate with their patients [
46]. Considering that the patient characteristics are crucial to understanding the recipients, identifying the patient’s burden, demands, and knowledge and beliefs would be one of the keys for successful implementation. In alignment with previous studies on the barriers of MBC, clinicians also reported an overall burden, time constraints, and skepticism on PROMs’ usefulness [
49,
50]. Notably, clinicians participating in this study were institutionally required to engage their patients in MBC, and recipients still reported several difficulties that contribute to decreased utilization of MBC, and requested an improved implementation process, standardized training, and support for MBC. However, clinicians still reported valuable positive themes and benefits of MBC in their practice (i.e., reviewing PROMs results with patients; collaborative discussion). Considering that collaborative evaluation between clinician and patient is a key component of MBC [
51], it should be emphasized continuously. Clinicians noted an overall positive view of MBC with specific evidence, including noticing certain patient types that are good candidates for MBC, and acknowledging the benefits of utilizing the PROMs results during their clinical practice. Similar to the previous analysis on barriers and facilitators, the staff group raised significantly higher issues and negative themes than positive themes or benefits of MBC. Since staff are the first-line personnel guiding and problem-solving patients confronting challenges on PROMs completion, these findings could imply setbacks and additional burdens when interacting with patients. This may also reflect limitations at the departmental level of support for staff and lack of feasible solutions for various issues that patients encounter in the process of participating in MBC. Related to the difficulties and issues that staff were facing, they reported a positive attitude toward and request for further in-depth MBC training and suggested their preferred training modality, which should be considered for future MBC training plans.
Strengths and implications
The current study has several strengths. This study took a novel approach to integrate the implementation framework and BPM focus groups to infuse MBC into a medical setting. To date, the current study is the first to conduct virtual focus groups with both clinicians and staff to ensure effective MBC implementation in adult ambulatory psychiatry. The inclusion of front line staff stakeholders in the MBC process is often overlooked and provides a unique perspective on the shortcomings of MBC implementation. Comparing different perspectives provided practical implications for MBC implementation, considering the variability in standout themes. In addition, the inclusion of a focus group in the planning of this study’s implementation has illuminated a breakdown in understanding the roles recipients (i.e., clinicians and staff) play in the endorsement of a new intervention (i.e., MBC), and how that endorsement could potentially impact the implementation of MBC (i.e., actual usage with patients in a recommended way) and sustainability infrastructure, in the context of PRISM. It is critical that institutions recognize their MBC recipients’ perceptions, attitudes, and unique roles in MBC before the implementation process to increase the likelihood for sustainability success. In line with the PRISM framework, recognizing three or four domains (i.e., intervention, recipients, external environment, and/or implementation and sustainability infrastructure) with room for improvement will allow institutions to recognize which part of the implementation process needs changes or adjustments for successful implementation. Furthermore, the current study was conducted within an adult ambulatory psychiatry in a medical center, which faces different challenges compared to community clinics or psychology clinics that previous MBC implementation studies have focused on. Lastly, this study utilized mixed methods to grasp the whole picture of current MBC practices in this institution. Qualitative methods increased in-depth evaluation of participants’ perceptions, attitudes, and current status of their MBC implementation and utilization within the institution, and quantitative analysis pointed out the gap between the clinicians’ willingness towards MBC utilization versus their clinical practice in reality.
Limitations and future directions
The current study featured a small sample size due to a limited participant pool to recruit employees engaged in the MBC process for focus groups, although the BPM method allowed for rapid idea generation, even with a smaller participant pool. Additionally, our findings are specific to a psychiatry and behavioral health clinic within a larger medical system which may not generalize to clinics where the clinicians and staff have no or minimal understanding of MBC, as the participants in this study were already engaging in MBC implementation after a brief, non standardized training. As such, the current study results may apply differently to the settings where the implementation level varies (e.g., prospective implementers; those who have been trained but not yet implemented; those who have implemented MBC as an EBP but are facing challenges). As qualitative coding still required a significant labor and time cost, it would be difficult to aggregate these codes rapidly for contemporaneous quality improvement projects. From a quantitative perspective, there was incomplete clinician and staff survey data. Additionally, the current study did not investigate patient perspectives on MBC, despite the patients being critical recipients of MBC. In future studies, investigators suggest collecting data from patients currently engaged in MBC to further enrich understanding of systematic strengths and faults. We also recommend an additional measurement be utilized to collect quantitative data reflective of staff attitudes.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.