Introduction
Pragmatic clinical trials (PCTs) have come to play an increasingly important role in mental health services research (Brierley et al.
2013; Aakhus et al.
2014; Priebe et al.
2015; Thomas et al.
2015; Wenborn et al.
2016). A PCT compares treatments under everyday clinical conditions and are designed to directly support real world policy decisions regarding the use of an intervention (Califf and Sugarman
2015; Loudon et al.
2015). The design is summarized by the pragmatic-explanatory continuum indicator summary (PRECIS) (Thorpe et al.
2009; Loudon et al.
2015). Pragmatic trial study participants are more representative than those recruited in traditional explanatory RCTs because eligibility criteria are often less strict. The PCT is also designed so data can be easily collected in clinical settings under conditions of everyday practice. Pragmatic trials can include questions from and important to multiple diverse stakeholders, including policy makers. PCTs therefore have the potential benefit of producing research that is actionable by being designed around application to practice, with an emphasis on successful implementation (Tunis et al.
2003; Glasgow et al.
2005). Other key pragmatic trial domains include intent-to-treat data analytic approaches, incorporation of research procedures that minimize the need for adjudication (i.e., reaching consensus), and the efficient rollout of PCTs in real-world health care systems. The National Institute of Health Common Fund has supported the NIH Healthcare System Research Collaboratory in an attempt to catalyze methods development in the design and rollout of pragmatic clinical trials (
https://www.nihcollaboratory.org/Pages/default.aspx).
A key aspect of pragmatic trials is an emphasis on efficiency in research processes so as to minimize costs per subject randomized (Zatzick et al.
2016). In part, this efficiency is achieved through the use of hybrid effectiveness-implementation designs in which assessments of program effectiveness and implementation outcomes may be evaluated simultaneously. There are three types of hybrid designs (Curran et al.
2012). Type I designs are primarily focused on evaluating the effectiveness of the intervention in a real-world setting, while assessing implementation is secondary; Type II designs give equal priority to an evaluation of intervention effectiveness and implementation; and Type III designs are primarily focused on the evaluation of an implementation strategy and, as a secondary priority, may evaluate intervention effectiveness, especially when intervention outcomes may be linked to implementation outcomes. Each design makes use of both quantitative and qualitative methods in mixed methods designs. In Hybrid I designs, for instance, quantitative methods are typically used to evaluate intervention or program effectiveness, while mixed methods are used to identify potential implementation barriers and facilitators. Mixed methods are used in Hybrid 2 designs to evaluate both process and outcomes of program effectiveness and implementation. In Hybrid 3 design, while quantitative methods are typically used to evaluate effectiveness, mixed methods are used to evaluate both process and outcomes of specific implementation strategies (Palinkas and Cooper
2018).
Although several pragmatic clinical trials have incorporated the use of qualitative methods in their designs (Brierley et al.
2013; Littlewood et al.
2015; Orgeta et al.
2015; Thomas et al.
2015; Wenborn et al.
2016), such methods are often both time and labor intensive (Green et al.
2015), and, thus, are inconsistent with the aims of PCTs. Moreover, the designs of these studies are not based on PRECIS standards (Loudon et al.
2015). In the absence of established guidelines for use of qualitative methods in PCTs (Glasgow and Riley
2013), our understanding of implementation processes in PCTs required for sustainable service delivery is limited (Palinkas and Cooper
2018; Zatzick et al.
2016).
Over the past decade, members of the investigative team have established a stakeholder partnership with the American College of Surgeons (ACS) Committee on Trauma (College) whereby the results of behavioral health pragmatic clinical trials can be translated into policy requirements and best practice guidelines for the regulation of US trauma care systems nationally (American College of Surgeons Committee on Trauma
2006,
2014; Love and Zatzick
2014; Terrell et al.
2008; Zatzick et al.
2016). The ACS oversees the development of national policy requirements and clinical best practice guidelines that inform the integrated operation of US trauma centers and affiliated trauma care systems (American College of Surgeons Committee on Trauma
2006,
2014). The ACS has successfully linked trauma center funding to verification site visits and other quality indicators (American College of Surgeons Committee on Trauma
2006,
2014).
In this paper, we introduce Rapid Assessment Procedure-Informed Clinical Ethnography (RAPICE), an approach to collection and utilization of qualitative data in pragmatic clinical trials of mental health services implementation that is informed by clinical ethnography and rapid assessment procedures. Additionally, an illustrative case study is presented that highlights how RAPICE can help understand the interplay between pragmatic trial behavioral health implementation processes and associated trauma center policy.
Clinical Ethnography
Ethnography refers to both a process and outcome of research that produces an interpretation and description of a particular group, organization, or system, including behaviors at all levels, customs, norms, roles, and methods of interaction (Bernard
1988; Green et al.
2015). It is typically carried out through participant observation and in-depth interviews, with the researcher immersing him/herself in the regular, daily activities of the people involved in the setting. In most cases, this is a long-term investment of time and energy, with regular observation occurring over weeks, months or years.
Participant observation is the “process of learning through exposure to or involvement in the day-to-day routine activities of participants in the research setting” (Schensul et al.
1999, p. 91). “It involves establishing rapport in a new community; learning to act so that people go about doing their business as usual when you show up; and removing yourself every day from cultural immersion so you can intellectualize what you’ve learned, put it into perspective, and write about it convincingly” (Bernard
1988, p. 148). Participant observation can take many forms, ranging from exclusive participation to exclusive observation (Geertz
1976; Rosaldo
1984; Luhrman
1989; Zatzick and Johnson
1997).
Clinical ethnography is defined as culturally- and clinically-informed self-reflective immersion in local worlds of suffering, healing, and wellbeing to produce data that is of clinical as well as anthropological value. This approach seeks to combine and balance the anthropological method of participant-observation with clinical evaluation of others and reflexive evaluation of self (Calabrese
2013). It is a method involving an immersive ethnographic study of illness and clinical practice, with the additional intent and result of improving clinical outcomes (Kleinman and Benson
2006; Lie et al.
2011). A key feature of clinical ethnography is the training of clinicians in ethnographic methods (Brody
1981; Dao et al.
2018).
There are numerous examples of the application of ethnographic methods in clinical settings, including mental health settings. Ware and colleagues (
1999), for example, used ethnographic methods of field observation and open-ended interviewing to investigate the meaning of continuity of care in mental health services. Observations were carried out at two community mental health centers and a psychiatric emergency evaluation unit in Boston. Sixteen recipients and 16 services providers at these sites were interviewed. In a follow-up study, the investigators used ethnographic data to develop (Ware, Dickey et al.
2003) and validate (Ware, Tugenberg et al.
2003) the trustworthiness of CONNECT, a measure of continuity of care. Willging and colleagues used ethnographic methods and a review of legal documents and state monitoring data to examine the impact of Medicaid reform on mental health services (Willging et al.
2005) and the transfer of all publically funded behavioral health services under the management of one private corporation in New Mexico (Willging et al.
2009).
In dissemination and implementation (D&I) research, participant observation and ethnography are commonly paired with in-depth interviews and quantitative data collection to produce a more comprehensive understanding of the ways in which a project was implemented, describe the extent of fidelity to the intervention, and identify and understand barriers and facilitators of implementation (Torrey et al.
2012). Researchers often use key informant interviews, in-depth interviews as well as group interviews, to create a detailed account of the implementation process and its context (Green et al.
2015). For instance, Brunette and colleagues (
2008) examined the extent to which an integrated dual disorders intervention was implemented by 11 community mental health centers participating in a large study of practice implementation. Trained implementation monitors conducted regular site visits over two years, interviewing key informants, conducting ethnographic observations of implementation efforts, and assessing fidelity to the practice model. These data were coded and used as a basis for detailed site reports summarizing implementation processes. Palinkas and colleagues (
2008) used ethnographic methods of participant observation and extended semi-structured interviews with trainers, clinical supervisors, and clinicians representing 11 agencies in Honolulu HI and Boston MA to understand predictors and process of treatment implementation in the Clinic Treatment Project (CTP), a randomized effectiveness trial of evidence-based treatments for depression, anxiety, and conduct problems in children.
Rapid Assessment Procedures
While clinical ethnography seems to be an approach that is well suited to mental health services research, it is usually applied in a manner similar to classic ethnographic techniques, which are used over a long period of time, usually by a single individual. One method for efficient collecting and analyzing qualitative data in a relatively shorter period of time is a technique developed by anthropologists known as Rapid Assessment Procedures or RAP. This approach is designed to provide depth to the understanding of the event and its community context that is critical to the development and implementation of more quantitative (cross- sectional or longitudinal) approaches involving the use of survey questionnaires and diagnostic instruments (Scrimshaw and Hurtado
1987). Distinguishing features of RAP include: (1) formation of a multidisciplinary research team including a member or members of the affected community; (2) development of materials to train community members; (3) use of several data collection methods (e.g., informal interviews, newspaper accounts, agency reports, statistics) to verify information through triangulation; (4) iterative data collection and analysis to facilitate continuous adjustment of the research question and methods to answer that question; and (5) rapid completion of the project, usually in 4–6 weeks (Beebe
1995; Harris et al.
1997). Essentially, the methodology is based on techniques of participant observation and non-directed interviewing.
Rapid assessment procedures have been used in evaluation studies of healthcare organization and delivery (Vindrola-Padros and Vindrola-Padros
2018). For instance, it has been used to assess the mental health service needs of older adults (Palinkas et al.
2007) and victims of school shootings (Palinkas et al.
2004); to provide information on values, beliefs and cultural perspectives necessary for designing effective and socially valid health education programs, such as the design of a HIV/AIDS prevention program in East Los Angeles based on how Latinas view virus transmission (Scrimshaw et al.
1991); and to develop materials relevant to health promotion and disease control, such as posters depicting Oral Rehydration Salts replenishing fluids lost from bouts of diarrhea and vomiting (Herman and Bentley
1992). It has been used to conduct formative program evaluations to make midcourse corrections to improve program effectiveness, such as UNICEF’s efforts to provide leaders with information on parts of a program that needed improvement (Pearson
1989); and to design summative program evaluations, such as the assessment of effects of primary health care programs on nutritional and health practices of families (Scrimshaw and Hurtado
1987). It has been used to generate hypotheses to be tested in large scale studies, such as the collection and analysis of focus group data to determine factors influencing decision of parents to have children immunized (Coreil et al.
1989); and to collect information on topics that are time-dependent, such as providing the UN with information needed to determine how to prioritize funding for Primary Health Care (PHC) programs (Scrimshaw and Hurtado
1987).
RAP has also been used in conducting evaluations of program implementation (Goepp et al.
2008; Schwitters et al.
2015; Wright et al.
2015). For instance, Ackerman and colleagues (
2017) used “rapid ethnography” to understand efforts to implement secure websites (patient portals) in “safety net” health care systems that provide services for low-income populations. Site visits at four California safety net health systems included interviews with clinicians and executives, informal focus groups with front-line staff, observations of patient portal sign-up procedures and clinic work, review of marketing materials and portal use data, and a brief survey.
Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE)
The procedures that constitute the RAPICE method are outlined below. The RAPICE method is initially presented with the potential for application to a broad range of pragmatic trial contexts. This description of RAPICE methods is then followed by a specific case example of the use of RAPICE in a pragmatics trial in the acute care medical setting.
In 2011, Zatzick and colleagues articulated the combination of RAP and clinical ethnographic techniques in developing and implementing a single site stepped collaborative care comparative effectiveness trial targeting PTSD and related comorbidities (Zatzick et al.
2011). The method combined intensive participant observation by the ethnographically informed clinician with periodic review of field observations with the clinically oriented social scientist. As with RAP, RAPICE applied in a pragmatic clinical trial is distinguished by the following: (1) formation of a multidisciplinary research team including a member or members with clinical and/or administrative expertise and ethnographic and mixed methods training, enabling efficiency in data collection and analysis through division of labor; (2) development of materials to train team members in ethnographic methods and rapid assessment procedures that minimize the burden placed on any single study participant; (3) use of several data collection methods (e.g., participant observation, informal and semi-structured interviews, field jottings and logs, quantitative surveys) to verify information through triangulation; (4) iterative data collection and analysis in real time to facilitate continuous adjustment of the research question and methods to answer that question; and (5) rapid completion of the mixed method component of the project, which may vary depending on project aims and mixed method design.
When combined with quantitative methods in mixed method designs, RAPICE can be used to collect and analyze data to address important research questions that govern the principles and practice of implementation science. These include the following: (1) what factors act as barriers and facilitators to implementing a specific evidence-based policy, program or practice in a specific setting or context; (2) do these barriers and facilitators exist in other settings or contexts; (3) what are the characteristic features of the process of successful implementation; (4) what strategies are associated with successful implementation of specific policies, programs and practices; (5) how do we determine whether or not these strategies are successful; and (6) why are these strategies effective in producing successful implementation outcomes?
Procedures for the collection and analysis of ethnographic data on implementation context, process, barriers, facilitators and outcomes are outlined as a series of steps in Table
1. These steps can be further condensed into three stages. The first stage involves field observations by the clinical team of participant observers and a periodic review of that data with the mixed methods consultant. This stage usually begins with preliminary telephone discussions with potential sites, followed by training calls and training site visits. The training site visits are often conducted by the study Principal Investigator who, along with other members of the research team, acts in the capacity of a participant observer (PO). During the visit, the PO participates in meetings with site staff, collects whatever documents are available that record procedures implemented, and completes field notes and jottings that are audio recorded and later transcribed by research study staff in order to log site visit and trial activities. Informal and semi-structured interviews with site staff participants can occur before, during, or after the training site visits. The second stage involves pragmatic data analysis that includes discussion and coding of the qualitative data by team members. During this stage, a formative evaluation of the intervention may be conducted to assess the need to adapt the intervention for the specific setting and population and identify requirements for specific adaptations. The third stage involves translation of the data analysis to inform suitability of the intervention for a particular population or service delivery context, new intervention designs and methods for testing in subsequent clinical trials, and evaluation of the factors contributing to the outcomes of the current implementation trial.
Table 1
Steps to RAP informed clinical ethnography
Stage 1 |
1. Participant observer(s) (PO) conduct(s) informal interviews with and/or site visits to participating pragmatic trial organizations |
2. PO talks with a spectrum of study participants and visits staff meetings, observes clinical procedures, and/or conducts informal and semi-structured interviews with participants |
3. PO writes field jottings of observations and informal interviews, maintains logs of data collection activities and schedules, and digitally records semi-structured interviews for transcription |
4. PO presents field notes, logs, interview transcriptions, and any supporting material collected during the site visits to the Mixed Methods Consultant (MMC) |
5. MMC reviews the presented data and queries the PO to gain more insight into the data and its context. Suggestions for additional data collection may be provided by the MMC based on assessment of quality and lack of evidence of theoretical saturation |
Stage 2 |
6. Depending on the pragmatic trial context (e.g., trial resources devoted to mixed methods analyses) qualitative data is subjected to immersion/crystalization, and/or focused thematic content analysis, and template analysis |
7. PO provides a preliminary interpretation of the meaning and significance of the data, which is organized as set of a priori and emergent themes and description of their interrelationships |
8. MMC then provides a preliminary interpretation of the meaning and significance of the data organized in the same fashion and attempts to triangulate findings from multiple data sources (observations, interviews, documents) |
9. PO and MMC then identify points of convergence and divergence in the two interpretations |
10. A discussion between PO and MMC then ensues until consensus is reached regarding the meaning and significance of the data |
11. Depending on pragmatic trial context (e.g., trial resources devoted to mixed methods analyses), if consensus is not achieved, PO conducts follow-up interviews and/or returns to the field sites to collect additional data that could be used to resolve interpretation divergence |
12. Depending on pragmatic trial context (e.g., trial resources devoted to mixed methods analyses), if consensus is achieved, MMC makes recommendation for identification of disconfirming cases. PO returns to field sites to collect information that could be used to construct such cases |
13. Depending on pragmatic trial context (e.g., trial resources devoted to mixed methods analyses), interpretation of study findings is presented to study participants to confirm validity and comprehensiveness |
Stage 3 |
14. Analysis of qualitative data collected using RAPICE is then integrated with quantitative data to provide comprehensive understanding of implementation process and outcomes |
15. The understandings gained from methods integration is then applied to improve implementation outcomes (formative evaluation) or to address the key study questions related to implementation (summative evaluation) |
The approach to documentation of site visit and trial activities simultaneously aims to satisfy the pragmatic trial requirements for the minimization of time and resource intensive research methods that require extensive adjudication (Thorpe et al.
2009; Loudon et al.
2015) and the implementation science goal of understanding and documentation of trial processes that could yield sustainable maintenance of screening and intervention procedures (Curran et al.
2012; Zatzick et al.
2016). In the case study described below, logs and field notes were subsequently reviewed on an approximately monthly basis with the investigation’s mixed methods consultant (MMC); however, such reviews can be conducted more frequently (e.g., weekly) depending on project aims and requirements. The MMC reviews the data and then queries the PO to gain more insight into the data and its context. These queries then provide a framework for a preliminary interpretation of the meaning and significance of data by the PO, expressed in terms of a set of a priori and emergent themes and their interrelationship. The MMC then provides his/her own interpretation of the meaning and significance of the data using the same format of a priori and emergent themes and interrelationships. A discussion ensues until both the PO and the MMC reach consensus as to the meaning and significance of the data. RAPICE data collection may be conducted by masters-level investigators with appropriate training in conducting ethnographic fieldwork and semi-structured interviews. Review of collected data and preliminary analyses with a doctoral level researcher with expertise in mixed methods is recommended.
Data analysis in RAPICE can incorporate different styles of analysis that vary markedly in resource and adjudication intensity. The immersion/crystallization style can be used exclusively in scenarios where pragmatic trial resources devoted to qualitative analyses are minimal. The immersion/crystallization style is used to provide interpretations of the data by both the PO and MMC, and “consists of a prolonged immersion into and experience with the text and then emerging, after concerned reflection, with an intuitive crystallization of the text” (Miller and Crabtree
1992, p. 19). In this analytic approach, the investigators prepare short descriptive statements or “memos” to document initial impressions of topics and themes and their relationships (Miles and Huberman
1994).
The more time intensive editing style utilizes a methodology of focused thematic analysis (Saldana
2016) that would be appropriate to better-resourced mixed method pragmatic trials. For focused thematic analysis, field notes documents and interview transcripts are independently coded by the PO and MMC to condense the data into analyzable units. Segments of text ranging from a phrase to several paragraphs are assigned codes based on a priori (e.g., from a semi-structured interview guide) or emergent themes (also known as open coding, Strauss and Corbin
1998). Following the open coding, codes are assigned to describe connections between categories and between categories and subcategories (also known as axial coding, Strauss and Corbin
1998). Lists of codes developed by each PO are matched and integrated into a single codebook, constructed through a consensus of team members and consisting of a list of themes, issues, accounts of behaviors, and opinions that relate to organizational and system characteristics that influence program or practice implementation. Based on these codes, computer programs like QSR NVivo or Atlas.ti may be used for well-resourced pragmatic studies to generate a series of categories arranged in a treelike structure connecting text segments grouped into separate categories of codes or “nodes.” These nodes and trees are used to further the process of axial or pattern coding to examine the association between different a priori and emergent categories. They are also used in selective coding of material to identify the existence of new, previously unrecognized categories, and specific examples of co-occurrence illustrated with transcript texts. Through the process of constantly comparing these categories with each other, the different categories are condensed into broad themes using a format that places implementation within the framework of the organizational and system characteristics.
As with thematic analysis, template analytic techniques can be introduced into better- resourced pragmatic trial designs. “Template analytic techniques all share the use of a template or analysis guide, which is applied to the text being analyzed” (Miller and Crabtree
1992, p. 19). The template can be used to generate themes, patterns and interrelationships and is based on theory, research tradition, pre-existing knowledge, and/or a summary reading of the data that occurs when using the immersion/crystallization analytical style. Templates can be used to identify the presence or absence of a priori barriers and facilitators to implementation using constructs from established frameworks like the CFIR, to make comparisons of the presence or absence or characteristics of themes elicited from data collected at sites or from different categories of participants (e.g., clinicians versus patients or administrators), to document changes in occurrence of themes over time, or to integrate quantitative and qualitative data.
Because RAPICE is conducted in clinical settings, reporting requirements and assessment of quality should be consistent with standards that have been proposed for health services that emphasize rigor (Cohen and Crabtree
2008; O’Brien et al.
2014). One of the best illustrations of this approach is the Consolidated Criteria for Reporting Qualitative Studies (COREQ; Tong et al.
2007), a 32 item instrument that assesses quality in three domains: research team and reflexivity, study design, and analysis and findings. However, because of the role of participant observation and reliance on the immersion/crystallization style of data analysis, it may also be appropriate to adopt the standards proposed by Charmaz (
2014) that is consistent with constructivist grounded theory and lists 19 criteria grouped under the categories of credibility, originality, resonance, and usefulness.
Results
An example of the data collected from semi-structured interviews conducted during the site visits is provided in Supplemental Data Table
1. The interview transcript was obtained from a staff member at one of the sites participating in the DO-SBIS study. Analysis of this transcript by the MMC suggested two possible overarching themes: use of non-evidence based practice alcohol interventions and sustainment of non-evidence based practice alcohol interventions. The first theme addressed the RE-AIM Adoption outcome and included procedures for conducting an assessment of current alcohol use and a psychoeducational intervention. The assessment included questions regarding how alcohol is used, changes in drinking behavior, reasons for use the day of injury, and opinions about safety and effects of drinking. The psychoeducational intervention included review of drinking hazards, evaluation of pros and cons of drinking, plans for seeking help, establishment of goals, and sources of support. The second theme addressed the RE-AIM Maintenance outcome and focused on sustainment of the alcohol assessment and psychoeducational intervention and included plans for staffing and identification of barriers to sustainment (i.e., limited time and interest). Initial analysis by the PO revealed some slightly different themes, as well as some convergent coding. Initially, themes related to training (Adoption) and implementation of alcohol screening and brief intervention were discussed. Next, the content of the intervention delivered during the trial was explored. Finally, the impact of the study termination and staff turnover was discussed with regard to the maintenance of high quality alcohol screening and intervention going forward. The semi-structured interview revealed that the prior pragmatic trial had built upon existing resources to provide an alcohol screening and brief intervention program that adequately addressed ACS screening and brief intervention verification criteria. Additional observations regarding the trauma center’s alcohol screening and brief intervention implementation were made by the PO in the years after DO-SBIS during the conduct of TSOS. Field observations revealed that site staff turnover had continued and that additional training would be needed in order for the site to regain the higher quality alcohol screening and brief intervention that had been delivered during DO-SBIS.
Alcohol Screening and Brief Intervention Field Jotting
Three themes derived from consensus coding procedures were observed in a field jotting related to a TSOS trauma center site visit (Supplemental Data Table 2). The first theme involved interactions with trauma center staff present at the site visit; these discussions revolved around trauma team roles in recruitment and intervention delivery. A second set of themes related to team discussions of injured patient cases enrolled. These elements of the field jotting described patient and family characteristics. The final set of themes related to the site’s approach to maintenance of College regulatory requirements targeting alcohol screening and brief intervention. While the provider conducting the alcohol screening and brief intervention performed a mandatory consult, evidence-based motivational interviewing brief intervention approaches were not implemented.
Table 2
American College of Surgeons policy and implementation process summary matrix
Reach | Implied that screening should be available for all patients; unclear whether verification site visits check on this or if this data is available within trauma registries | Not addressed | Not addressed | Not addressed |
Effectiveness | Pragmatic trial literature including DO-SBIS study supports | Pragmatic trial literature supports | Some initial clinical trials, but insufficient for requirement or practice guideline not addressed | Insufficient literature Not addressed |
Adoption Likely Adopter Groups | Early, Middle & Late Adopters required | Early Adopters likely to uptake | Early Adopters could explore uptake | No adoption likely |
Adoption Centers familiar with topic (e.g., alcohol, PTSD) | Mandate enforces familiarity with issue | Guideline suggests familiarity with issue | Informational section introduces/disseminates patient-centered care | Familiarity optional |
Implementation Centers required to implement procedure | Mandate requires; verification site visit confirms | Guideline only suggests appropriate practice, no implementation required | Informational section only introduces idea; no implementation required | No requirements |
Implementation & Maintenance Adequate staffing and resources allocated | Not addressed; mandate does allow on-site providers to lobby institution for additional staffing and resources | Not addressed; Practice guideline does allow on-site providers to lobby institution for additional staffing and resources | Not addressed; Informational section does allow on-site providers to lobby institution for additional staffing and resources | No requirements |
Implementation & Maintenance High quality procedures implemented and maintained | Not addressed unless specifically required by mandate; on-site providers may be aware of issues related to the variable quality of screening and intervention procedures delivered | Not addressed; on-site providers may be aware of issues related to the variable quality of screening and intervention procedures delivered | Not addressed; on-site providers may be aware of issues related to quality | No requirements |
During piloting of the TSOS trial, providers were asked to reflect on the progression of ACS behavioral health policy beginning with alcohol and extending to PTSD. Consensus coding identified multiple themes in this interview segment. The first theme pertained to familiarity with the ACS guideline prior to the pragmatic trial. In this instance, the study participant was not aware of the screening protocol recommended by the guideline: “actually, the first time I heard about it was when you contacted us and started talking about PTSD, and then I did go to the orange book and look at what the college [ACS], what their recommendation was.” The second theme pertained to prior experience with PTSD screening. At this site, staff had conducted PTSD screening, albeit with a protocol that was not outlined in the ACS guideline. According to the provider, “we have been doing a screening for PTSD in our trauma follow-up clinic for many years. We had a questionnaire that the patients would score and then those that scored at a certain level, we would try to get into some type of treatment.” Finally, a third theme pertained to current practice in screening for PTSD. When asked if the PTSD screening was still being used, the provider commented on the inability to screen all patients for PTSD (Reach):
It still would be happening, unfortunately we don’t administer it to every patient, which I think from this study that we’ve done, the incidence of PTSD is so much greater than we probably expected. And so, when the surgeons or the mid-levels see a patient and they feel like perhaps PTSD is a problem for them, then they would ask for the screening to be done. And I think, going forward, it should be done on everybody.
Logging of ACS Policy Summit Activity
Supplementary Data Table 3 presents a distillation of information logged as part of an ACS Committee of Trauma Policy Summit orchestrated by the study team in September of 2016. The distillation provided a general overview of summit activities while also articulating specific objectives and deliverables. Analysis of the log identified three categories of ACS policy activity, requirement, guideline level recommendation, and suggested mention but no current verification specifications. The log also listed disorders for which ACS requirements (Alcohol), guideline level recommendations (PTSD), and suggested mention but no current specifications (patient-centered care) exist. In summary, while alcohol screening and intervention had been required by the ACS, and PTSD screening, intervention and referral recommended, verbiage about patient-centered care has been recommended for discussion in the resources guide without a recommended verification plan or action.
National Trauma Center Survey Data on Alcohol Screening and Brief Intervention Uptake
As evidence of the Reach of the intervention, the two trauma center surveys attained 73% (148/204) and 78% (172/221) national level I trauma center response rates respectively. The surveys also documented that prior to the alcohol screening and brief intervention requirement 72.4% of level I trauma centers routinely screened patients for alcohol, and that after the requirement 91.9% of trauma centers screened patients for alcohol (χ(1) = 21.1, P < 0.001). Similarly, before the requirement, 40.7% of centers reported delivering evidence-informed interventions; after the requirement 64.0% of centers delivered an evidence-informed intervention (χ(1) = 17.1, P < 0.001). The second survey also asked questions about routine PTSD screening and intervention; less that 10% of sites reported routine PTSD screening and intervention when the survey was conducted (i.e., 2011–2012).
ACS Policy and Site Implementation Process Matrix
Table
2 presents the RAPICE-informed integration of pragmatic trial implementation process data as characterized by the RE-AIM framework with categories of ACS policy activity. The matrix derives from a number of data sources including field notes and jottings, semi-structured interviews, national trauma surveys and iterative multidisciplinary team discussions. National trauma center survey data documented the enhanced reach of behavioral health screening and intervention procedures over time. Effectiveness data derived from study team and other pragmatic trials informed ACS policy (American College of Surgeons Committee on Trauma
2006,
2014).
Themes derived from study team discussions were how different types of ACS regulatory policy (i.e., alcohol requirements, PTSD practice guidelines, patient-centered informational additions to the ACS Resource Guide) influenced site-specific adoption, implementation and maintenance processes. With regard to adoption, the study team observed how the alcohol policy mandate required that all trauma centers implement alcohol screening and intervention procedures; trauma centers were required to demonstrate alcohol screening and intervention procedure implementation during ACS verification site visits that occurred every three years. In contrast, clinical practice guidelines and other Resource Guide informational suggestions were only likely to be rolled out by early adopter centers. Over the course of the DO-SBIS and TSOS trials, the study team made longitudinal field observations regarding the extent to which high quality behavioral health screening and intervention procedures were being implemented in the wake of policy requirement or clinical best practice guideline recommendations. Multiple factors, including provider team compositions, trauma center administrative support, staff turnover, and other trauma center resource availability, were observed to influence the longer-term implementation and maintenance of behavioral health screening and intervention procedures. RAPICE-informed study team reviews suggested that while a regulatory mandate successfully required site adoption of alcohol screening and brief intervention, the requirement did not ensure high quality procedures would be implemented and maintained with adequate staffing and resources.
Discussion
This paper provides an example of the RAPICE-informed integration of implementation science and pragmatic trial approaches as applied to U.S. trauma center behavioral health policy. Over the past decade, members of the investigative team spent hundreds of hours immersed in the acute care medical trauma center clinical context as clinician-investigator participant observers. The study team collected multiple types of data over this period, including field notes and jottings during trauma center site visits and logs of national trauma center policy summit activities. Informal and semi-structured interviews were conducted with trauma center providers involved in pragmatic trial behavioral health screening and intervention procedure rollouts. The study team also conducted two national surveys of trauma center behavioral health screening and intervention uptake over this period. RAPICE allowed for an ideal mixed method for the integration of these multiple data sources and was feasibly embedded within the study team nationwide pragmatic trial rollouts.
As noted earlier, qualitative methods have become a standard feature of implementation research through their use in mixed methods and hybrid effectiveness-implementation designs (Palinkas
2014; Palinkas et al.
2011; Green et al.
2015). However, the use of qualitative methods in PCTs possess a number of challenges, not the least of which is minimizing the cost per subject randomized without sacrificing methodological rigor. There have been several attempts to establish standardized criteria for rigor in qualitative methods. These criteria often have counterparts in criteria used for quantitative methods; hence internal validity is framed in terms of “credibility,” external validity is framed as “transferability,” reliability is framed as “auditability,” and objectivity is framed as “confirmability” (Lincoln and Guba
1985). Although the use of such criteria and the specific elements listed has been a subject of debate among methodologists (Patton
2002; Padgett
2017), they are supported by several strategies designed to enhance rigor and trustworthiness of qualitative methods. These strategies include prolonged engagement, triangulation, peer debriefing and support, member checking, negative case analysis, and auditing (Patton
2002; Padgett
2017; Levitt et al.
2018).
In the DO-SBIS and TSOS, four of these strategies (triangulation, peer debriefing and support, and member checking, and prolonged engagement) were embedded within the RAPICE methodology. Triangulation involved the use of multiple theories or perspectives to interpret the data (theory triangulation), use of participant observation and semi-structured interviews to study implementation process and barriers (methodological triangulation), use of more than one observer during field visits to achieve inter-subjective agreement (observer triangulation), and use of interviews, observational field jottings, and national trauma center survey materials) (data triangulation) (Patton
2002). Further, comparisons of qualitative and quantitative data on similar topics such as organizational culture and climate were conducted to identify points of convergence in a mixed methods design. For instance, as was found in a previous study (Zatzick et al.
2014), there was a lack of congruence between measures of organizational culture and climate (Glisson et al.
2008) that reflected relative stability in trauma centers and ethnographic observations that suggested otherwise. Peer debriefing was conducted between team POs and the MMC. Member checking was conducted through soliciting feedback on preliminary results from participants at an ACS Policy Summit. Finally, the principal investigator has spent hundreds of hours over the past two decades immersed in the acute care medical trauma center clinical context. “The clinician investigator’s in-depth immersive participation in conjunction with collaborative team activity and expert consultant review exemplifies the criteria of prolonged engagement and persistent observation that constitute crucial elements of quality and verification in qualitative research” (Zatzick et al.
2011, p. 125). With regard to prolonged engagement, RAPICE enables time efficient field observation and review procedures that constitute ideal “nimble” mixed method approaches for the pragmatic trial. It also allows for real-time workflow integration.
In addition to the four strategies used in the TSOS study, RAPICE can easily incorporate other strategies for enhancing rigor and trustworthiness such as negative case analysis and auditing. However, as a front-line provider immersed in the clinical setting, the PO’s presentations to the MMC are vulnerable to internal data reduction and selective recall biases. Compared to traditional methods of ethnographic research and semi-structured interviews, RAPICE may be associated with less detailed field notes, fewer observations of important events or implementation processes, fewer and shorter semi-structured interviews with study participants, and less depth in coding, analysis and interpretation of qualitative data collected. These limitations, in turn, may adversely impact the utility of qualitative data in mixed methods designs to achieve the aims of convergence, complementarity, expansion, development, and sampling (Palinkas et al.
2011).
On the other hand, use of RAPICE allows for more opportunities to conduct “repeated measures” of qualitative data through multiple site visits. This represents a different form of prolonged engagement in that the observations may be of shorter duration, but over a longer period of time than is the case with ethnographic methods used in other mental health services research (Ware et al.
1999; Willging et al.
2009). RAPICE also offers richer ethnographic contextual material when compared to implementation research that relies on semi- structural interviews or focus groups because it is able to place the data collected in context in real time. When combined with longitudinal study team field observations in the TSOS Study, RAPICE allowed for an in-depth appreciation of contextual factors that were observed to influence the rollout and sustainability of behavioral health screening and intervention in the wake of ACS policy.
Perhaps most importantly, RAPICE directly addressed pragmatic trial efficiency standards. Rather than requiring time and resource intensive in-depth qualitative research procedures, RAPICE facilitated efficient mixed method data collection and analyses that are in harmony with the pragmatic trial aim of reducing costs per subject randomized through combined input and iterative discussion from both PO and MMC team member perspectives, structuring of roles of research team members to capitalize on their expertise. RAPICE also allows an increasingly more research intensive analytic approach that can potentially incorporate immersion/crystallization, editing/thematic content, and template analysis methods in phases.
The findings of the case study also highlight the importance of understanding the interplay between national behavioral health policy and implementation processes as described by frameworks such as RE-AIM. National trauma center surveys conducted over the past decade documented the enhanced reach of alcohol screening and brief intervention procedures in the wake of ACS policy requirements. Combined with longitudinal field observations, semi-structured interview data highlighted site struggles with staff turnover and the maintenance of high quality screening and intervention procedures over time. Analysis of field jotting data suggested that although alcohol screening and intervention procedures might satisfy ACS verification site visit criteria and thus require adoption by all trauma center sites, the procedures did not adhere to principles derived from the evidence-based practice literature (Zatzick et al.
2014).
Another strength of the RAPICE method is the iterative generation of novel data about implementation processes. The summary matrix derived from field observations, semi-structured interview coding, national survey data and, most importantly, multidisciplinary team discussions, yields important insight into the limitations of implementation strategies that rely on policy requirements alone (Fixsen et al.
2005). While clinical practice guidelines familiarize providers with mental health disorders, and policy mandates make implementation a requirement, the delivery of high quality services and maintenance of adequate staffing and resources are by no means guaranteed, unless further incorporated into ACS policy.
The RAPICE method also may reveal ways in which emerging implementation science constructs may not fully capture novel health service delivery system contextual factors. From an implementation science perspective, acute care medical trauma center inpatient settings are distinct from other primary and acute care settings in that the ACS has the ability to mandate screening and intervention procedures. Thus, the integration of behavioral health screening and intervention services at United States trauma centers is occurring in a unique regulatory “make it happen” context, in contrast to a negotiated “help it happen” implementation context in other general medical settings (Greenhalgh et al.
2004). The ACS has demonstrated its commitment to using empirical data to further the integration of substance use screening and brief intervention at trauma centers (American College of Surgeons Committee on Trauma
2006,
2014).
The description of effectiveness-implementation hybrid trials has not routinely included the targeting of health care policy change as a potential implementation mechanism (Curran et al.
2012). A review of the literature revealed few discussions of the use of mixed methods to better understand the targeted use of health policy as an implementation strategy in effectiveness-implementation hybrid designs (Purtle et al.
2018). The DO-SBIS and TSOS trials may therefore defy categorization as hybrid studies until the novel “make it happen” trauma care systems health service delivery context is better articulated through methods such as RAPICE. The two trials described in the current paper may constitute effectiveness-implementation hybrid designs that use a novel, yet time-tested, ACS policy mechanism as a targeted implementation strategy. The TSOS trial simultaneously aims to determine the effectiveness of the stepped collaborative care intervention model in reducing PTSD symptoms and comorbid conditions, while also assessing the potential utility of the implementation strategy that uses ACS policy to target regulatory mandates for trauma care systems nationally.
RAPICE also provides qualitative data and analysis comparable to that utilized in mixed methods designs in implementation research. In addition to the use of qualitative data for the purpose of convergence described earlier, data collected and analyzed through RAPICE procedures can be used to complement the functions of quantitative methods or expand and explain the findings acquired through the use of quantitative methods (Palinkas
2014; Palinkas et al.
2011).
Finally, RAPICE offers greater transparency in the integration of investigator and study participant perspectives on the phenomena of interest. Unlike Rapid Assessment Procedures as they have been used in public health, the use of RAPICE in this case study did not involve the participation or the insight of community members, or in this case, clinicians participating in the pragmatic trial. Consistent with the aims of the pragmatic trial, input from clinicians at the field sites preceded the implementation of the TSOS. However, in assessing the process of implementation and identification of potential barriers and facilitators, the insider perspective on implementation normally provided by community stakeholders in RAP studies was provided largely by the PO and other study team members (e.g., trauma surgical champions) who simultaneously occupy the dual roles of clinician-investigators and front-line trauma center providers. The outsider perspective was provided largely by the MMC. Consistent with the RAP methodology (Beebe
1995), the two perspectives were integrated in the final analysis through their exchange between the PO and the MMC during the peer-debriefing stage.
Conclusions
When applied to evidence-based practice implementation in pragmatic trials, RAP- informed clinical ethnographic methods combine efficient documentation of implementation context, processes and outcomes with periodic review by an expert mixed-method consultant. The methods straddle the continuum between time intensive qualitative procedures and cursory study team review of field observations. Although we have focused here on the potential use of RAPICE in pragmatic clinical trials to examine implementation barriers, facilitators, processes and strategies within specific settings and contexts, we should note that it could be an equally useful in other forms of health research including needs assessments and efficacy and effectiveness clinical trials. Future comparative mixed method investigations could contrast themes derived from RAPICE with themes derived from more in-depth qualitative procedures. In this way, the value added to the clinical trial and implementation processes evaluations by more in-depth qualitative research procedures could be more directly assessed.
In this applied example, the RAPICE approach simultaneously satisfied the pragmatic trial requisite for minimization of time intensive research methods that require extensive adjudication, and the implementation science goal of understanding and documentation of trial processes that could yield sustainable maintenance of screening and intervention procedures through ACS policy. The RAPICE approach furthers the goal of integrating pragmatic trial and implementation science methods by embedding a multidisciplinary implementation team within a health care system pragmatic trial rollout.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.