Rationale and design of the Multicenter Medication Reconciliation Quality Improvement Study (MARQUIS)

The MARQUIS conceptual framework is based on Brown and Lilford’s model for evaluating patient safety interventions, which is an adaptation of Donabedian’s “structure—process—outcome” model [17‐21]. The model distinguishes interventions that focus on management processes (e.g., provider training) from those that focus on clinical processes (e.g., tools supporting medication list comparisons across care transitions). MARQUIS involves both types of interventions, focusing primarily on the latter. To better understand why interventions succeed or fail, we assess contextual factors (i.e., micro- and macro-organizational structure and existing management processes) and intervening variables (e.g., team climate, safety culture, and knowledge of medication safety principles, Figure 1). Also important to understanding whether interventions succeed or fail is intervention fidelity, or the faithfulness with which the intervention is performed, which can be influenced by how usable the tools are and the degree of training and support given to front-line clinicians (i.e., items at the level of intervention in Figure 1). These are measured as an “intervention score” updated monthly to assess what toolkit components have been adopted and how the state of medication reconciliation changes over time (Table 1). Along with the MARQUIS intervention itself, each of the contextual factors may affect the patient-level outcomes being assessed, such as unintentional medication discrepancies, patient satisfaction, and healthcare utilization (i.e., items to the right of the intervention, Figure 1).

The MARQUIS toolkit, described elsewhere, [22] synthesizes best practices in medication reconciliation and provides aids to facilitate their implementation. The toolkit components were informed by a systematic review of medication reconciliation interventions, [12] the AHRQ-funded conference of stakeholders,[16] and the work of the MARQUIS investigators and advisory board.

Each toolkit component is framed as a standardized functional goal (e.g., “Improve access to preadmission medication sources”). This approach is ideal for complex QI interventions, [23] allowing sites to: 1) integrate intervention components with their baseline medication reconciliation efforts, information system capabilities, and organizational structures; and 2) add, customize, and iteratively refine the toolkit components and their implementation over time. This approach also improves generalizability, allowing other organizations to apply the lessons learned regardless of their culture or unique circumstances.

While recognizing the importance of flexibility, it was nevertheless important to have some common elements across sites. Thus, each site prioritized the implementation of certain toolkit components based on their potential for improvement and effort required. These included provider education on medication history taking, patient education and teach-back at discharge, patient risk stratification, and more intensive medication reconciliation efforts in high-risk patients.

Six U.S. sites are participating in this study: 3 academic medical centers, 2 community hospitals, and 1 Veterans Affairs hospital. We purposely chose sites that vary in size, academic affiliation, geographic location, and use of health information technology (Table 2). However, all sites had several common features: 1) medication reconciliation was a priority; 2) hospital leadership was committed to making further improvements in the process; 3) an active hospitalist group was engaged in QI; 4) a suitable hospitalist and/or pharmacist clinical champion at each site; and 5) each site planned to use primarily its own resources to pursue this effort.

Patient subjects are drawn from the medical and surgical inpatient, non-critical care units of each site, and are included if hospitalized long enough for a “gold-standard” medication history to be obtained by a study pharmacist (i.e., generally more than 24 hours). Institutional Review Board (IRB) approval was obtained from the Partners Healthcare System. In addition, each study site’s IRB reviewed the study: four considered it an exempt QI project, while two sites required informed consent of patients prior to participation. Informed consent has been incorporated into the data collection process at these sites.

MARQUIS utilizes SHM’s mentored implementation approach [24], providing each site with a hospitalist mentor to facilitate toolkit implementation. Each mentor has QI expertise and performs distance mentoring through monthly calls with the study site’s mentee/clinical champion, based upon the MARQUIS Implementation Guide which explains how to use the toolkit [25]. Each study site also receives two visits from the mentor, important from a QI standpoint (e.g., to maintain institutional support and enthusiasm among the local QI team, and better understand local practices) and from a research standpoint (e.g., to assess intervention fidelity and other barriers and facilitators of implementation). Additionally, SHM provides sites with an assigned lead project manager and research assistants located at SHM headquarters to assist with monitoring progress and collecting and analyzing data. Through the mentored implementation infrastructure, MARQUIS is affordable, adaptable, generalizable, scalable, and feasible for wide dissemination.

At each study site, a local QI team, led by the mentee/clinical champion conducts monthly meetings to oversee intervention implementation and data collection, as well as to address protocol questions and determine the effectiveness of the interventions. Sites can access a central website with additional resources and a listserv. The monthly conference calls with their mentor and ad lib email communications promote a consistent approach across sites [16].

Study outcomes are assessed from 6 months pre-intervention through 21 months post-intervention (Table 3). The primary outcome is the number of potentially harmful unintentional medication discrepancies per patient, determined by a trained on-site pharmacist taking a “gold standard” medication history on a random sample of patients (20–25 per month). This history is then compared to the primary team’s medication history and to admission and discharge orders. For discrepancies in admission or discharge orders not caused by history errors, the pharmacist reviews the medical record for a clinical explanation, and if necessary, talks with the medical team. This allows sites to distinguish unintentional medication discrepancies (i.e., due to reconciliation errors) from intentional medication changes. Physician adjudicators, blinded to the status of intervention implementation, record and categorize unintentional medication discrepancies with respect to: 1) timing (admission vs. discharge); 2) type (omission, additional medication, change in dose, route, frequency, or formulation, or other); 3) reason (history vs. reconciliation error); 4) potential for harm; and, 5) potential severity.

Secondary outcomes include accuracy and timeliness of the medical team’s documented admission medication history, absence of discharge reconciliation errors, unplanned healthcare utilization (i.e., readmission or emergency department use), and patient satisfaction. Unplanned healthcare utilization to the same site within 30 days of discharge is determined from hospital records on all eligible patients. We examine patient satisfaction scores on the global satisfaction and the medication specific dimensions from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, and will be aggregated by service, unit, and time period as data are available.

Contextual factors are measured using surveys of providers directly involved in the medication reconciliation process. Additionally, we measure intervention fidelity using direct observation during site visits, and evaluate training, support, and other steps offered to improve fidelity. Importantly, the extent of implementation is quantified as an intervention “score” for each toolkit component (Table 1) and factored into the analysis. The score is completed by the clinical champion at each site, informed by surveys administered to front-line clinicians who are directly involved in the medication reconciliation process when necessary.

In an effort to ensure consistency of on-site pharmacist data collection, the research team: 1) conducts monthly phone meetings with on-site pharmacists in which a patient case is reviewed for consistency and all discrepancies discussed; 2) provides on-site pharmacists with an updated ‘frequently asked questions’ (FAQ) document for managing new situations; and, 3) conducts site visits with the research team’s pharmacist to observe data collection processes and provide feedback, including how to improve process efficiency.

To ensure the consistency of the adjudication process, the principal investigator (PI) conducts a quarterly conference call with the sites’ physician adjudicators to discuss cases. In addition, the PI and a co-investigator review 6 cases from each site quarterly and review the results individually with each site’s adjudicators. A FAQ document for adjudicators is updated and redistributed as needed.

The study sites utilize a web-based data collection and reporting system built specifically for this study. The system creates HIPAA-compliant de-identified data sets for the coordinating data center and all investigators. The system allows for identification, classification, and adjudication of all discrepancies. Unintentional discrepancies identified by the on-site pharmacist are flagged in the system for physician adjudication. The data center provides detailed reports to trend discrepancies, facilitates uploads of patient-specific administrative data, tracks implementation of intervention components, and provides tools to support mentored implementation. It also provides tracking for patient enrollment compared to monthly targets.

The primary outcome will be analyzed using multivariable Poisson regression, including random effects and clustering of patients by site and treating physician. To account for temporal trends and the varied introduction of interventions by site, we will employ an interrupted time series analysis on all 3,600 patients across the 6 sites, evaluating outcomes monthly for 6 months pre-intervention and 21 months post-implementation [26]. The outcome is assessed as both a change from site-specific baseline temporal trends (i.e., change in slope) and sudden improvement with implementation of the intervention components (i.e., change in y-intercept). If each site has concurrent controls, these can be entered into the model to partially adjust for the effect of concurrent interventions. The model also allows for the detection of iterative refinement of the intervention (i.e., continuous improvement over time), as well as ceiling effects (i.e., lack of continued improvement beyond a certain threshold).

We will assess implementation of each component of the toolkit monthly, using the scoring system on a scale of no adoption to complete adoption for each component (Table 1). Because many of the sites have already implemented pieces of the interventions, their scores often do not start at zero, and scores increase with the implementation of interventions to a maximum score of 318. The scores for each component will be entered into the multivariable model as time-varying covariates, such that we can determine whether implementation of a particular component is correlated with improved outcomes thereafter. This allows us to make inferences about the most important components of the intervention.

For a stable estimate of temporal trends, each site’s data collection goal is approximately 22 patients per month, beginning 6 months pre-intervention through 21 months post-intervention. Due to our study design it is impossible to know a priori the nature of our post-intervention data, and therefore what our actual power would be to look at the effect of any specific intervention. However, based on prior research, we assumed that the number of medication discrepancies would follow a Poisson distribution and that, in the absence of an intervention, each hospitalized patient would have an average of 1.5 potentially harmful medication discrepancies in admission and discharge orders combined [27]. We also conservatively assumed that an intervention would be implemented at only 1 of 6 sites with 12, not 21, months of follow-up due to delays in planning and phasing in the intervention widely. This would yield data from 133 patients pre-intervention and 266 patients post-intervention. With these estimates and alpha = 0.05, we would have 90% power to detect a reduction in the mean number of medication discrepancies from 1.5 per patient to 1.1 per patient [27].

As sites began to implement the intervention, one methodological issue that arose was the extent to which sites should over-sample data from hospital areas receiving early versions of the intervention. We decided on a 3:1 ratio of intervention to control patients during the intervention period. This allows for concurrent controls during the spread of the intervention, while maintaining an adequate sample of intervention patients to evaluate effects on patients outcomes.

We will evaluate the influence of contextual factors, intervention fidelity, and intervening variables on implementation and outcomes using a mixed methods approach (Table 4). Measures of context are gathered using front-line staff and site surveys, direct observation, focus groups and interviews. At baseline, each site completed a site leader survey and front-line staff surveys to provide a semi-quantitative measurement of these issues. After the start of intervention implementation, a qualitative researcher conducts on-site focus groups with front-line staff and the QI team and interviews hospital leadership. At 12 months post-intervention, follow-up interviews are conducted by telephone. For focus groups and interviews, a convenience sample of staff is selected by role and department to ensure broad representation [28].

Intervention fidelity is assessed by direct, semi-structured observation of the site’s medication reconciliation process by their mentor during site visits at 3 and 12 months after implementation of the intervention. The observation protocol evaluates five steps of the medication reconciliation process: taking an admission medication history, identifying high risk patients to receive a high intensity medication reconciliation intervention, performing discharge medication reconciliation, performing discharge medication counseling using the teach back method, and forwarding the discharge medication list to the next provider of care after discharge. The mentor observes the actual process to identify if the intervention is being implemented as designed (content fidelity) and rates how well it was performed on a 1 to 4 scale (process fidelity) [33]. The observation forms also allow documentation of systems issues that impact the medication reconciliation process based on the Systems Engineering Initiative for Patient Safety (SEIPS) model’s [34] 5 domains: people, technology/tools, tasks, organization, and environment. Mentors received group training on how to assess fidelity, using a coding manual with standardized examples and training from a human factors expert. Mentors share feedback about the direct observations during the site visit with the site leader and QI team.

Intervening variables (i.e., in determining intervention fidelity) are assessed from surveys of front-line staff pre- and post-intervention. Topics include the quality of education and training received in the intervention (e.g., in taking medication histories), degree of input into intervention design, and adequacy of staffing and time to complete medication reconciliation processes. Other barriers and facilitators of intervention implementation are determined from structured and open-ended questions regarding the front-line staff’s opinions of the medication reconciliation process and its perceived impact on patient care.

Figure 2 outlines the timeline of the study. The study’s three critical time points are: intervention start, interim evaluation with iterative refinement of the intervention and a second draft of the implementation guide 9 months after the start, and end of the intervention after 21 months.