Study protocol: transforming outcomes for patients through medical home evaluation and redesign: a cluster randomized controlled trial to test high value elements for patient-centered medical homes versus quality improvement

We took several complementary approaches to explore different trial design options in the complex health reform environment for the TOPMED trial. First, we matched the trial design to the PRECIS pragmatic trial framework and reviewed trial literature. Second, we aligned trial implementation and components with major health reform elements. Third, a series of stakeholder interviews and focus groups were held to understand the values, understandings, perceptions, and intentions in regard to the health reform efforts and the potential for the trial design to be nested within these efforts. Finally, a pilot was completed to test various elements prior to the study start.

Changes to the trial design from this initial set of tasks are outlined in Table 1. The PRECIS framework recommends health professional and patient autonomy over trial protocol inflexibility and prescription, and the trial was broadened to allow flexibility in quality improvement (QI) decision-making, meeting practices where they are in terms of capacity for change. The alignment with payers and other stakeholders demonstrated that certain principles could be agreed upon, but flexibility was needed and simplification was required for specific improvement. Patients focused on access and information gaps; subsequently, we included access-related high-value elements, and all trial elements were translated to patient-specific language.

A 6-month pilot at a single practice site was performed to test the various interventions and refine efforts. At the pilot site, preparation and protocol elements were tested in a series of six monthly practice facilitation visits. At each visit, an aspect of the trial design was tested for feasibility, clarity, and usability. The pilot team was interviewed for the results, and results demonstrated that the expectations were too vague and time-consuming. A memorandum of agreement was drawn up with specific expectations for both the research team and the clinic team. These included a role-based specification by clinic of time expected, use of data, and communication required to participate, as well as clear roles for researchers. The protocol visits by practice facilitators were honed to focus on key drivers of quality improvement and transformation: leadership, team-based approach, QI infrastructure, setting new improvement targets, and follow-up on previous improvement targets. Reports were assessed for comprehension and revised, and use of the IT system for study participation was reduced, allowing clinics to use their own protocols and tools when they matched the function of the IT system.

Participants are eight primary care clinics (clusters) that are deeply engaged in health reform, are willing to use a population management registry tool, are willing to receive practice facilitation and improve their performance, and use an electronic health record (EHR) system at baseline. A goal was set to recruit a diverse set of participant clinics by ownership, location (rural versus urban), and size (from 3 to 20 providers). Patients, providers, and staff of these clinics are consented separately when their specific opinions or viewpoints are elicited (via surveys or interviews), with no penalty for non-participation. Data is collected at the clinics during facilitation sessions, from the EHR through automated extraction and from patients, providers, and staff via paper and online surveys.

The final trial design, shown in Figure 1, is a two-arm, cluster randomized controlled trial with clustering at the clinic level and stratification by size, location, and population. We considered a factorial design and a third arm with extant patient-centered medical homes but no specific interventions (usual care), but contamination prevented a factorial design and the third arm required access to unavailable clinic data.

These components consist of three aspects aimed at the clinic level: 1) incentives with audit and feedback based on overall and incremental goal performance; 2) an HIT system called the Integrated Care Coordination Information System (ICCIS) that measured care on individual patients as well as progress towards clinic goals; and 3) tailored practice facilitation. The sole difference between the intervention and control arm is that the control arm is given free rein to address any health reform goal and the intervention arm is encouraged—both through practice facilitation and incentives—to focus their transformation efforts on achieving as many high-value elements as possible. A summary is shown in Table 2 below. Across five topic areas for high-value elements, 12 subtopics were developed into 26 specific measures that allow flexibility and increase achievement. Components such as identification of at-risk population and care management are measured by provision of care plans, offering advanced directives, and outreach to high-risk populations. Patient engagement and proactive goal setting involve self-management support and reminders. Integrating information focuses on clinical information exchange and follow-up post-utilization, while population management focuses on the use of population management tools to proactively manage metrics and certain outreach components. Finally, access was added based on patient-identified need; after-hours access and availability of providers are the identified elements from the focus group. Alignment with health reform metrics is moderate to strong. Moderate alignment was defined as some overlap with the general area of health reform, but the metric did not exist in the reform model. For instance, risk stratified care management concepts were not measures in the reviewed initiatives but rather general goals. Strong alignment means that the precise measure was or is in place elsewhere, but the benchmark was more advanced or specific. For instance, follow-up on utilization is a specific CPC metric, but the study benchmark of 70% is more aggressive.

Clinics are eligible for the same amount of incentives. Practice facilitation is offered monthly to both intervention and control since recent summary literature suggests that practice facilitation both makes it possible to transform and directly affects outcomes of trials. At each visit, clinics are free to select from a wide variety of choices based on their current needs; while the control arm has 35, the intervention arm has 12 options. Previous goals are assessed by the practice facilitator and new ones set. The IT system is available at all times, with functions of risk stratification (built in scores), care management tracking, and self-assessment of all PCMH elements available for clinical team review. The incentive reports are built into the IT system, with the intervention and control receiving a general score based on setting and meeting QI goals, and intervention receiving a multiplier based on the number of high-value elements met. The multiplier, whose maximum was 2, set 30% of the potential incentive aside for HVE achievement. Once the clinics meet a HVE threshold of 2/3 of the elements achieved, they earn up to a 10% bonus for achieving additional elements. Additional file 1 shows the proposed IT reports for intervention and control.

The process to generate the scores was tested during the pilot study. In general, effort at QI is rewarded on a 0%–100% scale; factors included in the score generation are organization around QI, leadership involvement, team function, and setting and follow-up on goals. From the practice facilitator’s notes each month, a team of three investigators review each element and decide on a total score. This score is fed back at each visit, with a summary report of overall performance versus other clinics each quarter. The pilot clinic demonstrated a strong response to the reward of effort and the competition engendered by a ‘grading’ approach; they did require significant explanation of the scoring rubric and requested the opportunity to offer feedback to get a score adjusted.

There are three sets of measures planned for the trial. First, primary and secondary outcomes are aligned with the triple aim for cost/utilization, experience of care, and population health, with specific expected decreases planned as clinically meaningful and to set sample size. Primary outcomes are reduction in total cost of care for patients, with a focus on hospitalization rate and ED visit rate of patients as indicator measures. A reduction in utilization up to 20% per year has been seen in some related studies, but a conservative estimate of 10% relative reduction in utilization is planned for this study. Measurement of this outcome will be completed through a state-based All Payer All Claims database, which allows for attribution of patients to a clinic based on visits to primary care, and contains all claims for that individual across insurance types. This outcome and its specific measurement are strongly related to the health reform goals, using the same metrics for success. For experience of care, the Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) surveys at the patient level will be used, with an expectation of a 10% improvement in experience of care metrics closely related to the high-value elements. There are 11 composite scores, and 4 are determined to be closely aligned to the study aims: 1) providers support you in taking care of your own health; 2) providers pay attention to your mental or emotional health; 3) follow-up on results; and 4) access to care. A secondary outcome is a composite quality score at the clinic level based on standard chronic illness health reform measures used for the clinics. A 10% increase is expected in the control clinics, since previous work has shown that clinics often focus on common quality measures first. Table 3 outlines primary, secondary, and monitoring outcomes, along with their respective metrics and data sources.

Other secondary outcomes include measurements at the cluster and patient level based on stakeholder suggestion. For clinics, culture, microsystem team performance, coordination and collaboration, and adaptive reserve were felt by stakeholders and seen in the literature to be crucial to measure before and after the trial; thus, we will use the clinician staff questionnaire (CSQ), the Team Development Measure (TDM), the Collaborative Practice Assessment Tool (CPAT), and a novel collaboration tool for external collaboration between clinical teams. Qualitative interviews with patients and with clinics will be performed to understand the impact of study more broadly. Monitoring goals were discussed above but required monthly assessments of clinic engagement with QI goal setting and follow-up as well as the high-value elements will provide protocol monitoring metrics. Deviations from protocol will be recorded and used to calculate protocol adherence.

The sample size was based on eight hypothetical clinics with an average of 10,000 patients each. The intracluster correlation was estimated at .02, and the necessary sample size to detect a change of 10% in cost and utilization over the year was <6,000 patients with average costs of $730 per month (average Medicare spend for our region) and average hospitalization rates of 14% at a power of > .80. Besides the monitoring metrics, above, no interim analyses or stopping guidelines were planned due to the low risk of the study.

For randomization, with a small number of clusters (8), the study requires significant work to assure baseline similarities, where possible, and analytic techniques to adjust for site-specific differences. To that end, we used a sequence allocation that was computer-generated and used matching by key characteristics to place the clustered clinics into match groups. Match criteria included organization type and location; total patient population, and the number of patients that were high risk according to a standardized risk scoring system based on comorbidities that predict hospitalization. Sequence generation was done by a random number seeded by time; matched pairs of clinics are randomly allocated by this seed to either intervention or control. For the patient experience of care surveys, patients are randomly chosen by a computer program, with oversampling of ‘high risk’ at 50% of the overall sample. The principal investigator (PI) wrote the randomization algorithm, the matching criteria, and had multiple team members approve them as fair, unbiased, and without error prior to implementation. Once the program was run, it was sent directly to the other team members without intervention by the PI.

For blinding, incentives vary by arm and participants are shown their own incentives. To partially blind them, the terms ‘intervention’ and ‘control’ are not used, and all reports are only shown to individual clinics without noting differences between arms. Similarly, blinding for the study team (to avoid experimenter’s bias) is not possible, since the reports are seen across clinics by the team. To reduce bias, the practice facilitator is required to log all interactions and have these reviewed by a third party for time equality (same resource amounts provided to each), for minimal discussion of differences between clinics and arms, and for similar internal milestone achievement by each clinic. Blinding for the analyst is planned, with intervention and control arms labeled only as A and B, and clinic and patient identifiers are masked. Consent was sought from clinics, clinicians participating, and patients receiving the CG-CAHPS survey; other data collection is completed, done under a waiver of consent.

Descriptive analysis will compare baseline characteristics of clinics and individuals across outcomes and primary confounders (demographics, chronic illnesses, previous medical care, previous adherence to quality measures). A difference in difference calculation will compare changes in rates of outcomes for intervention versus control. Primary analysis will be a multi-level, two-stage analysis to account first for patient level and clinic level differences and then changes over time. Modeling will be done using generalized estimating equations; these can be adjusted for the level and the outcome variable specifics. The risk of selection bias by cluster is high, so to account for this, we pre-assess the population by risk of hospitalization using a standardized risk score prior to trial start. This group becomes the analytic cohort, and the size of each high-risk group is used to stratify clusters. We will generate the intracluster correlation, account for it by adjusting multivariate analysis, and perform propensity matching within the analytic cohort to assure similarity of baseline cohorts at the clinics. Overfitting will be tested using Copa’s test and model fit using Hosmer-Lemeshow, receiver operating curves, and Akaike information criterion for logistic-based models and Box-Cox test, GLM family test, Link and RESET test in addition to R², and mean-squared error for cost-based models.

Ethical approval was granted by the Oregon Health and Science University (OHSU) Institutional Review Board. At the time of manuscript submission, the trial is ongoing. While primary data collection has concluded, we are currently collecting data from CG-CAHPS surveys and have not yet begun analysis on primary outcomes.