The effect of adapting Hospital at Home to facilitate implementation and sustainment on program drift or voltage drop

Translating complex multi-component evidence-based interventions from study conditions and protocols to actual practice necessarily requires minor to major adaptation of intervention content, format, personnel roles and processes. Adaptations may be desired or necessary to customize to local circumstances or to account for new technology or temporal changes in disease epidemiology or clinical practice [1, 2]. Traditionally, it has been thought that variance from an evidence-based protocol is accompanied with at least some diminution of effectiveness. This effect has been termed “program drift” (i.e., decreased benefit arising from deviations from study protocols) and “voltage drop” (i.e., loss of benefit when interventions are applied in more heterogeneous patient populations and settings as they move from efficacy to effectiveness and into actual practice) [3].

More recently, the inevitability of diminished benefits due to program drift and voltage drop has been questioned. While fidelity to a study population, setting, and protocol may be critical for a biological or a clinical intervention to maintain the studied benefits, this may not be the case for health system interventions or more complex multi-component interventions. Adaptations may be made as the intervention is applied in more heterogenous patient populations or practice settings, thereby potentiating voltage drop. Other adaptations may be made to the intervention protocol to improve fit with the practice setting and the environment, thereby potentiating program drift. Other adaptations may address technological or medical advances that were not available or formally manualized at the time of the evidence-based study. In theory, these adaptations may have neutral effects. Some adaptations may enhance the effectiveness of the original intervention^3, thereby neutralizing any diminished effectiveness from program drift or voltage drop or perhaps even increasing the net benefit beyond that observed in evidence-based studies. Other adaptations may inadvertently diminish benefits. Indeed, some have called for the systematic study of adaptations to evidence-based interventions and a formalized structure for updating the evidence base to account for adaptation [4].

To improve our understanding of the effect of adaptations, we took advantage of an implementation of an evidence-based hospital admission avoidance Hospital at Home (HaH) intervention [5]. This implementation necessitated extensive adaptation of studied procedures to local setting and culture and evolving medical practice. In 2014, The Mount Sinai Health System, a seven-hospital system in New York City, began implementing HaH with 30-day post-acute care follow-up of patients. For select patients with specific diagnoses (e.g., pneumonia) who would otherwise be admitted, HaH services (e.g., intravenous antibiotics, fluids, oxygen, etc.) and daily clinician visits are provided at home along with durable medical equipment, phlebotomy, and home x-ray as needed. We examined whether HaH associations with outcomes changed over time as adaptations were put into place.

Funded as an innovation award from the Center for Medicare and Medicaid Innovation, this implementation of HaH was instigated with the goal of eventual sustainment in the local health system and scaling and dissemination nationally. The phased plan for implementation within the local health system was consistent with the concepts outlined in the Dynamic Sustainability Framework (DSF) [3]. The DSF posits that interventions should be implemented with consideration of fit with practice settings and the external ecological system of competitors, regulation, market forces, and population needs. The framework also puts forth that these elements are not static and will change with time. Our phased implementation anticipated a piloting phase followed by initial implementation in one hospital emergency department with phased expansion over several quarters to other health system settings. We anticipated the need to adapt HaH to each hospital’s unique culture, medical staff, electronic record system, community, and labor practices. Similarly, we anticipated that HaH might need to be adapted to changing policy and regulations, different payers, and a healthcare market in dynamic flux due, in part, to the implementation of the Affordable Care Act.

Over the 33 months of implementation, a series of program adaptations were implemented (see Table 2), including a) adding a 30-day post-acute transition component to the HaH model (at inception); b) expanding target diagnoses and modifying exclusions from those originally studied (Quarter 1) due to changes in medical practice (e.g., removed HIV exclusion from decade-old protocol of original HaH studies); c) outside contracting for certain services (e.g., infusion) rather than providing the service directly with program staff (Quarter 2); d) implementing HaH variations due to change in medical practice (e.g., palliative care unit at home [Quarter 1], observation unit services at home [Quarter 3]) or opportunity (e.g., rehabilitation at home services [Quarter 3]); e) adapting intake procedures (e.g., holding patients overnight in the emergency room [Quarter 4]) for patients entering the program late at night when delivering supplies to the home to initiate a HaH episode was less available; f) launching telehealth visits (Quarter 4); and g) changing staffing and staff roles (Quarters 2, 6, 8). Other more minor adaptations were made throughout the implementation period.

All adaptations were initiated by the program team (see Table 3). Modifications were most commonly made in format (or how the intervention was delivered) and in personnel and roles. Modifications were made at the level of the cohort (e.g., for patients who would otherwise have been admitted to an observation unit in the hospital), patient population (e.g., for patients with new diagnoses or with palliative care needs that would not have been eligible for the original HaH intervention) or organization (e.g., staffing modifications that affected the entire program regardless of hospital site). The nature of adaptations was most commonly to add or to substitute new program elements. In three instances, these adaptations were sufficiently extensive for the cohort or population that it involved integrating the HaH approach to additional patient populations (palliative care patients) or to other clinical constructs (e.g., observation unit services or postacute rehabilitation).

HaH services substituting for a hospital stay were received by 295 patients. The average LOS was 3.2 days. Of our sample, 12.2% had escalations, 8.6% had readmissions within 30 days, and 5.8% had ED visits within 30 days. The overall rating for hospital care was rated at the highest level by 68.8% of patients.

Graphs of these outcomes, adjusted for patient characteristics and season, over the nine quarters of implementation are provided in Figs. 1, 2, 3 and 4. A median of 33 patients (range 11–44) received services each quarter. The graphs show occasional spikes in events such as readmissions. Confidence intervals and standard errors were moderately large at any point in time; however they indicate that changes, if any, were small over time. The graphs indicate the quarter of initiation for the adaptations listed in Tables 2 and 3.

Regression models controlling for patient characteristics and season of admission are available in Additional file 4. Older age was associated with higher ED revisits and higher ratings of care. African American patients had shorter LOS and higher 30-day ED revisits and readmissions. Having Medicaid was associated with higher ED revisits and lower ratings of care. Poor self-rated health was associated with longer LOS, greater escalations, and higher ratings of care.

In the regressions, we did not observe a relationship between quarter from program inception and LOS (− 0.007 days/quarter; SE 0.02, p = 0.75). The odds ratios (OR) per quarter from program inception was mildly increased for escalations (OR 1.09, 95% CI 1.01 to 1.18, p = 0.03), but was not increased or decreased for 30 day ED visit (OR 0.93, 95% CI 0.86 to 1.01, p = 0.09), 30-day readmission (OR 1.00, 95% CI 0.93 to 1.08, p = 0.99), and the patient providing the highest rating for overall hospital care (OR 0.99, 95% CI 0.93 to 1.05, p = 0.66).

We made adaptations to an evidence-based model of HaH at program inception and throughout a 33-month implementation. Our experience reinforces concepts from the DSF. Many adaptations were made to improve HaH operations. Others were made to respond to patient needs and referring provider requests such as for palliative care patients who needed acute hospital-level care. Some adaptations were made to improve scalability such as extending intake hours into nights and weekends. Other adaptations were made because new opportunities became available to enhance the patient experience such as the initiation of video telehealth and community paramedicine visits. These adaptations would have been difficult to include at program inception or even to script into phased introductions as the availability of the new opportunities or need for adaptation was unknown at that time. In some cases, adaptations were necessary to adapt to differing practice contexts at different hospitals. Expansion to other hospitals usually required new intake procedures to adjust to local culture, differing payer mix and social determinants of health in the surrounding community, and even different unions. The intervention also had to be adapted to payment and market forces affecting our hospitals (e.g., unanticipated creation and expansion of observation services during the period of implementation) and vendors (e.g., vendor budget issues, including vendor bankruptcy). We believe that the need and opportunity for dynamic adaptation occurs to varying degrees in the implementation of all new programs and is not limited to the HaH implementation experience we chart in this paper [2].

We modified the protocols used in our HaH program from the existing research evidence base described in the literature. For example, the addition of new diagnoses, alteration of clinical exclusions, and the addition of 30-day postacute follow up were changes from studied protocols. Despite this, for selected findings of LOS and patient satisfaction, our outcomes were similar to those reported in the original studies in Table 1. Further, in a previous report [6], these outcomes over the period of implementation compared favorably to those of a comparison group of similar patients who were hospitalized in regular inpatient units. Thus, our implementation of an evidence-based intervention with its associated adaptations was accomplished with no obvious “program drift.”

Additionally, we made adaptations in which we modified the HaH approach to changing context each quarter. For example, we added palliative care and other programs, as well as telehealth and community paramedicine visits. However, we did not observe significant changes in effectiveness across a variety of measures. LOS, 30-day readmissions, ED visits, and patient ratings of care did not significantly change over time. Tracking and reviewing these outcomes on a monthly basis as part of a continuous improvement activity may have been important in maintaining these effects through the implementation period. In the case of escalations, we observed a trend toward higher rates in later quarters. This was expected as our clinicians purposely started the program with less complex patients and enrolled increasingly complex patients as the team gained experience. The escalation rate was actually closely monitored throughout implementation, the program team was conscious of the trend, and cases of escalation were reviewed by the team on an ongoing basis in the interest of continuous improvement. Thus, we believe that the implementation of serial adaptations may have also avoided the phenomenon of “voltage drop.”

Our study was limited by sample size by quarter. There are also occasional quarters where readmissions or other outcomes appear to increase. However, these occurrences are transient and the trend over the nine quarters is one of no change. A further limitation is that individual adaptations could be hypothesized to potentially enhance or diminish net benefits or program efficiency. For example, the community paramedicine intervention was intended to improve response time, enhance ability to adjust treatments in the home, and to reduce trips to the ED. One could hypothesize that this might enhance effectiveness. On the other hand, observation services at home were a departure from the evidence-based model, and one could hypothesize that these patients might be less likely to benefit. Because adaptations were made as the need or opportunity arose, it is not possible for us to isolate and examine the effect of any individual adaptation in this analysis.

The implementation of evidence-based programs necessarily requires adaptation. We made significant adaptations to HaH at inception and then serial adaptations over the course of implementation. Over the course of the implementation, many of these important outcomes were tracked and fed back to the program leadership leading to further adaptations. Our findings may indicate that adaptations to evidence-based programs may avoid diminished benefits due to potential ‘program drift’ or ‘voltage drop’.