Effect of implementation of the MORE^OB program on adverse maternal and neonatal birth outcomes in Ontario, Canada: a retrospective cohort study

MORE^OB program implementation over the 12-year study period, for each included hospital, is presented in Fig. 1. Designing a robust evaluation of the MORE^OB program implementation was challenging for several reasons, including the staggered implementation times across the hospitals and the long and variable program implementation phases [12‐14]. We approached our analysis using a quasi-experimental retrospective cohort design with analysis based on the approach used for stepped wedge trials. The stepped wedge is a type of study design which can be used to evaluate interventions implemented in phases at different points over time [15, 16]. In stepped wedge studies, outcomes are assessed repeatedly before and after implementation, and the overall effectiveness of the intervention is determined by comparing data points in the post-intervention section of the wedge, to those in the pre-intervention section.

Initially, the 67 hospitals that implemented the MORE^OB program prior to 2012 were included. The study population was comprised of the mother-newborn dyads born at these hospitals between fiscal years 2002–2003 and 2013–2014.

In preparing the dataset for analysis, we accounted for hospital mergers and closures over the course of the 12 years of follow up, which decreased the number of institutions from 67 to 63. To avoid instability in the multivariable analysis, we subsequently excluded the 8 institutions with annual birth volumes less than 250. Thus, 55 institutions were included in our final analysis.

We obtained a 12-year dataset of maternal-newborn hospitalization records from the Canadian Institute for Health Information’s (CIHI) Discharge Abstract Database (DAD) for Ontario hospitals between April 1, 2002 and March 31, 2014 (i.e., fiscal years 2002–2003 to 2013–2014, inclusive). The DAD includes hospital separation abstracts submitted by all acute care hospitals in Ontario. Each hospital abstract contains demographic information, medical diagnoses including most responsible diagnosis and 24 secondary diagnoses, interventions received, length of hospital stay, vital disposition at time of discharge and other data elements. Medical diagnoses are coded using the Canadian implementation of the International Classification of Diseases, 10th Revision (ICD-10-CA) from 2002 to 2003 to the present. The DAD captures over 98% of all births, including all live birth and stillbirth events occurring in Ontario hospitals through the maternal record. In addition, there are separate records for live born neonates and stillbirths that capture relevant morbidities, diagnoses and medical procedures. We linked maternal and newborn abstracts together via an encrypted maternal-newborn chart number [17].

We analyzed two previously-developed measures of obstetric patient quality of care: a composite individual-level indicator of incidence of any adverse maternal or neonatal events, and a hospital-level score, called the Weighted Adverse Outcome Score (WAOS). Both measures capture maternal and neonatal outcomes, and were developed through repeated consultation with leaders in nursing, obstetrics and anesthesia, among others [18]. The decision to use these two composite indices was made for several reasons. Using previously-developed measures facilitates comparison between our study and previously published research [5, 6, 18‐20]. Further, the MORE^OB program is comprehensive and multifaceted, and could affect a number of maternal and newborn outcomes. Therefore, using composite scores would better capture the potential impact of the program. Note that several modifications were required, most importantly the omission of the Apgar score, as it was not available in the hospitalization database.

The composite individual-level indicator of adverse event occurrence was based on the Adverse Outcome Index (AOI), a composite score designed to capture the quality of obstetrical care defined as “the number of deliveries complicated by one or more of the identified outcomes divided by the total number of deliveries” [18]. We modified the AOI for our analysis (henceforth, mAOI), and used it as a binary individual-level indicator, rather than as a proportion calculated at the monthly and hospital level, where a ‘1’ indicates the presence of any component of the AOI and a ‘0’ indicates no components had occurred for that individual. We decided to use an individual-level analysis for the mAOI to allow for unequal weighting based on hospital birth volume. The mAOI captures the number of deliveries affected by one or more of the adverse events included in the indicator (see Table 2 and Additional file 1). The WAOS additionally accounts for the severity of these events, by assigning weighted scores to each event (Table 2). Further, because the weights are summed, both the severity and number of events contribute to the score. For example, maternal death is assigned the highest score at 750 points per event, and 3rd or 4th degree perineal tears are assigned the lowest, at 5 points per event. If a delivery is complicated by multiple events, their scores are summed [18]. The weighted sums are then divided by the number of deliveries that took place during a particular time period (e.g., per month).

The timing of implementation for each hospital was divided into segments based on program implementation: the pre-implementation period, the implementation period which included implementation time for each of the three MORE^OB modules, and the post-implementation period. Two different datasets were prepared, one for each outcome. The mAOI was defined as described above and analyzed at the individual level, and the WAOS was defined at the level of the hospital, as the sum of all the weighted composite scores, divided by the number of deliveries that occurred in that hospital. Therefore, the analysis of the WAOS required the creation of a dataset aggregated by month and hospital.

The components of the composite indices were coded from the ICD-10-CA and CCI (Canadian Classification of Health Interventions) codes in the DAD, after reviewing published lists of codes used for maternal-newborn research using the DAD [19, 21‐23]. The codes used for our study, along with additional information about the components, can be found in Additional file 1. Components of the mAOI were modified based on the information that was available in the DAD dataset. Specifically, the Apgar score was unavailable and, therefore, not included, and the data quality of 3rd degree perineal tears was found to be problematic and therefore our component was restricted to 4th degree perineal tears. Information on the timing of module implementation for each hospital was obtained from Salus Global.

The mAOI was specified as the dependent variable and analyzed at the individual woman-level using random effects logistic regression, estimated using restricted pseudo-likelihood estimation. MORE^OB program implementation, coded as a step function taking the value 0 before implementation, increments of 0.25 with each subsequent module, and 1 after completion of all modules, was included as a fixed effect. To account for the underlying secular trend, (i.e., natural changes that occur over time [24]), time was modeled as a restricted cubic spline with five knots. Level of care, annual birth volume, and an obstetric comorbidity index [25] were also included as fixed effects. The previously developed and validated obstetric comorbidity index [25, 26] has been found to predict severe maternal morbidity and mortality and includes more than 20 maternal comorbidities and conditions that increase the risk of adverse pregnancy outcomes [25, 26]. To allow individual hospitals to deviate randomly from the secular trend, random intercepts and coefficients for time were specified for each hospital. The regression coefficient for MORE^OB program implementation was exponentiated to yield an adjusted odds ratio (aOR) for the effect of program implementation. This odds ratio compares the odds of an adverse event for a woman in a hospital after program implementation, to that of a woman in the same hospital before implementation. Three important assumptions of our model are that, after adjusting for covariates, there are no systematic differences in pre-implementation levels and trends across hospitals; that the effect of the intervention is constant over time (i.e., the models yield an estimate of the time-averaged hospital-specific effect of the intervention); and there is a gradient effect of the implementation across all modules, meaning a constant increase from the first to last modules.

The WAOS was analyzed at the hospital-level using mixed effects linear regression, estimated using restricted maximum likelihood estimation. Fixed and random effects were as described for the analyses of the mAOI, with the exception that the comorbidity index was aggregated to the level of the hospital in each month. The regression coefficient for the MORE^OB program implementation variable yields an estimate of the difference between the mean score post-implementation and the mean score pre-implementation.

For both outcomes, the fitted secular trend was obtained from the model and displayed graphically with the covariates (hospital level and annual birth volume, and comorbidity index) set to their mean values in the study sample.

Secondary analyses included separate models for individual components of the composite score, and examining sensitivity due to misspecification of the secular trend by specifying time as a categorical (rather than continuous) variable. Also, between-hospital heterogeneity in the effect of the program was allowed for, by adding a random effect for program implementation defined at the level of the hospital. All analyses were carried out using SAS v. 9.4.

In our analysis of adverse maternal and neonatal outcomes in 55 Ontario hospitals enrolled in the MORE^OB program prior to 2012, we found no evidence of improvements in either of the composite indices after MORE^OB implementation. In fact, there was a small increase in the occurrence of adverse events as captured by the mAOI. In subanalyses with individual components of the composite indices, we found small but statistically significant increased odds of maternal unanticipated operative procedures, 4th degree tears, and NICU admission; on the other hand, we found evidence of non-statistically significant decreases in neonatal birth trauma and maternal blood transfusion.

Two previous studies have assessed the effect of the implementation of the MORE^OB program. In the first study, implementation of the MORE^OB program in Alberta hospitals was associated with a reduction in 3rd and 4th degree perineal tears and decreases in maternal length of stay and neonatal morbidity [11]. However there were limitations to this study, including the analysis which was vulnerable to confounding by the underlying secular trend, as well as the short follow-up time period. In the Alberta study only 2.8% of deliveries occurred post-Module 3, whereas in our study 39% of deliveries occurred post-Module 3. Using a multiple baseline interrupted time series (ITS) design, the second study assessed the effect of MORE^OB implementation on mandatory reportable events collected by a Canadian healthcare liability insurer [10]. In that study, the authors reported a decrease of 4 mandatory reportable events at 3 years after MORE^OB implementation (95% CI:-0.5 to 8.1) and a decrease of 8 mandatory reportable events at 6 years post-implementation (95% CI: 1.4 to 15.1). However, this study included a smaller number of Ontario hospitals, 34, whereas the current study included 55 Ontario hospitals.

Other obstetric patient safety programs have been evaluated with mixed results. A recently published rapid review included 10 randomized controlled trials (RCT) and reported that “provider education and other quality improvement strategy combinations targeting healthcare providers may improve the safety of women and their newborns during childbirth” [7]. In an evaluation of a program implemented at an American hospital from 2004 to 2006, it was reported that the program was associated with a decrease in the AOI [5], as well as a decrease in the number of liability claims and associated payments [4]. The nine elements of the program included items such as the development of new protocols and guidelines, the hiring of an Obstetric Safety Nurse, the implementation of an Obstetric Patient Safety Committee, the involvement of Obstetric Hospitalists to provide 24-h, 7-day per week in-house coverage, as well as team training and an electronic fetal heart rate certification program [4, 5]. However, this program was implemented at one hospital, and a simple pre/post linear regression analysis was used, making it difficult to draw conclusions about what the wider impact of such a program would be. An RCT carried out at 15 American hospitals assessing the effect of a teamwork training program on the AOI reported no effect of the program on this outcome [6]. Based on the literature to date, it is difficult to conclude what elements of an obstetric patient safety program might be most likely to improve effectiveness.

In our analytical approach, we allowed for a gradient effect of MORE^OB implementation (from the first to the final module) but thereafter, the effect of the MORE^OB program was assumed to be stable. An alternative approach may have allowed for either a gradual decay or further gradual improvements after completion of all modules, using, for example, an ITS approach with segmented regression analysis allowing for both a step change and a slope change. However, the staggered implementation of the MORE^OB program, combined with the lengthy and variable amount of time required for hospitals to complete the program, precluded the use of an ITS approach. In a classical ITS analysis, the estimated pre-implementation trend (secular trend) is used to predict what the outcome event rate would have been at the end of the study, had the intervention not been implemented (the counterfactual estimate) [12‐14]. With a long separation, using the secular trend to predict a counterfactual estimate is risky due to projection far outside the observed range of the data.

While the staggered implementation of the MORE^OB program, which occurred over an 8 year period, presented analytic challenges, it also provided an advantage, in that our results are less likely to be confounded by concurrent interventions or local policy changes. Using our multivariable model, we were essentially able to compare outcomes from all hospitals after implementation of the program to their own pre-implementation outcomes as well as to the outcomes from other hospitals who had not yet implemented the program at the same calendar time. As well, the use of previously developed and used [5, 6, 18‐20] indicators for adverse maternal and neonatal outcomes, the mAOI and WAOS, further strengthens our results. Given the multifaceted nature of the MORE^OB program, we considered that indicators capturing multiple components of maternal/newborn care would be more appropriate. Further, aggregation of the components helps to address the rarity of these adverse outcomes in the Canadian healthcare setting. And, given that it had been used for research carried out in Canada [19], this allows for the comparison between our estimates and what has been reported elsewhere. The AOI rate reported by Hutcheon et al. of 5.7% is lower than the 6.8% that we report here, and the WAOS we report is also higher (4.9 versus 1.9). It is not entirely clear why these rates would differ inter-provincially across Canada, and given the severity of these outcomes, this might highlight an area for further research.

We believe using composite indices as the primary outcomes was the most appropriate choice however it is possible that this masks the effect of the program on specific maternal-newborn outcomes that were not assessed by our study. As well, our analysis was limited to the data elements available in the DAD hospitalization database, and therefore we had to use a modified version of the AOI, which did not include elements such as Apgar score or 3rd degree perineal tears. Data quality is carefully managed and documented by CIHI [27], and we made every attempt to identify any outstanding data quality issues including examining the study outcomes over time and by hospital site, and liaising with CIHI when issues were identified. Yet, it is possible that unidentified changes to coding or data quality remain. Also, the occurrence of maternal and neonatal adverse outcomes is affected by a host of factors unrelated to participation in a program such as the MORE^OB, such as maternal age, parity and comorbidity. We attempted to account for this by including hospital level of care, birth volume and a maternal comorbidity index in our models however, it is possible that unmeasured confounding remains. An additional limitation was the lack of clinical performance measures in this evaluation, such as compliance with guidelines, safety reports, or documentation of near misses. Further, changes to workplace culture were not assessed. It has previously been reported that participation in MORE^OB improved workplace culture and increased knowledge [8], and it is difficult to understand how improvements to workplace culture would not result in improvements to patient outcomes. Inclusion of objective measures of behavior such as those described above, along with assessments of workplace culture might help to shed light on this issue in future studies.

It is perhaps not surprising that we were unable to detect an impact of the MORE^OB program on a composite indicator of adverse maternal and neonatal outcomes. In this evaluation, which is one component of a larger mixed-methods evaluation of the MORE^OB program, we only measured adverse outcomes. Although these are important outcomes from the perspective of patients, hospitals and policymakers, they cannot fully capture other aspects of MORE^OB training related to improved communication, improvements in safety culture and increases in knowledge. Our mixed-methods evaluation of the MORE^OB program also assessed individual participant responses, including care providers’ views about the program, changes in their levels of knowledge, and their assessments of the culture change within their organizations. Our findings from that study suggest that the MORE^OB program had a positive impact on these factors. It is well-established that clinical behavior and practice change takes time to become entrenched [28, 29]. As well, when a hospital embarks on the 3-year commitment to implement the MORE^OB program, they must also find strategies to sustain the activities and practice changes once their dedicated funding for the program runs out. If the activities and practice changes are not firmly established and culture change is temporary, there may be no detectable net benefit of the program on clinical adverse outcomes over time. The MORE^OB program does attempt to mitigate this effect, in that it is not just an educational program, rather every attempt is made to embed the program within the culture of each institution. A dedicated core team in each hospital carries out ongoing activities related to the program [9].

Improving patient safety and reducing adverse maternal and neonatal outcomes remain issues of critical importance in obstetrics, and our study demonstrates some of the important challenges associated with this area of research. Our study also demonstrates the critical nature of large datasets and long follow-up, which is especially important when studies are carried out in high-resource settings where adverse outcomes are rare. Great care is also needed in the selection of appropriate research and analytical methods. Future studies should look at the effect of obstetrical safety programs on other adverse outcomes such as near misses, as well as including objective measures of behavior change.