Implementation of a multi-site neonatal simulation improvement program: a cost analysis

Neonatal resuscitation is a critical, albeit low-frequency event that necessitates high levels of individual and teamwork skills. Patient harm and medical errors may occur due to suboptimal communication, lack of adherence to neonatal resuscitation guidelines, and other systems issues [1]. Simulation-based debriefing for healthcare professionals may facilitate teamwork and improve patient outcomes [2, 3]. On-site simulations or in situ simulations that take place in workplace settings allow for realistic and educational scenarios where healthcare teams are encouraged to uncover individual or team weaknesses, system errors, and latent safety threats that can prompt changes in practice. Despite several reports on such simulation programs, there has been little data on the costs and resources involved in their implementation [4‐8].

The California Perinatal Quality Care Collaborative (CPQCC) developed the Simulating Success quality improvement program to help California hospitals implement in situ simulation-based neonatal resuscitation training [9, 10]. The Simulating Success program consisted of online didactic training modules to review the foundations of simulation-based training as an educational methodology; face-to-face training on the core principles of developing, conducting, and debriefing simulation-based training; recurrent expert debriefings of in situ simulations at participant sites with monthly online check-ins; and follow-up site visits to provide continued feedback and support. This study aimed to examine the costs associated with the design and implementation of the Simulating Success program. The experience shared here may inform future implementation of other in situ simulation-based quality improvement programs.

Table 2 summarizes the breakdown of Simulating Success program cost. The cost for designing and delivering the program totaled $228,148.36 for CPQCC. Personnel cost accounted for the largest share of the CPQCC cost (92.2%), followed by program-related travel (including transportation, food, and lodging) (6.1%), equipment/supplies (1.5%), and space use (0.2%). Cost for implementing the program at the four study sites varied from $18,056.60 to $30,326.27. Distribution of the cost categories was relatively similar, with personnel cost accounting for 77.8%-87.1% of the site-specific costs. Site A had the highest implementation cost and used a total of 534.5 person-hours, whereas site B had the lowest implementation cost and used a total of 233.0 person-hours (Table 3).

After allocating all CPQCC-incurred program design and delivery costs to relevant participant sites, the cost of the Simulating Success program averaged $39,210.69 per site (95% CI: $34,094.52–$44,326.86) (Table 4). In sensitivity analyses varying the useful life of manikins from 5 to 12 years, the estimated mean cost became $39,935.73 per site and $39,089.84 per site, respectively. If the program was expanded to 30 or 50 sites, the estimated mean cost would be $37,669.55 per site and $36,863.42 per site, respectively. If the program was taken as is and directly adopted at other sites (i.e., disregard the program design cost), the estimated mean cost would be $35,645.22 per site.

The Simulating Success program appeared to substitute some of the neonatal resuscitation training needs at participant sites. This was evidenced by a lower monthly cost of other neonatal resuscitation training during the program implementation period (mean: $1,112.52 per site) than the baseline pre-implementation period (mean: $2,504.01 per site) at all four sites (Table 5).

In the sustainability period immediately after the Simulating Success program ended, monthly cost of all neonatal resuscitation training averaged $1,965.12 per site, which was also lower than the average monthly cost in the pre-implementation period (Table 5). However, there was large variability across the four sites resulting in a large 95% CI ($457.82–$3,472.41). The monthly cost of neonatal resuscitation training was lower in the sustainability period than in the pre-implementation period at two sites but higher at the other two sites.

Reports on the implementation of simulation programs in various settings are ample in the realm of healthcare, with many suggesting cost-effective results [20‐23]. However, studies reporting simulation implementation specifically within the NICU and its associated cost are sparse. Our study addressed this gap by presenting data on the Simulating Success program, which was designed to promote and facilitate simulations at multiple hospitals with the goal of improving local patient outcomes by addressing teamwork and local systems issues for practicing NICU healthcare professionals. Our findings inform the financial and logistical requirements for completing a collaborative of this magnitude and scope.

It should be noted that establishing a multi-site program across multiple sites in a state as large as California is a difficult task to accomplish. Expectations for the central design team included significant time investment from personnel, meetings to communicate needs and progress with participant sites, development of a training curriculum (online and in-person), provision of supplies and equipment, and appropriate coordination of space use and travel for in-person training and site visits. For instance, the design, training, and subsequent program support required a total of 3,455.8 person-hours from a diverse team of CPQCC staff and collaborators (e.g., neonatologists, nurse practitioners, registered nurses, research assistants, and a technical writer). Execution of the Simulating Success program benefited from the bandwidth of CPQCC and its well-established network within California. Nevertheless, the needs and expectations of participant sites are often diverse, and the most effective simulation-based training programs need to be tailored to the needs of each specific sites. Within the context of Simulating Success, the central design team drew on CAPE’s pre-existing curriculum and adapted it via close collaboration with CAPE faculty. Therefore, when developing future multi-site simulation implementation programs of this scale, adequate capacity of the central design team that implements initial facilitation should be considered to ensure project completion.

The varying cost of implementing the Simulating Success program across participant sites likely reflect variation in the complexity of personnel involved (e.g., number and type of healthcare professionals) and the needs for neonatal resuscitation training at each site. In particular, the number of NICU beds varied by six-fold across the four study sites included in our analysis which may signal different personnel needs. The cost of implementing the Simulating Success program was generally higher at sites with more NICU beds, whereas participant sites that had fewer NICU beds tended to incur lower costs even in the pre-implementation period. For instance, the Simulating Success program costed $3,053.71 per month at the site with the fewest NICU beds to $3,821.71 per month at the site with the most NICU beds. Sites with different NICU sizes and staffing structures likely differ in the type and number of providers involved in neonatal resuscitation training activities and the intensity of their training activities. Further research comparing different structures of the training team and modalities (e.g., amount of virtual versus in-person training, frequency and size of simulation sessions, and composition of core team) will help inform ways to reduce costs while preserving the clinical benefit of the program.

Although the design and delivery of the Simulating Success program itself involved high resource utilization, it may have financial benefits in the long run. For instance, the program activities helped offset the need for some of the other neonatal resuscitation training as shown in our data. Two of the study sites had a lower monthly cost of overall neonatal resuscitation training in the sustainability period than in the pre-implementation period. Although we do not know the exact type of training activities that was occurring at participant sites prior to the Simulating Success program, it presumably involved some standard neonatal resuscitation training which does include simulation albeit not in situ simulation. This is consistent with other studies demonstrating the cost-neutral effect of an in situ simulation model using minimal permanent space and redirected faculty educational time [20]. While the unpredictable nature of NICU workflow and patient acuity can sometimes make in situ simulations challenging, integrating practice into regular work shifts may enhance efficiency and also more accurately reflect how teams would work in real life. Having the experienced CAPE team that had already refined strategies of simulation and debriefing for neonatal resuscitation may have reduced the overall cost. If there can be sustained benefit in improved efficiency in training and practice, the high program cost may be further offset in the long run. Moreover, our sensitivity analysis suggests that the high cost of Simulating Success program design can be more economical if there are more participant sites to share it.

In addition to the financial impact, it is also important to consider the program’s effect on improving clinical practice. Research on the Simulating Success program showed that although it did not improve neonatal survival without chronic lung disease, the program may have an impact on unit practice [10]. In addition to improving clinicians’ technical and behavioral skills, participant sites were able to holistically benefit from the program, with many participants reporting improved team participation, ability to identify latent safety threats, and process/system changes within respective NICUs [1, 9, 10, 24]. In focus group discussions, neonatal healthcare professionals at participant sites reported that the Simulating Success program helped provide an environment in which clinicians felt less punitive and safe to describe team performance and team communication, which can facilitate culture change in the unit [24]. Future studies evaluating the cost-effectiveness of such programs should consider these benefits in clinical practice and system improvements in addition to patient outcomes.

We recognize several limitations of this study. First, out of the 14 sites that completed initial training, we only had adequate cost data from four of them. This exemplifies the challenges in collecting cost data for implementation projects in busy clinical settings. For instance, accurate tracking of cost information may involve data collection approaches that participants are less accustomed to, activities that are above their typical workload, and information that may not be readily documented/available (e.g., extracting nuanced information from invoices, documenting specifics about meeting durations and room sizes, and recording details about attendee credentials and transportation). When there is limited staff, clinical care and actual implementation of the simulation program would take priority over these data collection activities. Therefore, experience on the financial impact of the four included sites may not be generalizable to all participant sites, especially given the larger NICU size at these four sites in comparison to other participant sites (median: 48 NICU beds versus 25 NICU beds). As the volume of training activities and staff involved may be smaller in scale at NICUs with fewer beds, it is possible that if costs from all participant sites were analyzed, the average program cost per site would be lower than our estimates. Likewise, our findings may not be generalizable elsewhere in the country because the Simulating Success program was implemented in California. However, whenever feasible, we used national mean unit prices in cost estimation (e.g., mean hourly wage rates for personnel time which accounted for the largest share of the program cost). Second, some participant sites had limited healthcare team availability and high turnover in key roles which was time-consuming in conducting simulations and re-training [10]. As a result, our data likely overestimated the cost and underestimated the efficacy of the Simulating Success program. Third, although we included a sustainability period and collected cost data in the three months immediately after the program ended, the longer-term impact of the Simulating Success program remains unknown. It is uncertain whether the intensive training and support provided during the program had a lasting effect on provider practice and teamwork over time. Longer-term follow-up data would be helpful to further inform the financial and clinical benefits of the program. Finally, all NICUs participating in the Simulating Success program were located in urban areas. Although we anticipate the impact of the program on NICUs in rural areas would be similar to sites with fewer NICU beds in our sample, future research formally evaluating such programs in rural NICUs will be instrumental.

Establishing a multi-site neonatal in situ simulation program requires an investment of sufficient resources, personnel, and time from design and participant sites. This should be adequately recognized and carefully considered prior to project initiation. However, such programs have the potential to improve simulation-based training and bring positive changes in practice.