Transfer of learning and patient outcome in simulated crisis resource management: a systematic review

Despite an abundance of existing literature on simulation-based education and CRM, we identified only nine articles that examined transfer of learning to the workplace by healthcare providers or changes in patient outcome after simulation-based CRM training. The vast majority of the literature has been limited to lower-level outcomes, such as reaction of participants and learning that has been measured using further simulation scenarios. This approach leaves the studies open to the criticism that learners may have been taught to perform well only in the simulator and not necessarily in real life.

These findings are relevant to various stakeholders such as healthcare providers, researchers, educators, policy makers, healthcare institutions, and broader organizations. Although limited in quantity and quality, the literature suggests that simulation CRM training may have a significant impact on transfer of learning to the workplace and on patient outcome.

Currently, no consensus exists on the learning outcomes unique to simulation (i.e., simulated patient outcome, simulated behaviours, etc…) and how best to assess these factors. For example, Kirkpatrick does not adequately capture studies like that of DeVita et al. 30 where the main outcome measure was survival of the simulated patient. This may be because the Kirkpatrick model for evaluating learning interventions was not originally developed for simulation education.12 Although Kirkpatrick’s model is most often used to appraise the quality of educational research, we agree with Yardley and Dornan31 that other frameworks may be relevant for appraising the quality of educational research. They write,31 “Aggregative or interpretive methods of evidence synthesis that mix qualitative with quantitative evidence, or synthesize qualitative evidence alone, give better knowledge support and start from constructionist rather than positivist epistemological assumptions.” Medical education is pluralistic, and a positivist paradigm lens alone cannot capture its complexity. As a widely adopted framework specific to simulation education outcomes is presently lacking, it is important to recognize that the Kirkpatrick classification may not accurately capture all higher-level learning outcomes in simulation education. An ideal framework for simulation and education interventions would account for complexity of interventions, maintenance of behaviour changes, and differentiate between self- and external skill assessment and between simulated and real practice.

The data from this review provide evidence that CRM simulation training can improve behaviour at the workplace; however, whether this kind of training directly improves patient outcome is not as clear. Various measures to approach patient outcomes were used in the papers included in our review, including patient care efficiency (time to perform), complications, length of stay, and mortality. While most would agree that complications, length of stay, and mortality are appropriate criteria to assess patient outcome, it is debatable whether patient care efficiency is appropriate. Only one study found that simulation CRM training had a clearly significant impact on mortality following inhospital pediatric cardiac arrest.29 This study was simply a cohort study in a single hospital with no control group, thus results may potentially be due to other concomitant hidden interventions, and therefore, no strong definitive conclusion can be made regarding the causal relationship between the teaching intervention and mortality. Only an RCT with a control group could show that the teaching intervention is the reason for better survival. The practical requirements for designing studies that examine improvements in patient outcome can be difficult due to the need for larger sample sizes and a control group. For example, one of the studies included in this review did not have a sample-size calculation, and this likely resulted in an underpowered study. All of the studies included in this review involved one time-limited intervention on a small number of subjects.21 It is possible that modification of patient outcome requires a whole series of interventions on many subjects. Finally, although CRM programs without simulation teaching have been linked to decreased surgical mortality,32 we could not find a multicentre RCT that evaluated simulation CRM training on patient outcome. Nevertheless, if we compare with other high-stake industries, like aviation, despite several studies showing an improvement in pilots’ behaviour in the cockpit, studies showing the benefit of CRM pilot training on client safety are lacking.33

Another potential reason for our small sample size of studies may be the conservative nature of our inclusion criteria. The decision to include objectively measured change of behaviour at the workplace and to exclude self-assessment (Kirkpatrick Levels 1 and 2 – reaction and knowledge and skills learning, respectively) may have limited our analysis; however, self-assessment is largely recognized as inaccurate for healthcare professionals.18,34 The initial literature search was performed without any language restriction. Nevertheless, we included studies published in English or French only. Of course, we cannot ignore that a few papers were excluded because they were published in other languages. Given that the vast majority of scientific journals are published in English and all high-impact factor journals are in English, in our view, it is unlikely that the conclusions of our review would be significantly different if more languages had been included.

Overall, we found that the studies were at an intermediate or high risk of bias and reporting was suboptimal. First, there is clearly room for improvement in the approach used to report studies. For example, random sequence generation and allocation concealment were almost never reported properly in the included studies. We cannot determine if the studies were performed incorrectly or if “only” the reporting was poor. Second, it may be challenging to design studies on simulation-based CRM without risk of bias when investigating transfer of learning to the workplace and patient outcome. For example, when working in increasingly complex organizations, it is very difficult to ensure that risk of contamination is nonexistent and intervention is independent of other changes. We suggest that, as a field, the simulation community needs to commit to rigorous research reports. Also, larger and multicentre studies could balance the risk of contamination. In order to decrease the risk of bias as much as possible in future studies, we also suggest that researchers consider the risk of bias at an early stage when designing the protocol.

Moving forward, larger sample sizes, more multicentre studies, and studies with less risk of bias are required to provide a precise measure of the effect that simulation-based education has on healthcare provider skills in the workplace and patient outcome. Other systematic reviews show that there is no need for more Kirkpatrick Level 1 (reaction) and Level 2 (learning) studies, since learners are virtually constantly positive toward simulation training10,11 and learning occurs when measured in a simulated environment.1 Frequency of retraining, skill retention, and instructional design remain research priorities in studies investigating Kirkpatrick Level 3 (transfer of learning at the workplace) and Level 4 (patient outcome) outcomes. Universally recognized rigorous assessment tools are necessary to compare the effect of various teaching interventions and to assess CRM regardless of the clinical context. Finally, simulation training is often underused, potentially due to its cost. Future research could better explore the cost-effectiveness of simulation CRM training.