Improved procedural performance following a simulation training session may not be transferable to the clinical environment

Abstract

Objective:

Neonatal intubation is a life-saving procedural skill required by pediatricians. Trainees receive insufficient clinical exposure to develop this competency. Traditional training comprises a Neonatal Resuscitation Program (NRP) complemented by clinical experience. More recently, simulation is being used in procedural skills training. The objective of this study is to examine the impact of a simulation session, which teaches the skill of neonatal intubation by comparing pre- and post-intervention performance, and examining transferability of skill acquisition to the clinical setting.

Study Design:

First-year pediatric residents with NRP training, but no previous neonatal experience, attended a 2-h intubation education session conducted by two experienced respiratory therapists. Individual components of the skill were taught, followed by practice on a high-fidelity infant mannequin with concurrent feedback. Skills were assessed using a validated neonatal intubation checklist (CL) and a five-point global rating scale (GRS), pre- and immediately post-intervention, using the mannequin. Clinical intubations performed in the subsequent 8-week neonatal intensive-care unit (NICU) rotation were evaluated by documenting success rates, time taken to intubate, and CL and GRS scores. Performance was also compared with similar data collected on intubations performed by a historical cohort of first-year residents who did not receive the training intervention. Data were analyzed using descriptive statistics, Student's t-test and χ²-test as appropriate, and analysis of variance.

Result:

Thirteen residents participated in the educational session. Mean pre-intervention CL score was 65.4±18% (s.d.) and GRS was 3±0.7 (s.d.). Performance improved following the intervention with post-training CL score of 93±5% (P<0.0001) and GRS of 3.92±0.4 (P=0.0003). These trainees performed 40 intubations during their subsequent NICU rotation, with a success rate of 67.5% compared with 63.15% in the cohort group (NS). However, mean CL score for the study trainees during the NICU rotation was 64.6±20%, significantly lower than their post-training CL score (P<0.001), and significantly lower than the historical cohort score of 82.5±15.4% (P=0.001). In the intervention group, there were no significant differences between the pre-intervention and real-life CL scores of 65±18% and 64.63 %, respectively, and the pre-intervention and real-life GRS of 3.0±0.7 and 2.95±0.86, respectively.

Conclusion:

Trainees showed significant improvement in intubation skills immediately post intervention, but this did not translate into improved-clinical performance, with performance returning to baseline. In fact, significantly higher CL scores were demonstrated by the cohort group. These data suggest that improved performance in the simulation environment may not be transferable to the clinical setting. They also support the evidence that although concurrent feedback may lead to improved performance immediately post training intervention, this does not result in improved skill retention overall.