Evaluation of a new goal-directed training curriculum for point-of-care ultrasound in the emergency department: impact on physician self-confidence and ultrasound skills

This study was a quasi-experimental trial carried out to investigate the impact of two different POCUS curriculums on ED physicians’ performance/self-confidence. We conducted the trial at the ED training center of Peking Union Medical College Hospital (PUMCH). We recruited participants enrolled in PUMCH emergency POCUS training course from 28 December 2015 to 17 June 2017. Trainees enrolled before 31 December 2016 underwent training using the traditional curriculum, and those enrolled after received training using the new curriculum. As the enrollment to the PUMCH bedside ultrasound training course was open to all interested emergency physicians nationwide, there was no requirement for prior ultrasound education or hands-on experiences, and no specific control for trainees’ allocation to either the traditional or the new goal-directed curriculum.

We approached the two consecutive cohorts of trainees for consent and included only consent trainees in the study. We further excluded trainees who failed to finish the course or/and did not respond to follow-up. The institutional board of PUMCH reviewed and approved the study, considering this study presented no more than minimal risk to subjects.

For both standard and new curriculums, trainees participated in a two and half day training course in which they formed study groups of about four to seven members. One or two teaching physicians were responsible to coach individual study group in hands-on sessions.

Subjects enrolled in the training program before 31 December 2016 received training using a traditional curriculum. The curriculum started with an introductory lecture on the fundamental ultrasound knowledge, which covered physics and instrumentation of emergency ultrasound, followed by lectures on how to obtain standard ultrasound views without emphasis on relevant ultrasound anatomy, or normal/pathological images interpretation. After the educational lectures, the program provided hands-on practices on simulated manikins for trainees to obtain standard views with the guidance of instructors. The curriculum completed with the introduction of Rapid Ultrasound for Shock and Hypotension (RUSH) protocol and a lecture for using POCUS to aid the diagnosis of undifferentiated shock.

Subjects enrolled in 2017 underwent training using an improved, goal-directed curriculum. We identified the five core applications that ED physicians commonly perform to assess patients in their anatomical structures and underlying pathophysiological changes: Focused Assessment with Sonography in Trauma (FAST), thoracic, vascular, ultrasound-guided puncture (UGP) and echocardiography. Details of the five ultrasound applications are included in Appendix 1. The learning objectives of the training were to facilitate trainees to correctly carry out the above five ultrasound applications and obtain accurate assessment through a limited number of standard views. Also, we retained our emphasis on the POCUS procedure for undifferentiated shock diagnosis and introduced the “Tamponade/Tension, pneumothorax, Heart, Inferior vena cava, Respiratory system, Deep venous thrombosis/Aorta Dissection” (THIRD) procedure in the new curriculum [16].

Before formal courses, we provided trainees with pre-course learning materials to familiar with the necessary knowledge of ultrasound and the consensus of evaluation for undifferentiated shock patients. The pre-course materials contained reading literature and web-based video sessions. The ED of PUMCH has published details about the pre-course learning in May 2017 [16].

During the formal course, we improved the first introductory lecture for basic ultrasound knowledge by integrating hands-on practice with coaching, which included ultrasound probe holding, probe placement/movements, understanding of three ultrasound plains of tridimensional organ, explanation on four sides of the view, and summary on six steps of obtaining a high-quality view.

After the introductory lecture on basic ultrasound knowledge, we also changed the sequence of subsequent training sections with the improvement of contents and technologies. Subjects had the first lecture on the introduction of anatomy with 3D video records and photos of dissecting real organ. The second lecture provided the tips to gain a standard view in details and compared 3D images with normal ultrasound views. Trainees practiced drawing ultrasound views on simulated manikins after this session. Each trainee was assigned to a group for hands-on practice instructed by group instructor, the ratio of instructor to trainee was 1:7. Once the trainee performed required standard view on the manikins, simulated 3-D images were presented to the trainee to demonstrate various pathological findings under that particular standard view. We also provided standard patients with no pathological conditions to trainees alongside with the simulated manikins. Subjects had the last lecture on the interpretation of pathological ultrasound images with the real clinical case.

The final step of the new curriculum was to review and practice in groups all critical ultrasound diagnostic/treatment procedures (FAST, thoracic, vascular, UGP, and echocardiography) with emphasis on the THIRD protocol for undifferentiated shocks. Instructors presented simulated case scenarios and corresponded ultrasound findings while trainees were obtaining images on models.

We conducted a baseline survey of study participants before training for basic demographics and prior ultrasound experiences, which include seven single choice questions and one open-ended question. The participants were requested to rate for the self-confidence in performing of five core applications: FAST, thoracic, vascular, UGP, echocardiography and diagnosing undifferentiated shock (RUSH for old curriculum or THIRD for new curriculum). The confidence measurements ranged from 1 to 5 with 1 being the least confident and 5 being the most confident.

Multiple course instructors assessed trainees’ performance in carrying out the ED ultrasound procedures (FAST, thoracic, echocardiography, and UGP) after training. To give a fair assessment for trainees enrolled in the study, we assigned course instructors to evaluate the performance of trainees who they were not in direct coaching with during hands-on and group sessions. However, we could not completely blind the instructors of the trainees’ seniority and experiences. Instead, we specifically required an objective assessment of the trainees’ skill level using an anchored rating scale as the assessment tool: We assign score of one to participants who were unable to obtain a view and unable to explain normal image even with extensive instructor’s coaching; three when they were able to obtain a low-quality view and give some of the image interpretation with some coaching; five when they were able to gain a standard view with high image quality and make accurate image interpretation without assistance. The total performance score summing up the 4 subcomponents, 36 items ranges from 36 to 180. All course instructors received prior standardized training in the right using of the assessment tool.

One month after the completion of the training, we contacted trainees about the reassessment of self-confidence and post-training ultrasound practices. We chose to follow-up on trainees’ self-confidence 1 month after the training because we believe by that time trainees would have had enough time to digest the learning in the course and make enough attempts to practice POCUS independently.

This survey was administered online using the research electronic data capture (REDCap) medical data system.

We also performed multivariate linear regression analyses to examine the independent association between training programs and outcomes while controlling for the potential baseline confounders. Only imbalanced baseline characteristics between the two trainee cohorts were included in the multivariate regression model (either p value < 0.05 or relative differences greater than 10%). The statistical significance threshold was set at p < 0.05. Individuals with missing data were excluded from analysis. All data analyses were performed using STATA 12.0 (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP.).

During the two consecutive training programs, we enrolled and consented 66 trainees from over 20 hospitals throughout China. We excluded two trainees who did not finish the course for personal reasons. Twenty-seven trainees enrolled before 2017 received traditional POCUS training, while 37 trainees received training using the new curriculum.

On average, trainees had 5.58 years of practice experiences in ED, and 51.6% had some prior ED ultrasound training. Many indicated that they had performed ED ultrasound before, with the most frequently performed being ultrasound-guided puncture (45.3%), followed by vascular (37.5%), echocardiography (28.1%), and the least performed being diagnostic procedures for undifferentiated shock (7.8%). Similarly, the prior experiences with different ultrasound procedures were also reflected in the corresponding baseline self-confidence, with diagnosing undifferentiated shock being the one with the lowest confidence. However, a small proportion of them indicated that they were independently performing ED ultrasound before training (9.4%) (Table 1).

After training, all trainees reported significantly increased self-confidence in performing ED ultrasound as compared with baseline. One month after training completion, the confidence tended to decrease moderately from the end of training but was still higher than the baseline measurements. For example, confidences in performing shock protocol were 1.50, 3.44 and 2.59 for baseline, training completion and 1 month after training, respectively (Tables 1, 2).

Trainees received traditional/new training programs that differ in several baseline characteristics. Trainees with new curriculum tended to have more years practicing in ED (7.41 vs. 3.07, p < 0.001), more likely to have prior ED ultrasound training (64.9% vs. 33.3%, p < 0.001), performed slightly more ED ultrasound before (total amount: 17.35 vs. 13.70, p = 0.208; counts of types of ED ultrasound: 2.03 vs. 1.19, p = 0.060). Trainees who received the new curriculum also had higher baseline self-confidence measurements than those who received traditional curriculum (Table 1). However, trainees with new curriculum are less likely to indicate they were independently performing ED ultrasound (8.1% vs. 11.1%) with an insignificant p value of 0.691 at the threshold of 0.05.

Trainees with new curriculum had statistically significantly higher measurements in self-confidence after training, reflected in numbers both at training completion and at 1 month after training. Even after adjusting for the differential distribution of baseline self-confidence by calculating the absolute delta of post-training confidence from baseline, the new curriculum was associated with higher trainee self-confidence (post-training vs. baseline and 1-month post-training vs. baseline). For example, in FAST performance, the confidence measurements were 3.92 (new) vs. 3.37 (traditional) for post-training, 3.27 vs. 2.26 for 1-month post-training, 2.03 vs. 1.78 for increased confidence post-training as relative to baseline, and 1.38 vs. 0.67 for increased confidence 1-month post-training as relative to baseline.

Trainees with new curriculum also ended up performing more ultrasound post-training than trainees received traditional curriculum (14.2 vs. 1.44, p < 0.001) and had higher performance scores at training completion (165.76 vs. 113.44, p < 0.001).

When adjusting for potential confounding factors such as imbalanced baseline characteristics and baseline confidence levels, the new curriculum was consistently associated with higher increases in self-confidence measured at either training completion or 1 month after (except for ultrasound-guided punctures), more substantial number of ultrasound performed in 1-month post-training, and higher performance scores at training completion (Table 3).