The SIMULATE ureteroscopy training curriculum: educational value and transfer of skills

Newly appointed residents with experience of 0–10 procedures and no structured simulation training in URS were recruited from UK deaneries as well as centres from Austria, Germany, Greece, Switzerland, Turkey, China, Japan, Canada and the United States of America. Training sessions were conducted for the simulation arm of the RCT [9] utilising an expert-developed training curriculum on five separate occasions (Fig. 1). Participants were guided through each phase of the curriculum with local board-certified urologists. Live OR assessment was requested as follow-up from each participants’ supervisor, who were blinded to whether they received simulation training, as part of the greater trial [9].

Available training models with the highest levels of evidence for semi-rigid and flexible URS were identified as URO-Mentor (Simbionix, Israel) VR simulator, Uro-Scopic Trainer (Limbs and Things, UK) and Scope Trainer (Mediskills, UK) [10]. A Delphi process was undertaken for curriculum development, with input from experts and trainees in urolithiasis, as previously described [9].

The curriculum begins with didactic lectures including:

It continues with select tasks and cases on the URO-Mentor VR simulator, suitable cases on bench models and diagnostic cases (Fig. 1) on fresh frozen cadavers (FFC), if locally available. All efforts were made to keep training standardised across sites, in line with the developed curriculum. Didactic lectures and supervision took place in multiple languages, according to the site of delivery. FFCs could only be utilised for UK participants (n = 14) and VR Simulation was unavailable for Japanese trainees (n = 5); however, participants were taught these tasks and cases on dry-lab models to compensate.

Throughout training sessions, supervising faculty assessed participants on their performance using the well-validated objective structured assessment of technical skills (OSATS) assessment scale [11]. Video recordings of performances were also obtained and distributed to two urolithiasis experts for the blind assessment using OSATS. All trainees were invited to complete an anonymous post-training evaluation survey, asking to rate different aspects of training on a 5-point Likert scale.

The primary outcome of this study is to report the educational value of the SIMULATE URS curriculum. Other outcome measures were content validity, transfer of skills and inter-rater reliability. Educational value was measured using participant opinion through post-training surveys. Acquisition and transfer of skills were measured by improvement in candidate performance through the curriculum and the OR, as per OSATS scores. Measurement of inter-rater reliability was conducted between the two experts as well as live rating by faculty. Secondary analyses were performed to compare the OR performances of those exposed to FFC (n = 14) vs non-FFC (n = 32) and VR (n = 41) vs non-VR (n = 5) groups to further evaluate these expensive components of the curriculum.

Descriptive statistics were used for questionnaire data. OSATS scores were converted to percentages since there were aspects which could not be rated through video assessments such as “Use of Assistants” and Knowledge of Instruments”. Pearson’s correlation coefficient and Cohen’s kappa tests were utilised to investigate correlation and agreement, respectively, in scores for inter-rater reliability, using SPSS^® Statistics version 26 (IBM^®, Armonk, NY, USA). A mean OSATS score was used for the remainder analyses. GraphPad Prism version 8 (San Diego, CA, USA) was utilised for all graphs and perform other basic statistical tests. Following the establishment of normality through Shapiro–Wilk tests, unpaired t tests were performed to compare differences in performance between participants who were exposed to FFC, not exposed to VR and those that only trained using VR and bench models. A one-way ANOVA test was used to measure improvement in OSATS scores over the curriculum. A p value < 0.05 was considered statistically significant for all tests.

This study recruited a total of 46 participants (Fig. 1), supervised by a total of 19 specialists over the five conducted sessions. Participants were all residents, in years 1–4, enrolled onto a urology training programme, with ages ranging between 24–43 years (Mean 29.4 ± 3.7). 11 of 46 participants were female (24%). As per our eligibility criteria, all recruits were novices in URS and had performed a mean of 2 semi-rigid (Range 0–9) and 1 flexible URS (Range 0–7) procedures. 70% of participants returned follow-up data in the OR (n = 32), as assessed by their local supervisors, using OSATS.

The training programme was evaluated by participants on a Likert scale (1 = ‘Strongly Disagree’, 5 = ‘Strongly Agree’) and well received. Respondents stated that their skills improved upon completing the training curriculum (Mean 4.39 ± 0.78) and that they gained transferrable skills for the OR (Mean 4.33 ± 0.67). Furthermore, trainees thought that simulation-based training is essential for patient safety (Mean 4.56 ± 0.62) and that there is a role for further dedicated procedural training curricula (Mean 4.56 ± 0.62).

There was marked improvement in performance throughout the curriculum (Fig. 2) and transferred to the OR for both semi-rigid URS (p = 0.0006) and flexible URS (p = 0.0003). Technical performance was observed to be particularly low when participants performed cases on FFCs for both semi-rigid (Mean 58% ± 4) and flexible URS (Mean 53% ± 12) alike. In terms of semi-rigid URS, no differences were observed in OR performance between trainees exposed to FFCs (n = 9, Mean: 70.1% ± 14) and those who trained using VR and bench models (n = 18, Mean 70.6%; p = 0.95). Similarly, no statistically significant differences were observed between those not exposed to VR (n = 5, Mean 52.0%) and trainees who trained on VR and Bench (n = 18, Mean 70.6%; p = 0.07). Likewise for flexible URS, both comparisons were insignificant for FFC vs non-FFC [(n = 9 vs n = 16), (Mean 67.7% vs 69.9%), p = 0.79] and non-VR vs VR and bench [(n = 5 vs n = 16), (Mean 52.6% vs 69.9%); p = 0.10].

Comparisons were made between live OSATS rating and the two experts video assessors (Fig. 3). There was generally a poor agreement between all three parties, but moderate correlation between the video assessments of experts A and B. Strong correlation was observed between raters A and B for semi-rigid URS Bench-1 (r = 0.72) and Bench-2 (r = 0.68) cases as well as fURS Bench (r = 0.70) and Cadaveric (r = 0.76) cases. There was one isolated case of fair correlation between the live raters and rater B (r = 0.55) for fURS cadaveric case.

Using a Likert score (1 = ‘Least Useful’, 5 = ‘Most Useful’), participants rated various aspects of the utilised simulators ≥ 3/5 for suitability (Supplementary Figure). The most highly rated modality was FFCs (Mean 4.10 ± 0.55), of which the highest-rated aspect was C-arm control (Mean 4.72 ± 0.46). Stent insertion was rated relatively lower (Mean 3.06 ± 1.39) and further explanations noting this task proved to be difficult due to the ureters being frozen was provided as qualitative feedback. This was followed by Uro-Scopic Trainer (Mean 3.88 ± 0.31), Advanced Scope Trainer (Mean 3.59 ± 0.33) and the URO-Mentor (Mean 3.57 ± 0.28). Both the dry-lab bench models scored highly in instrument handling, stone fragmentation and stone extraction. The URO-Mentor VR simulator scored a mean of 3.9 on ureteric navigation, stone extraction and stent insertion. Qualitatively, it was reported that flexible URS tasks were particularly useful on this simulator.