Simulation Training in Neuroangiography—Validation and Effectiveness

The number of endovascularly treated patients suffering from stroke or aneurysms has steadily increased over the last few years [1]. In 2015, five large randomized stroke trials established the superiority of mechanical thrombectomy in combination with intravenous recombinant tissue plasminogen activator (i.v. rt-PA) over i.v. rt-PA alone [2‐6]. Since then, the number of treated patients has risen continuously. Moreover, the time window has been recently broadened [7, 8], which will further increase the number of patients eligible for interventional treatment. Although mechanical thrombectomy is performed in more and more patients at centers with great expertise, demand is increasing for an on-call service around the clock at multiple smaller centers. Hence, a sufficient number of interventional neuroradiologists must be trained at a high level. Using high-fidelity simulators offers the possibility of structured training without endangering patients [9, 10]. Although the first prototypes of simulators specifically for neuroradiological applications were developed many years ago [11, 12], they are not yet widely used in neuroradiology. After proof of face and content validity, based on expert opinion [13] the next crucial step to establish such a new approach includes the demonstration of its construct validity, i.e. beginners and advanced users can be distinguished via the parameters measured by the simulator. Only when this prerequisite is met should such a simulator be used for training interventionalists. Concerning interventional procedures, construct validity was previously shown for renal [14, 15] and cardiac procedures [16, 17] as well as for carotid stenting [18, 19]. After the validity or neuroangiographies of another make, the ANGIO Mentor Express (Simbionix, Cleveland, OH, USA) has already been proven [20], the present study aimed at establishing the construct validity of the Mentice VIST simulator; however, construct validity is not directly related to the effectiveness of a simulator. To be used as an effective training tool in practice, improvements in psychomotor skills should be measurable [21, 22]. Therefore, the second aim of the study was to quantify improvements of different quality measures of a diagnostic neuroangiography, resulting from repeated simulator training. Of interest was, in addition to an already confirmed training effect for beginners and intermediates [10, 20, 23, 24], whether this impact could also be demonstrated for experts.

In this study it was possible to prove the construct validity of a neuroangiographic simulator by demonstrating significant performance differences between beginners and experts. We were furthermore able to show the effectiveness of such a simulator by directly measuring significant improvements in psychomotor skills of beginners in cases derived from real patient anatomy.

This is in line with previous studies on different scenarios, such as carotid artery stenting [13, 18] and several infra-aortic applications [9, 17] and addresses one major concern regarding neuroangiographic simulations: In 2008, Carroll and Messenger stated that “medical simulation has made the transition from an experimental technology to the clinical world”, and that “perhaps the most pressing issue […] regarding medical simulation is validation” [27].

To prove the construct validity, we had clearly separated groups of beginners with a maximum of 30 cerebral angiographies performed and of experts with at least 100 procedures. The simulator data for all 10 procedures evidently demonstrated this subdivision. As previously confirmed for cardiac angiography [28], beginners need more time to find and examine the target vessels. Thus, total duration and fluoroscopy time differed significantly. Beginners more often produced roadmaps; accordingly, the total amount of contrast agent was significantly higher for beginners. The fact that the simulator only shows the contrast of the arteries and contains no parenchyma or the venous phase may serve as a reason for the missing difference in the duration of the series. Thus, in this parameter no difference was present between experts and beginners. Overall, these results contradict the findings of Nguyen et al. who were only able to identify the amount of contrast agent as a distinguishing feature for the experience level [20]. A possible reason could be their small number of 2 compared tasks, while our study design consisted of 10 procedures.

Observations of gaze direction, blink frequency, pupil size and dwell time are recognized means of examining attention and cognitive stress [29]. Based on such data Richstone et al. were able to distinguish unequivocally between beginners and experts in surgical laparoscopy [30], but for endovascular cardiac interventions, Currie et al. found hardly any differences [31]. The assumption that experts would turn their gaze less often away from the X‑ray screen could therefore not be confirmed by them. The present study focused on the direction of each subject’s gaze; no differences between the two groups could be observed.

Heart rate was recorded, as it was assumed that it would differ between various levels of workload and thus between beginners and experts according to the effort made. In previous work on anesthesiologists, Martin et al. as well as Weinger et al. found significant differences in several heart rate parameters, including mean heart rate, during different phases of an anesthetic procedure [32, 42]. In a later work of the first group on pre-hospital emergency medicine, heart rate variability discriminated better between different levels of workload compared to the mean heart rate [33]. Currie et al. also measured numerous parameters such as heart rate variability, electrodermal activity and skin temperature in physicians undertaking cardiac endovascular procedures but could not find differences between experience levels [31]. Hence, mean heart rate seems to be an unreliable predictor, matching the results of our study, where it was not suitable to distinguish between beginners and experts.

The NASA-TLX is a frequently used tool to assess subjective workload, especially in anesthesia and the field of emergency care [34], but also in other areas, e.g. radiotherapy [35] or flight simulation [36]. In comparison with the previously mentioned complex technical tools, the NASA-TLX offers an easy and fast method for recording the workload. The modified version Raw-TLX (RTLX), without the weighting process of the subscales, is even easier to use [37]. In a study on surgical robotics, differences were found in NASA RTLX by examiners with different experience levels [38], just as in our participants. Apparently, there are no data on whether NASA RTLX changes differently through training of beginners compared to experts. The satisfaction of our participants only increased when the other measurable parameters also improved, which was only true for beginners. We therefore assume that the simulator is well suitable to give adequate feedback on one’s own performance.

An increase in performance with respect to the two essential parameters of total duration and fluoroscopy time has been demonstrated in our study for beginners and confirmed the results of Spiotta et al. and Zaika et al. [10, 24]. Among our participants, however, this effect could not be observed among experts. Thus, not every experience level benefits from simulator training that focuses exclusively on diagnostic angiography. This supports the assumption that in the best case skills are learned that an expert already possesses. For beginners, the training effect can be objectively read from the metrics of the simulator, as known for peripheral endovascular procedures [14, 39]. In addition, we have shown that repeated exercises of the same cases are helpful for beginners not only to increase their metric values, but also to increase their own satisfaction. Spiotta et al. also noted an increase in confidence in the acquisition of skills, both in terms of knowledge of the anatomy and the technique of vessel selection [23]. Not least for this reason, several centers have now begun to implement a structured milestone-based curriculum and propose to integrate simulation training into formal neuroendovascular training [23, 40]

When demonstrating a training effect, it is always difficult to distinguish the extent to which a real gain in specific psychomotor skills overlaps with increasing familiarity with the operation of the simulator [39]. An improvement merely by habituation would be recognizable in all subjects; however, since only the group of beginners improved their performance relevantly, this effect seems to be negligible here.

Notably, in the RTLX evaluation, the degree of frustration of the experts slightly increased during training, whereas it decreased among the beginners (both not significant). Comments from the participants showed that individual experts were disturbed by the differences between simulation and reality, whereas beginners were not. The participants of a simulator training should therefore be informed in advance with respect to the differences that can be expected in the behavior of the simulator in relation to reality. Otherwise, they approach the training with a variety of ideas and demands, and frustration and anger can easily arise [1].

Construct validity should distinguish not only between beginners and experts, but ideally also between various levels of experience [16]. In our study, the small number of test persons prevented further division of study participants. Also, the number of cases, in particular of repeated cases was limited in this study. A higher number of cases might also show differences in expert neuroradiologists.

Construct validity of a high-tech simulator could be demonstrated for diagnostic neuroangiography and especially beginners showed a measurable training effect through repeated practice. Further studies should demonstrate the benefit of such simulation training for the patient.