Virtual reality suturing task as an objective test for robotic experience assessment

We enrolled 38 subjects in the study with a previous experience range from 0–2,000 robotic cases. All participants completed the “Tubes” module twice, in which raw and percentage values were obtained for the overall score and the 9 individual metrics included raw and software calculated percent (%) score. The median time to complete the task was 4.5 (2.5–13.6) min on the first attempt and 3.9 (1.9–14.2) min on the second attempt. In addition to the overall score, the individual metrics that improved from the first attempt to the second were: use of the workspace, economy of motion, missed targets, and time to complete the task (Table  2).

Educational experience consisted of residents (n = 9), novice robotic surgeons (n = 7), intermediate (n = 9), experienced (n = 7), and expert robotic surgeons (n = 6). Compared to expert surgeons, residents in training and novice robotic surgeons showed significantly lower overall MScores on the Tubes module on both attempts (first attempt p = 0.039 and second attempt p = 0.023) (Table  3). The median overall MScore on the first attempt was 47.5 (range 0–95) and the second attempt was 62.5 (range 2–98). The MScore on the first attempt was correlated to the number of previous robotic cases (Spearman Correlation 0.465 (p = 0.003) (Fig.  2a). The second attempt was no longer correlative (Spearman 0.200; p = 0.228) (Fig.  2b). Significant differences in the mean MScores were noted comparing the novice robotic surgeons and surgeons with some robotic experience on the first (44.7 vs. 63.9; p = 0.005) and second attempt (56.0 vs. 69.9; p = 0.037). The overall scores did improve on the second attempt by 8.82 %, in which the novice group improved to a greater extent than the experienced group (11.4 % vs. 6 %; p = 0.012). We graphed each individual subject’s 1 ^st and 2 ^nd attempts and connected each score with a line to display trends of improvement within each sub-group of experience level (Fig.  3). In this figure, novice and residents without attending level experience on the robot nearly unanimously improved on the second attempt. The other levels of robotic experience seem to be less predictable, possibly due to their particular robotic expertise or experience with this particular simulator.

In order to identify metrics most influential to the overall MScore, we performed logistic regression and identified that experienced surgeons had more out of view penalties if adjusting for missed targets (p = 0.014) on the first attempt despite having higher overall scores. On the second attempt, time (p = 0.014) and missed targets (p = 0.004) were the most significant factors between the novice and experienced groups (Table  4).

Therefore, the “Tubes” simulator module within the DVSSS does have construct validity to determine if the subject has performed more than 10 cases previously. The virtual simulation task, therefore, may be useful as an objective assessment of proficient use of the robotic console defined as basic functional use of the robot system (not surgical proficiency). Limiting the test to only one difficult virtual reality simulation may limit the amount of information obtained, however, the test can be performed quickly (approximately 5 min) and efficiently.

A previous study has suggested that the use of virtual reality robotic simulation may serve as an assessment tool in a variety of settings [ 19]. We tested a wide range of robotic surgical experience to determine if one task or module (“Tubes”) could have the ability to provide assessment value in proctoring in a future study. Proctoring requires another surgeon to assess the new surgeon’s ability to perform a particular surgery and report to the credentialing authority [ 20]. The proctor’s prior training, experience and ability to judge competency may be highly variable; however, the proctor does have the authority and responsibility to recommend further training prior to a surgeon being given unrestricted privileges to perform robotic surgery [ 10, 20]. Therefore, introducing an objective measure to assess the surgeon’s comfort with the robotic system may be helpful to identify those surgeons who may need to take part in a standardized robotic curriculum prior to robotic privileging.

The simulator can identify particular tasks the user may need additional practice or training on, making the test a learning opportunity (Fig.  1). Metrics such as workspace utilization, economy of motion, missed targets and time all play important factors in the overall MScore and can provide an opportunity for self-reflection and improvement. No to be understated, the more experienced group of robotic surgeons had more “out of view” errors. More experienced surgeons may be moving robotic arms out of the field, potentially causing safety concerns to which a common reaction would be: “I know where the arm is.” Even experienced surgeons should use the simulation as an opportunity for improvement. Another scenario would be if a novice scored exceptionally well on the MScore tasks. The surgeon would still go through the usual proctoring and prove the ability to troubleshoot the robotic system, but may not need additional mentored robotic console training.

The simulator can provide these metrics; however, in order to incorporate them into proctoring, a benchmark needs to be set to provide assessment. Many proficiency based surgical curricula and training programs are based on pass rates of 80 %–91 % to progress to the next level [ 9, 21]. We then identified that an overall MScore of 75 % would serve as the lowest possible score that could still distinguish surgeon experience even if the surgeon is granted a second attempt. On first attempt only 1 of 20 (5 %) novice surgeons were able to pass compared to 8 of 18 (44 %) surgeons with >10 robotic cases (p = 0.001). When given a second attempt, 2 additional novices (15 %) and 1 additional surgeon with some robotic experience (50 %) were able to pass with maintenance of statistically significant difference between the groups (p = 0.020). Overall, 23 failed both attempts and 6 passed both attempts, 3 passed on the first but not second and 6 failed on first and passed on second attempt. We emphasize the contrast in distinguishing familiarity with the robotic console and basic operation, not surgical proficiency. The cutoff values are arbitrary but should be consistent to compare surgeons to their peers and provide baseline proficiency.

If a surgeon “fails” their first attempt, the natural inclination is to try again. Therefore, we had the subjects perform the task a second time to determine improvement levels, as described above. We found that both the novice and experienced groups were able to improve their overall MScore on the second attempt, with the novice group able to improve to a greater extent. The improvement provides suggestive information that the simulator does have a learning curve and repeated measures may improve their virtual reality score, which may in turn provide improved operative efficiency [ 9]. Thus, turning the proctoring experience into an opportunity for improvement with specific recommendations on areas of focus. Based on the simulator results, the learner can be directed to a surgeon-specific training curriculum if needed.

The culture regarding operative safety has drastically improved in the last few decades with the use of safety checklists, pathways, and guidelines [ 22]. One component of safety that has not been investigated sufficiently is the incorporation of new technology and the surgeon. Previous studies suggest a reduction in errors may be achieved with the use of surgical simulation similar to flight simulation in the aviation industry [ 23]. Currently, simulators for surgical training, practice and warm-up are not widely utilized [ 24]. Barriers of cost, validation, and optimal specific simulators and tasks have hampered widespread adoption.

Our study is limited by the sample size and number of repeated measures. In addition, we relied on the surgeons to remember the number of robotic cases they have performed, which may be subject to recall bias. Of note, we did not ask their previous experience with simulators. The study was developed in the context of a robotic training course, which interjects some bias regarding the participants. The novice surgeons and residents are not yet at the point of requiring robotic surgical privileges, although a range of skill was needed for the purposes of the study. Additionally, the use of surgical simulators have limitations in that the DVSSS is attached to the actual console and can only be performed when not in use for patient care. Similar software has been used in the Da Vinci Trainer (Mimic Technologies, Seattle, WA, USA) as a tabletop simulator that may be more mobile though may have less working space [ 12]. The simulator’s virtual reality environment is improving but continues to have low fidelity compared to actual human surgery. Therefore, simulator based testing can only offer an assessment regarding operation of the robotic equipment and not surgical decision-making. We stress that this single simulator test is helpful, but may not be ready for widespread use and standardization. Subjects may not be familiar with the simulator and may need to perform practice sessions first; due to time constraints of the study, we selected one tool and performed it twice. The use of simulators prior to incorporating them into credentialing should be rigorously studied and tested, such as the Fundamentals of Laparoscopic Surgery examination for general surgeons [ 25, 26]. Proctors with robotic experience would be needed to evaluate actual robotic surgical proficiency. With the help of telemedicine, these opportunities may be offered to hospitals that do not have experienced robotic surgeons available [ 27].