Augmented reality for the virtual dissection of white matter pathways

The European Low-Grade Glioma Network (ELGGN) workshop group, consisting of 15 experts with neurosurgical, neurolinguistic, and neuroscientific experience, met in December 2019. Since the attending ELGGN members were experienced in cadaver and in vivo fiber dissections, we decided to evaluate AR and its potential for virtual fiber dissection courses with this target group.

The prepared cases consisted of patients who underwent microsurgical glioma resection during awake craniotomy at our department. Table 1 gives an overview on the patients and tumor characteristics as well as the clinical course. The study approved by the local ethics committee and was performed in accordance with the Declaration of Helsinki. All included patients provided written informed consent.

Firstly, the cortical surface and tractography of existing fiber bundles, based on commonly used clinical software, and delineation of other relevant regions of interest, such as a tumor to be resected, are required. The MRI sequences which were used for the DTI FT were performed on 3 T magnetic resonance scanners (Philips Medical System, Netherlands B.V.). All patients obtained MRIs according to the standard glioma protocol at our department including a T1-weighted three-dimensional (3-D) gradient echo sequence with intravenous contrast administration for anatomical co-registration, a T2-weighted 3-D FLAIR sequence, and DTI sequences with 32 orthogonal sequences. For the visualization of white matter pathways, we performed DTI FT of eloquent pathways such as the corticospinal tract (CST), the arcuate fasciculus (AF), the inferior and middle fronto-occipital fasciculus (IFOF, MFOF), the frontal aslant tract (FAT), and the superior and inferior longitudinal fasciculus (SLF, ILF). Therefore, we used our standard deterministic algorithm with a fiber assignment by continuous tracking (FACT) (iPlan® Net Cranial 3.0.1 and Brainlab Elements, Brainlab AG, Munich, Germany). The regions of interest (ROI) were chosen based upon anatomy and function. The latter ROIs were based on preoperative navigated transcranial magnetic stimulation mappings (nTMS) [12]. Figure 1 shows the 3-D reconstructions and the visualized steps of the prepared cases.

Here, the special feature for fiber dissection was the opportunity for participants to virtually dissect prepared white matter pathways and objects by the use of a spatial computer and AR goggles (Brainlab AG, Munich, Germany; Magic Leap Inc., Plantation, Florida, USA). In contrast to a virtual reality setup, AR allows to still see the room and other participants while the 3-D objects are projected inside the actual environment.

A 3-D model of the chosen objects can be added into the AR display (Figs. 2 and 3). The pre-selected cases were presented chronologically (Fig. 1). The participants were divided into subgroups of five. Each participant was equipped with an AR goggle headset. In each subgroup of five, one was designated as “master.” The master was equipped with a controller which allowed them to zoom and rotate the model. They were also able to perform virtual dissection of the fiber tracts by peeling away additional layers. In additional, the master’s view was presented via live screen for review by the remainder of the group who were not wearing goggles for that particular session. Figures 2 and 3 show screenshots of the participant view during the application of AR for fiber dissection. Figure 1 shows the step-by-step neuroanatomy dissection revealing function-related fiber tracts.

To evaluate the application of AR for fiber dissection, participants were asked to complete a questionnaire on their normal use of tractography, as well as this simulation experience of using AR goggles and a spatial computer for fiber dissection. They were also asked about possible future applications and suggestions for improvement (Table 2).

Table 2 summarize the overall experience, the usefulness for fiber dissection courses, and the value for educational and clinical purposes of AR for fiber dissection as rated by the participants (Table 2).

After the general evaluation of AR fiber dissection by questions 1–14, we also asked for suggestions for future improvements. The participants’ answers on the open questions 15 (“What should be improved before augmented reality for fiber dissection can be used for virtual fiber dissection?”) and 16 (“What should be improved before augmented reality for fiber dissection can be used for clinical purposes?”) are incorporated in the “Discussion” section of this manuscript.

The described setup in this study currently is the only approach allowing for an augmented reality illustration of real-life cases in an easy and straightforward way not requiring repeated rendering and multistep workflows [11]. While AR was already described extensively in other surgical applications, spatial computing allowing for an accurate and real-time visualization of white matter anatomy was only feasible recently [8], yet as an overlay via microscope integration, neurosurgeons know AR for some years [2, 6].

Overall, the participants in this study of AR fiber tract dissection felt that this was a positive development with the potential for valuable educational and clinical benefits in the future.

As shown by the results of question 1–4, meanwhile, tractography is standardly used for clinical and scientific purposes (Table 2). Moreover, proven by the composition of the participants, various disciplines frequently use this technique for the visualization of white matter pathways, and 92.3% of participants evaluated the presented cases as reflecting their clinical reality. Hence, the present results of the evaluation of AR for fiber dissection as a new technique for visualization are of high relevance for learning, teaching, discussing, and understanding of both the methods for tractography and anatomy.

As was evident from both the individual reactions during the evaluation, as well as the formal questionnaire, the participants were excited and impressed by the technology (questions 5–8, Table 2). Based on the survey responses, the primary applications were felt to be for the educational and training of students, residents, and junior scientists. In addition, the relevance for board-certified neurosurgeons was highly rated, with a median score of 9 of 10 (Table 2).

As expected at the outset of the study, the clinical applicability of AR fiber dissection is restricted at the moment. This was confirmed by the participants. However, 14 of 15 participants said they felt that this AR technique could be developed for clinical settings as a tool for pre-operative preparation of the respective surgeon (Table 2).

Questions 15 and 16 were open questions, allowing participants the opportunity to make suggestions for improvements to the presented technique. Their answers were mainly related to the interaction of the user with the spatial computer, the goggles, and the options of visualization.

Participants recommended that, for future applications, more anatomical structures should be included in the simulation to improve orientation. In fact, this is already available but was not offered as an option for this simulation assessment. The anatomical reconstructions used included pre-selected language pathways, the 3-D reconstruction of the tumor, and the modeling of the patients’ skin (Fig. 1). We also included a reconstruction of the cortex to evaluate this additional request (Fig. 1).

For this study, we used data from patients with brain tumors to demonstrate the virtual dissections in these patients. The participants also suggested that including fiber tracts from healthy subjects would be helpful to facilitate greater understanding of normal anatomy. We feel that a particular strength of the virtual approach is that we are able to use AR to dissect both disease-free normal anatomy and actual tumor cases, improving our experience and understanding of tumor-related changes to normal anatomy in a manner which is not possible in cadaver courses.

The participants also suggested that it would be helpful to develop a more individualized interactivity, which could allow each individual user to control the peeling (virtual dissection), zooming, and rotating of the reconstruction. Likewise, participants asked for development of the ability to separately include and exclude single pathways and to label the fiber tracts with their name and function, either as a default setting or after their identification. During the simulation and assessment session, participants were individually able to walk through the reconstruction. However, functions like zooming, rotating, and the inclusion/exclusion of structures could only be performed by the single master controller, who was the only enabled interactor. We chose this format to facilitate a clearer presentation. However, the currently version of the software already enables all participants to operate individual controllers, so the limitation experienced by the participants was simply due to the setup for the assessment trial.

Another suggestion for future development was to enable the user to interact with the reconstruction with their hands, without needing a controller. Some participants also asked for a wider field-of-view in the goggles, to improve orientation, especially during zooming and rotating of the reconstruction.

In summary, the participants felt that the overall AR system was useful and had good potential for development and a wide range of potential future applications in education and clinical settings. The requested changes are useful for improving the usability of the system in the future.

This report only aimed to evaluate a new AR technique for fiber dissection but not the different methods and algorithms for tractography. However, the latter is frequently discussed, so we sought to evaluate this through questions 12–14. As reflected by the answers and through discussions during the meeting, the actual tractography techniques are also an issue when evaluating techniques for fiber tract visualization. The visualized fiber tracts were rated as anatomically correct by 60% and as incorrect by 40%, which must also be seen as a major limitation of tractography. Most of the participants (66.7%) favored the setting of function-based ROIs and 16.7% of participants did not see a difference between anatomy-based and function-based ROIs. Moreover, while 64.3% of participants felt that DTI FT was sufficient for the presented approach of fiber dissection, 35.7% of participants asked for a more sophisticated approach for future applications. The variation of opinion regarding tractography algorithms and methods could be seen as a limitation of this initial study. However, these issues can be addressed with relative ease and speed, so we do not feel that it is a significant limitation for future development of the technique itself.

Apart from these limitations, here, it must also be mentioned that further options might exist for the visualization of fiber tracts in future, for example smartphone-based solutions.