Holographic patient tracking after bed movement for augmented reality neuronavigation using a head-mounted display
DOI: 10.1007/s00701-021-04707-4
© The Author(s) 2021
Received: 26 October 2020
Accepted: 4 January 2021
Published: 29 January 2021
Abstract
Background
Holographic neuronavigation has several potential advantages compared to conventional neuronavigation systems. We present the first report of a holographic neuronavigation system with patient-to-image registration and patient tracking with a reference array using an augmented reality head-mounted display (AR-HMD).
Methods
Three patients undergoing an intracranial neurosurgical procedure were included in this pilot study. The relevant anatomy was first segmented in 3D and then uploaded as holographic scene in our custom neuronavigation software. Registration was performed using point-based matching using anatomical landmarks. We measured the fiducial registration error (FRE) as the outcome measure for registration accuracy. A custom-made reference array with QR codes was integrated in the neurosurgical setup and used for patient tracking after bed movement.
Results
Six registrations were performed with a mean FRE of 8.5 mm. Patient tracking was achieved with no visual difference between the registration before and after movement.
Conclusions
This first report shows a proof of principle of intraoperative patient tracking using a standalone holographic neuronavigation system. The navigation accuracy should be further optimized to be clinically applicable. However, it is likely that this technology will be incorporated in future neurosurgical workflows because the system improves spatial anatomical understanding for the surgeon.
Keywords
Patient tracking Augmented reality Head-mounted displayAbbreviations
- AR:
-
augmented reality
CN
conventional neuronavigation
FRE
fiducial registration error
HMD
head-mounted display
IR
infrared
TRE
target registration error
Introduction
Infrared (IR) navigation systems are broadly used in neurosurgery. These systems guide the surgeon by providing anatomical information about the surgical field and surroundings and allow for a minimal and direct approach to the surgical target [2, 10].
Neuronavigation systems work on two main principles: patient-to-image registration and patient tracking after bed movement. For patient-to-image registration, a transformation matrix has to be calculated between image-space and physical space in order for them to overlap. This is usually performed through surface matching or point-based matching, where fiducials in physical-space and image-space are matched using an iterative closest point algorithm. Patient tracking is the alteration of image space after patient movement in physical space, with the purpose of correcting registration.
Augmented reality is a technology that superimposes a virtual image into the user’s view of the real world. AR-HMDs have seen a major development over the recent years and interest has been shown especially in neurosurgery [3, 12]. As neuronavigation systems, AR-HMDs have two advantages compared to conventional neuronavigation (CN): First, anatomy is shown in a stereoscopic 3D manner so the surgeon does not need to translate the anatomy from a 2-D external screen to the 3-D surgical field, theoretically reducing the chance of interpretation error and leaving more room mentally to concentrate on other surgical tasks. Second, images can be superimposed directly onto the surgical work field, diminishing attention shifts which makes surgical tasks more efficient [4]. To use an AR-HMD as a neuronavigation system, two main principles need to be developed: patient-to-image registration and patient tracking after bed movement. The first has been extensively investigated and several registration methods have been suggested [5–7]. The latter has, to the best of our knowledge, not yet been investigated within the neurosurgical field.
We designed a reference array for use with an AR-HMD to reposition the holograms after registration according to the movements of the head in the Mayfield. Furthermore, we improved our previously published point-based registration method by optimizing the pointer for better accuracy [13]. We present a proof-of-concept on three patients undergoing a neurosurgical procedure to illustrate the complete workflow of a neuronavigation system using an AR-HMD.
Methods
Inclusion
Patients were included if they were admitted to the study center for an intracranial neurosurgical procedure with the use of standard IR neuronavigation and scheduled within the period of 24–28 February 2020. All data was collected from our institutional ongoing prospective patient registry as approved by our local ethics committee (KEK 2017–01120). All patients were informed on the purpose and the course of action for the study preoperatively, and all provided informed consent prior to inclusion. No diagnostic or treatment decisions were made based on the holographic navigation system, and registration and measurements were performed simultaneously with the setup of standard neuronavigation. Standard neuronavigation was used for the remaining procedure.
Segmentation
Software
We programmed our dedicated holographic navigation software in a real-time 3D development platform (Unity, Unity Technologies, San Francisco, USA) using C++ and C# and an integrated development environment (Visual Studio, Microsoft, Redmond, USA). We used an image recognition library (Vuforia, PTC, Boston, USA) to integrate marker tracking. Using the software, the surgeon can summon, manipulate, and register patient holograms that are directly superimposed over the real field-of-view. The software and AR-HMD work completely independent without the use of external devices and is not connected to the existing CN system in any way.
Registration
Reference array
Results
Patient 1
Patient 2
Patient 3
Discussion
In this study, we presented three neurosurgical cases where we showed a proof-of-concept of our custom AR neuronavigation system. With this system, initial registration can be corrected for bed movements using a reference array. We measured the accuracy of initial point-based registration using FRE, and we evaluated the accuracy of registration correction after bed movement. We measured a mean FRE for initial registration of 8.55 mm. Patient tracking was successfully conducted in all cases.
Several indexes can be used to evaluate navigation accuracy. The target registration error (TRE) is considered of most interest to the surgeon and is often used to evaluate the accuracy of IR neuronavigation systems. When using a point-based registration method, which is most frequently used, FRE can be calculated automatically after registration. Although a specific FRE is not correlated with a specific TRE and therefore is not advised as an index to illustrate the accuracy of a specific registration [1], as an evaluation for the accuracy of a new neuronavigation system, it is a good indicator for navigation accuracy. Furthermore, FRE was used since there is still not a golden standard for measuring the TRE intraoperatively in a holographic scene. This is also why we did not collect quantitative data on the accuracy of the reference array. We acknowledge that determining the accuracy of the registration correction using visual observation is subjected to observation bias. However, effort is put into creating such a method so in the future we can conduct measurements in a lab environment and collect quantitative data. Furthermore, to minimize disruption of the OR workflow, we conducted only 2 registrations per patient with 2 subsequent FRE measurements. Considering the current variability between different registrations on the same patient, a larger sample of registrations per patient would be preferred for future experiments.
Although the technical workflow functioned as desired, the accuracy of initial registration was not adherent to the standard necessary for the clinical use of a neuronavigation system. Moreover, we recognized several technical limitations when using the software.
CN uses retroreflective spheres that can be tracked by an IR camera. In standard clinical practice, initial registration is conducted using an IR tracked probe on either anatomical landmarks, fiducial stickers, or surface matching. The accuracy of registration in IR-based neuronavigation is highly dependent on the method of registration: anatomical landmarks and surface matching are not as accurate as registration with fiducial adhesives [8, 9, 11, 14, 15]. For all methods of registration, a reference array with retroreflective spheres fixated to the head clamp or patient’s head is used to correct registration after intraoperative bed movement.
The concept of patient tracking for image-space is now implemented in our previously designed holographic neuronavigation system with the use of a 3D printed reference array with visual markers. To the best of our knowledge, we are the first presenting such a system. The system has several advantages when compared to CN. First, a stereoscopic 3-D hologram is directly superimposed on the patient, allowing a surgeon to directly assess the internal anatomy in relation to the real patient. Second, it provides a direct superimposition of the hologram in the working field during the macroscopic part of the procedure which in CN can only be achieved while using the microscope which is not preferred during that stage. This has ergonomic advantages when compared to a separate side-screen, improving the efficiency of the workflow and diminishing attention shifts.
Several problems that we described when using the previous version of our neuronavigation software still persisted. Despite improvements on probe design, initial registration was relatively inaccurate. We expect the usage of anatomical landmarks for registration to be a major contributor to this problem. Anatomical landmarks were used since this is the clinical standard in our department for registration when using an IR neuronavigation system, and as a proof-of-concept for the software, extra imaging for using adhesive fiducials was deemed too much of a burden for the patients. Anatomical landmarks account for a relatively large fiducial localization error which is the accuracy in which the fiducial points can be localized. Therefore, the point-clouds will not match as accurately as compared when using fiducial stickers or bone screws where the fiducials can be more accurately targeted. Moreover, probe tracking was inaccurate when registering anatomical points under suboptimal lighting or behind the head clamp. The latter we solved for the third patient by choosing more optimally placed anatomical landmarks. This way, the anatomical landmarks were easier to reach and the navigation accuracy improved. To improve registration further, in the future, we aim to use adhesive fiducials when possible. Furthermore, we are exploring methods similar to mask registration where the patient’s head can move freely during navigation. For this, a single marker will be adhered to the patient’s head which then will be used as reference point for the registration. This will allow constant tracking and correction of the holographic scene.
Problems with hologram stability were also still present. When large objects moved within the field-of-view of the AR-HMD before the hologram was matched on the reference array, the device could lose its spatial orientation. This could lead to severe drifting of the hologram. We plan to improve virtual “anchoring” of the hologram to recognizable geometrical static shapes in the real environment. Additionally, we expect future AR-HMDs to improve spatial mapping, which could lead to better hologram stability.
During the procedure, some functional difficulties regarding the usage of the device were reported. It was often difficult to find an optimal screen brightness and hologram opacity to allow the user to see both the hologram and patient clearly. This may cause difficulties for the surgeon when wearing the AR-HMD while marking the incision or performing a technical maneuver. This issue may be resolved by using more translucent hologram shaders or by moving the hologram to a floating position above the work field when performing a complex action.
Conclusion
This early report shows a proof of principle of intraoperative patient tracking using a standalone holographic neuronavigation system. The navigation accuracy, hologram stability, and functional difficulties should be further optimized for the system to be clinically applicable. However, it is very likely that this 3-D technology will be incorporated in future neurosurgical workflows because of the distinct advantages it provides for neurosurgeons.
Compliance with ethical standards
Conflict of interest
T.P.C. van Doormaal is founder and CMO of AugmedIT bv, a start-up company that develops Augmented Reality tools for surgeons. All other authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.Additional informed consent was obtained from all individual participants for whom identifying information is included in this article.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.