Segmentation-based registration of ultrasound volumes for glioma resection in image-guided neurosurgery

In brain surgery for tumor removal, neurosurgeons usually plan the intervention on pre-surgical images. The most widely used modality for neurosurgery planning is magnetic resonance imaging [1, 2, 3]. To help physicians with the resection, neuronavigation systems can be used to link preplanning data positions to patient’s head locations. By tracking fiducial markers placed on the patient’s skull and surgical tools, an optical system computes an image-to-patient transformation. Consequently, by pin-pointing an intracranial location, neurosurgeons can obtain the same position in the preplanning images. However, initialization inaccuracies of the neuronavigation system may invalidate the image-to-patient transformation, affecting the quality of these images since the beginning of the resection [4]. Additionally, after resection starts, the preplanning data become even more unreliable due to the brain shift phenomenon: Structures observed in preplanning images don't remain in the same conformation and position during tumor removal [4]. As a consequence, the probability that pathological elements are missed increases, reducing the survival rates of the operated patients [5, 6]. To overcome this problem, intraoperative images can be acquired [7]: They provide an updated view of the ongoing procedure and hence compensate the brain shift effects. A solution is represented by intraoperative magnetic resonance imaging (iMRI) [8]. It is demonstrated to be a good option [9] since its high image quality provides good contrast in anatomical tissue even during the resection [10]. However, the high costs of iMRI and the architectural adaptations required in the operating room seem to prevent this modality from being deployed more widely. A valid alternative is given by intraoperative ultrasound (iUS) [11, 12, 13]. Some authors reported that for certain grades of glioma, iUS is equal or even superior to iMRI in providing good contrast between tumor and adjacent tissues [14, 15]. Moreover, US represents a lower-cost solution compared to MRI. In our work, we focus on intraoperative 3D ultrasound used in neurosurgical procedures.

The more the resection advances, the more the initial acquisition of iUS becomes unreliable due to increased brain shift effects. Therefore, an update of the intraoperative imaging may be required. In [16], the authors acquired US volumetric data in subsequent phases of glioblastoma resections in 19 patients and compared the ability to distinguish tumor from adjacent tissues at three different steps of the procedure. According to their observations, the 3D images acquired after opening the dura, immediately before starting the resection (we indicate this phase as before resection), are highly accurate for delineating tumor tissue. This ability reduces during resection, i.e., after that most of the resection has been performed but with residual tumor, and after resection, i.e., when all the detected residual tumor has been removed. In fact, the resection procedure itself is responsible for creating small air bubbles, debris and blood. Besides this, a blood clotting inducing material¹ commonly used during neurosurgical procedures causes several image artefacts [14, 17]. Successive studies regarding other types of tumor resection confirmed the degradation of image quality in US during resection [18]. Therefore, it would be helpful to combine US images acquired during and after resection with higher-quality data obtained before resection. Such a solution may also be beneficial to improve the registration of intraoperative data with higher-quality preplanning MRI images. In fact, instead of combining directly degraded US data with preplanning imaging, it would be useful to register first the pre-surgical MRI data with US volumes acquired before resection, in which few anatomical modifications occurred. Afterward, intraoperative US data acquired at the first stage of the surgery (which therefore has a higher quality) may be registered to subsequent US acquisitions, and then the preplanning data could be registered to those by utilizing a two-step registration [19]. In this context, neuronavigation systems could be used to co-register intraoperative images acquired at different surgical phases. However, these devices are prone to technical inaccuracies, which affect the registration procedure from the beginning of the resection [4]. Moreover, the available neuronavigation systems usually offer only a rigid registration, which is not sufficient to address anatomical changes caused by brain shift. In our work, we propose a deformable method to improve the registration of US volumes acquired at different stages in brain surgery.

Few solutions have been proposed to improve the US–US registration during tumor resection in neurosurgery. In [20], the authors studied the performance of the entropy-based similarity measures joint entropy (JE), mutual information (MI) and normalized mutual information (NMI) to register ultrasound volumes. They conducted their experiments with two volumes of an US calibration phantom and two volumes of real patients, acquired before the opening the dura mater. Different rigid transformations were applied on each volume, and the target registration error (TRE) was used as evaluation metric. The accuracy of the registration was examined by comparing the induced transformation to move the original images to the deformed ones, with the transformation defined by the entropy-based registration method. In both of the datasets, NMI and MI outperformed JE. In another work [21], the same authors developed a non-rigid registration based on free-form deformations using B-splines and using normalized mutual information as a similarity measure. Two datasets of patients were used, where for each case a US volume was acquired before the opening of the dura, and one after (but prior to start of tumor resection). To assess the quality of the registration, the correlation coefficient was computed within the overlap of both volumes and before and after registration. Furthermore, these authors segmented the volumetric extension of the tumor with an interactive multiscale watershed method and measured the overlap before and after the registration. One limitation of the aforementioned two studies is that no experiment is conducted on volumes acquired at different stages of the surgical procedure, but only before the resection actually begins. In a real scenario, neurosurgeons use intraoperative data to find residual tumor after a first resection, which is conducted after the opening of the dura mater.

One of the first solutions to register US data obtained at subsequent surgical phases utilized an intensity-based registration method to improve the visualization of volumetric US images acquired before and after resection [22]. The results are computed for 16 patients with different grades of brain supratentorial tumor and located in various lobes. Half of the cases were first operations, and half were re-operations. Pre-resection volumes were acquired on the dura mater, or either directly on the cortex (or tumor) or on a dura repair patch. The post-resection ultrasound was used to find any residual tumor. The authors used mutual information as similarity measure for a rigid registration. In the further non-rigid transformation, the correlation coefficient objective function was used. To correctly evaluate their findings, for each of the 16 cases, a neuroradiologist chose 10 corresponding anatomic features across US volumes. The initial mean Euclidean distance of 3.3 mm was reduced to 2.7 mm with a rigid registration, and to 1.7 mm with the non-rigid registration. The quality of the alignment of the pre- and post-resection ultrasound image data was also visually assessed by a neurosurgeon. Afterward, an important contribution to neurosurgical US–US registration came by the release of the BITE dataset [23], in which pre- and post-resection US data are publicly available with relative landmarks to test registration methods. One of the first studies involving BITE dataset came from [17]. The authors proposed an algorithm for non-rigid REgistration of ultraSOUND images (RESOUND) that models the deformation with free-form cubic B-splines. Normalized cross-correlation was chosen as similarity metric, and for optimization, a stochastic descendent method was applied on its derivative. Furthermore, they proposed a method to discard non-corresponding regions between the pre- and post-resection ultrasound volumes. They were able to reduce the initial mTRE from 3.7 to 1.5 mm with a registration average time of 5 s. The same method has been then used in [19]. In a compositional method to register preoperative MRI to post-resection US data, they applied the RESOUND method to register first pre- and post-resection US images. In another solution [24], the authors aimed to improve the RESOUND algorithm. They proposed a symmetric deformation field and an efficient second-order minimization for a better convergence of the method. Moreover, outlier detection to discard non-corresponding regions between volumes is proposed. The BITE mean distance is reduced to 1.5 mm by this method. Recently, another method to register pre- and post-resection US volumes was proposed by [25]. The authors presented a landmark-based registration method for US–US registration in neurosurgery. Based on the results of 3D SIFT algorithm [26], images features were found in image pairs and then used to estimate dense mapping through the images. The authors utilized several datasets to test the validity of this method. A private dataset of nine patients with different types of tumor was acquired, in which 10 anatomical landmarks were selected per case, in both pre- and post-resection volumes: For this set, they were able to reduce the mTRE from 3.25 mm to 1.54 mm. Then, they applied the same method on the BITE dataset and reduced the initial mean error to 1.52 mm. Moreover, they tested their approach on the more recent RESECT dataset [14]. By using the same method on the pre- and post-resection volumes, the mTRE was reduced from 3.55 to 1.49 mm.

Our solution proposes a segmentation-based registration approach to register US volumes acquired at different stages of neurosurgical procedures and compensate brain shift. A few approaches already applied segmentation methods on US data to then register MRI and iUS [27, 28]. Our solution represents the first segmentation-based method aimed at US–US volumes registration. Our approach includes a deep-learning-based method, which automatically segments anatomical structures in subsequent US acquisitions. We chose to segment the hyperechogenic structures of the sulci and falx cerebri, which remain visible during the resection and thus represent good corresponding elements for further registration. In the following step, parametric and nonparametric methods use the generated masks to register US volumes acquired at different surgical stages. Our solution reduces the initial mTRE for US volumes acquired at subsequent acquisitions in both RESECT and BITE datasets.

The manual annotations, even if sparse, are good enough to train the CNN model to segment the anatomical structures of interest, as shown by the DICE coefficients in Table 2. Moreover, Fig. 4 shows that automatically generated segmentations are more precise than the manual annotations, with a better contours refinement and larger number of identified structures. However, some pathological tissues are wrongly segmented by our method (see Fig. 4d). This may be due to the fact that in US data the glioma of grade II appears as hyperechogenic structures, with an intensity similar to the elements of interest. In future work, we could consider to separately segment pathological tissue and then exclude it during registration. A similar consideration can be made for the resection cavities in volumes acquired during and after resection, which appear as bright as sulci and are wrongly segmented by the proposed method (Fig. 6). Furthermore, from a qualitative comparison with other segmentation methods involving US data, we can highlight some advances of our approach. First of all, with respect to [27, 33], a higher number of anatomical structures are included in our manual annotations. Therefore, the potential range of clinical scenarios in which our method could be applied might be wider. Secondly, a trained neurosurgeon has clinically validated the manual annotations (Table 1). This is not the case for other segmentation-based methods [30, 28], in which no precise rating of the manual masks is provided.

The second important contribution of this work is the registration of US volumes acquired at different surgical stages. First of all, the segmentation method gives evidence of being able to generate meaningful masks to guide the registration task. In fact, the proposed registration method is able to reduce the mTREs of three sets of volumes from two different datasets (Table 4, 5, 6) by using the corresponding anatomical structures previously segmented. From numerical and visual results, we can notice that even if minor corresponding segmented elements are missing in volume pairs, our method is able to reduce the initial registration errors. However, in the case of volumes acquired after removal, resection cavities may be segmented by our method due to their intensity similar to the sulci. Consequently, the mTRE in Table 5 is reduced less with respect to Table 4, since these structures have no or few corresponding elements in volumes acquired in previous steps. This is a limiting factor of our registration method, which is completely based on the masks generated by our trained model. In future work, we could try to segment such structures and exclude them during the registration. Only another work [25] focused on the registration of US volumes acquired before and after resection of RESECT dataset (Table 5). The mTRE obtained by the aforementioned approach is better than our method, which, however, is the first one to provide results for the volumes obtained before and during resection of the RESECT dataset. In this set of volumes, our registration performs quite well, reducing the initial mTRE to 1.36 mm.

Regarding the BITE dataset, our algorithm improves the initial registration (see Table 6), proving not to be over-tuned on RESECT dataset. Note that in contrast to our approach, all other methods compared in Table 6 have only been tested on the BITE dataset. Thus, the results may be over-tuned on this limited set of volumes and the approaches could lack generalization. On the contrary, our solution is the second one after [25] to propose a more generalized method, which has been tested on registering the volumes of both RESECT and BITE datasets. Therefore, our method is validated on a larger number of US acquisitions, providing a more generalized solution. Nevertheless, there might be some reasons why a few other approaches have smaller average mTREs for the BITE dataset (last section of Table 6). First of all, a numerical impacting factor for our results comes from case 12, where the TRE increases from 10.54 up to 11.08 mm, affecting the overall result. The capture range of our method is too low to register this volumes pair, which has a very large initial misalignment. In future work, we could improve the results by performing an initial registration which could increase the capture range of our method. Moreover, the limited improvement obtained by our method might be due to the lower quality of the BITE dataset with respect to the RESECT volumes, which is used for training the segmentation approach. Since our registration method is based on the generated masks, it is almost impossible for the registration method to converge to the right solution if the segmented masks are not accurate enough.

The total time required by each task of our method is visible in Table 3: The segmentation step requires 1.28 s and 28.55 s (before/during) and 29.40 s (during/after) that are needed to register the generated 3D masks. In addition to this, we should also take into account the time to reconstruct a 3D US volumes from 2D images, which is of a few seconds [14]. Considering the increase in the brain shift over the time and the average duration of a neurosurgical procedure [34], our algorithm is fast enough to register US volumes and therefore provides a meaningful solution for brain shift. Nevertheless, in future work we could optimize our algorithm in order to speed up the registration step.

To the best of our knowledge, our solution is the first one to propose a segmentation-based registration method which registers US volumes acquired at different surgical stages. Our approach provides some important contributions. Regarding the segmentation step, a model based on a 3D U-Net has been trained on a large number of anatomical structures, whose manual annotations have been validated by an experienced neurosurgeon. Even if the training is performed on a sparse set of annotations, the proposed solution is able to automatically segment hyperechogenic elements in US volumes. Moreover, the segmented anatomical structures prove to be meaningful elements which can guide the registration of US volumes acquired in the neurosurgical context. In fact, for two different datasets of US volumes acquired at different surgical stages, the initial mTREs are correctly reduced, demonstrating that our solution is not over-tuned for a specific dataset. Moreover, our work is the first one to be applied also on the US volumes of RESECT dataset acquired during resection, for which no previous work has been published.