How molecular imaging will enable robotic precision surgery

In the whole preoperative route of patient selection, surgical planning, and prediction, AI methods are starting to play an increasingly relevant role, assisting the physicians in delivering their best performance. So far, such AI methods have mostly focussed on the aspect of surgical planning, providing an automatic segmentation of tumors and organs at risk using both patient imaging and metadata. For such segmentation models, deep learning methods have predominantly taken over, often based on convolutional neural networks (CNNs). For image processing, such CNNs have been shown to overperform humans in classification and detection methods (e.g., [36, 37]). They are increasingly being used for tasks like multi-organ segmentation [38]. When dealing with imaging data, a CNN consists of stacked nonlinear convolutional layers that can extract features of high-level (shapes, content) and low-level (details, texture) from images. The network architectures reduce dimensionality first to extract high-level features in a so-called bottleneck representation of the image (see Fig. 2). From there, these features are further processed, where so-called skip connections contribute low-level information from early layers. In the end, the final layers convert the features into probabilities, for example, in the scenario of organ segmentation. AI organ segmentation can even be further optimized by feeding the neural network with relevant patient metadata, such as patient age, weight, clinical parameters, and previous treatment strategies (e.g., [39]).

To indicate organs at risk, such automatic multi-organ segmentations have been successfully tackled using CT scans [40‐42] or MR scans [43], even in case they were cropped [44]. The higher the accuracy of these segmentation models (often expressed as a higher “Dice score”), the lower the need for an expert physician to correct the automatic segmentation. Interestingly, combining such anatomy-based segmentations with molecular imaging (i.e., PET or SPECT) can further improve the AI results. A network having both inputs can learn the physiological uptake of different organs and enhance the segmentation (e.g., improved definition of (metastatic) tumor nodules). First approaches that use both anatomical and molecular images to enhance segmentation have been undertaken with PET/CT in automatic tumor segmentation in lung cancer [45‐47], head and neck tumors [48, 49], and non-Hodgkin lymphoma [50]. Here, a combination of the tumor detection task and the multi-organ segmentation can be advantageous as this mimics both nuclear medicine physicians and radiologists [51].

Next to assistance in surgical planning, AI methods such as deep learning might improve initial patient selection and even enable predicting treatment outcomes. The latter could entail the detection of abnormal anatomy during surgery, the occurrence of side effects, the duration of the procedure, or the length of the postoperative stay. Initial works have been published for the treatment of esophageal [52] and oropharyngeal cancer [53] using PET images as input. Despite the prediction being here the response to chemotherapy or radiotherapy, technically, the same approach can be used to train a system for surgical prediction by changing the training dataset to include surgical outcome information.

This is the most intuitive and straightforward option already proposed at the end of the 1990s [62]. First, particular landmarks are selected on the preoperative images and subsequently tipped with a calibrated tracked instrument in situ. The resulting point correspondences between preoperative images and tracking coordinates of the anatomical landmarks can be processed to yield the desired registration. The more anatomic landmarks used, the better the quality of registration. In practical terms, this is challenging and, unfortunately, often results in poor registrations. On the other hand, in robotic surgery, the robotic instruments are tracked using the propriosensory and can be used to tip the landmarks making an external tracking system unnecessary.

Several groups have proposed methods for registering the laparoscopic video with renderings of the preoperatively acquired images [63, 64], either using a 2D or 3D laparoscope. Such approaches rely on the fact that preoperative CT or MRI depicts the anatomy at the time of surgery reasonably well (i.e., no significant resections or strong tissue deformations have taken place). Besides, the lack of clearly distinguishable landmarks in the laparoscopic images makes this registration approach challenging. However, since the robot’s propriosensors track the laparoscope’s pose, no additional tracking hardware is required here. Also, with the continuous development of AI-supported surgical scene recognition, it is expected that there is a lot to be won in the accuracy of this method (e.g., [65, 66]).

While registering organ surfaces to preoperative imaging is feasible using the laparoscopic video feed, deep-lying structures may not be properly registered. In a robotic setup, this could be achieved with a positionally tracked ultrasound (US) probe. More specifically, the probe can be either endoluminal ultrasound (i.e., transrectal or transvaginal ultrasound, tracked via mechanics or a second robot [67]) or DROP-IN ultrasound (tracked via laparoscopic-video [68]). The most important advantage is the possibility of using in-depth information to improve registration results. However, it is limited to specific applications and organs where US is feasible (such as the prostate, kidney, liver).

From the methods mentioned above, we believe that evolutions of laparoscopic video-based registration, and to a lesser extent, ultrasound registration, combined with mechanical tracking of the robotic platform, will be the method of choice in robotic surgery in the years to come. This is because they can be integrated into the robotic suite and thus require less external hardware.

With the proper registrations in place, surgical navigation has the potential to translate the wealth of preoperative information (e.g., multi-modal scans, target definition, most efficient route, critical organ segmentations) to the operating room, directly visualized in the surgical robot console. Using molecular imaging modalities (i.e., PET and SPECT), this has primarily been oncological data, providing guidance towards, e.g., (metastatic) lymph nodes defined before surgery [18].

Unfortunately, the use of preoperative images for surgical navigation is challenging in a soft tissue environment, where organs deform and where surgery itself changes the anatomy dramatically [69]. Efforts have been made to cope with these inaccuracies by regularly reapplying registration of the preoperative images to the current surgical situation using C-arms and ultrasound (e.g., [70, 71]), as well as tracked gamma probes [72] or gamma cameras [73].

Alternatively, to cope with soft tissue–induced deformation, it has also been proposed to navigate intraoperative molecular detection modalities, basically using them as pointers within the navigation workflow. In this way, navigation is useful for orientation and rough localization in the patient. Simultaneously, the real-time feedback of the molecular detection modality allows for correction and confirmation of the actual tissue targets once close enough. This concept has been shown in vivo using gamma probes (e.g., [74]) and fluorescence cameras [75‐77], and preclinically even using robotic DROP-IN gamma probes [78] and robotic fluorescence cameras [33]. In our view, with the current status of this field, intraoperative imaging (e.g., ultrasound, freehand SPECT, fluorescence laparoscopy) remains indispensable in soft tissue environments.

Since the 2000s, it has been possible to stream a digital full image viewer showing either 2D images or 3D images in three planes, often visualized as a 3D render in a virtual environment (i.e., virtual reality (VR); e.g., [79]) where a priori segmented organs and structures can be highlighted (Fig. 4A). This applies not only to preoperative images but also to intraoperative information, such as the read-out of intraoperative ultrasound (US), displayed in relation to the surgical instruments [67], giving the surgeon a virtual context during surgery (Fig. 4B).

The next relevant improvement in this field came with the introduction of augmented reality (AR) into the operating room. Unlike the virtual environment provided with VR, AR is defined as an approach of visualization where “invisible” information (e.g., pre- or intraoperatively obtained imaging findings) are overlaid on the view of the real environment, providing immediate context with the actual patient (Fig. 4D and E). This concept was introduced in orthopedic and brain surgery as early as the 1990s [81] using CT/MRI and even in combination with microscopic imaging [82]. However, it took until 2000 for AR to make it to clinical applications [61].

A key challenge in AR is achieving a visualization that provides additional information without diminishing the camera’s information or giving a wrong perception. This limitation of AR has strongly limited its adoption. Different approaches have been proposed to improve depth perception (Fig. 5A–D), like the use of virtual mirrors [83], curvature-dependent transparency [84], virtual shadows [85], or object subtraction [86]. With the advances in machine learning, in particular, object subtraction has shown excellent performance for the detection of instruments in laparoscopy videos [87], opening a path towards visualization approaches where only the relevant information is overlaid to the surgeon [88] (Fig. 5E and F).

Gamma counting has been the most used surgical modality within RGS [17, 101]. Moving towards minimal-invasive procedures, long “laparoscopic” gamma probes are routinely used in sentinel lymph node biopsies (SLNB) in prostate cancer [102] and cervical cancer [103], but also in several other indications. Unfortunately, in the laparoscopic setting, their use is complicated by the limited movability. Being a rigid instrument, placement of laparoscopic gamma probes during tracing is restricted due to the limited range of motion available when working through a trocar [33]. This is further complicated in the robotic setting since the surgeon is no longer located in the sterile field. To improve positioning and at the same time regain autonomy for the surgeon and increase the range of motion available during tracing, a small-sized and tethered drop-in gamma probe was introduced [21, 33], a technology that is in line with the DROP-IN ultrasound technology that is used in robotic surgery [104]. Evaluations in prostate cancer SLNB indeed indicate an improved sentinel node detection rate for the DROP-IN (100%) versus laparoscopic gamma probe (76%) [22]. Recent studies indicate the DROP-IN also facilitates receptor-targeted surgery in the robotic setting (e.g., PSMA RGS [28]). These DROP-IN concepts applied to gamma probes in robotic surgery can be readily transferred to beta-probes [105] and as such allow for intraoperative detection of typical “PET-isotopes” (beta plus, e.g., ¹⁸F or ⁶⁸Ga) and even opens the door for “therapeutic isotopes” (beta minus, e.g., ⁹⁰Y).

Portable gamma cameras can form a 2D image of the tracer distribution directly in the operating room [106‐108] and probably even precede the use of gamma probes [109]. Initially developed for open surgery, their application has been shown in laparoscopic surgery as well, including prostate cancer, renal cell carcinoma, and testicular cancer SLNB procedures [110]. Although the gamma camera placement is more complicated during laparoscopy, its use helped verify the successful removal of the surgical targets. Providing only a 2D image, depth estimation is challenging, requiring frequent repositioning of the camera with different angles around the patient. To correlate the gamma image and the instruments in the field, radioactive markers on the instrument, e.g., a ¹²⁵I-seed on the tip of them and a dual-isotope mode on the camera, have been proposed with improved usability [111]. We, ourselves, have tried to use such portable gamma cameras during robot-assisted surgery. Unfortunately, this proved complex, resulting in frequent collisions between the robot and the camera. Usage of such cameras during robotic surgery is most likely only valuable when directly integrated into one of the robotic arms, or possibly the patient bed [109]—a solution that unfortunately has not made it to product but is technically feasible. Alternatively, truly laparoscopic gamma camera prototypes have been evaluated in phantoms [112]. In a similar preclinical setting, other research groups have demonstrated a beta-cameras’ potential as a future application in (robot-assisted) laparoscopic RGS [18].

3D freehand imaging of radiopharmaceuticals is a method whereby a tracking system is used to determine the position and orientation of a nuclear detector (e.g., gamma probe or gamma camera) within the operating room, allowing for a 3D reconstruction based on the signal collected at these various positions [113]. The most established RGS form of this, and the only one being applied in vivo yet, is freehand SPECT [97]. Combining this technology with a tracked US device, intraoperative freehand SPECT/US has also made it to patients [114‐117]. Preclinical research has also been performed to investigate freehand imaging of beta emissions [95, 118], high-energy gamma emissions [119], or even freehand PET imaging using a coincidence or time-of-flight principle with multiple detectors [120, 121]. While freehand SPECT is mainly used for open surgery procedures (e.g., SLNB or even receptor-targeted procedures [18]), this technology has also been translated to laparoscopic procedures, e.g., SLNB in gynecology and urology [77, 122] as well as radio-guided occult lesion localization in the lung [123]. However, the laparoscopic setting application is more challenging, where the limited movement of a laparoscopic probe restricts signal acquisition, and with that, reduces the quality of the freehand SPECT scan. Therefore, first steps are taken to translate the freehand imaging method towards robotic surgery using a tracked DROP-IN gamma probe [78, 124, 125]. Robotic SPECT has also been performed in a slightly different setting, using a robotic arm to autonomously create scans [126, 127]. These works have even extended to simultaneously acquire a cone-beam CT towards intraoperative SPECT/CT [128, 129]. Although the latter has not yet been tested in the surgical setting, one needs little imagination to hypothesize that such scans could be performed using robotic devices in the future. Interestingly, studies are being conducted to investigate if freehand SPECT can provide a reliable surgical roadmap (e.g., [130]). Some even suggest that freehand SPECT could potentially wholly replace the preoperative SPECT/CT. In line with this, no difference in performance was observed for 50 oral cancer SN procedures, where the surgeons were blinded for SPECT/CT [131].

The most common fluorescence imaging type is single-band, meaning that it uses a single band-pass filter to depict one sole fluorescent “color.” In the clinic, fluorescence detection of the visible dye fluorescein and near-infrared (NIR) dye indocyanine green (ICG) is currently a routinely applied tool for angiographic purposes. An increasing number of trials are being conducted in the area of lymphatic mapping (e.g., [134]), perfusion-based tumor resection (e.g., [135]), and receptor-targeted imaging (e.g., [136]). The main advantage of fluorescence is that it provides real-time visual feedback concerning the tracer uptake during surgery, especially useful to confirm successful lesion localization during excision. Interestingly, while fluorescence images satisfy the surgeon’s demand for optical feedback, being based on light, it is inherently unsuited to visualize lesions located more than a couple of millimeters in tissue (i.e., < 1 cm depth [137]), something that is possible with for example nuclear medicine or radiological techniques. This was recently confirmed by Meershoek et al., demonstrating that in 52% of the patients, lesions were missed during robotic SLN procedures in prostate cancer if fluorescence imaging was used alone, underlining the need for additional technologies such as RGS [21]. This renders fluorescence imaging mostly useful for superficial applications and visual confirmation of successful target localization.

Every fluorescent dye has its absorption and emission spectrum. By wisely choosing non-conflicting spectra of the individual dyes, several (targeted) fluorescent tracers can be depicted simultaneously using so-called multi-wavelength (also known as multispectral or multicolor) fluorescence imaging. First in-human studies illustrate this has the potential to separate different anatomical structures [138], a concept that could be especially interesting to improve the balance between surgically induced cure and side effects (e.g., decrease damage to healthy nerves and lymphatics [139, 140]). While most proof-of-concept studies have evaluated the use in microscopic neurosurgery (e.g., in glioblastoma [141]), there are also examples in laparoscopic surgery (e.g., parathyroid surgery [142], bladder cancer [143], liver cancer [144], and gynecology [145], or even robot-assisted laparoscopic surgery (i.e., prostate cancer [24])). Contrary to popular belief that only NIR-I (700–900 nm) is useful for fluorescence imaging, based on the relatively low tissue-induced absorption of light at these wavelengths, these studies do not visualize that much of a difference in intensity when used in a surgical environment, justifying the use of different wavelengths as well [146]. Even deeper NIR (e.g., 1000 to 1700 nm) has been suggested, theoretically providing higher resolution due to lower light scattering at these wavelengths [147]. However, limiting to the whole approach of multi-wavelength fluorescence is that only a small amount of fluorescent tracers is currently clinically approved (i.e., ICG, fluorescein, methylene blue, PpIX⁵-ALA/HAL [138]).

Since detection techniques have their strengths and limitations, hybrid imaging (also referred to as dual-modal or bimodal imaging) combines the imaging properties of different modalities into a single technique, providing “best-of-both-worlds”. This is not only relevant for modalities that combine anatomical information with molecular information, but even for modalities that combine two types of molecular imaging, where these might provide guidance during different parts of the image-guided surgery process. This is the case for the currently trending topic of hybrid fluorescence-radio-guided surgery that combines both fluorescent and nuclear signatures into a single tracer [15, 148]. This concept was first introduced in the clinic in 2010 using ICG-99mTc-HSA nanocolloid [149] for SLNB in various forms of cancer (e.g., prostate [150], penile [151], head-and-neck [152], cervical [153], and breast cancer [154], as well as melanoma [155]). Currently, it has been applied in > 1500 patients, of which some were already treated in the robotic setting. Following the success in SLNB, this hybrid concept is now increasingly adopted for research in receptor-targeted tracers (e.g., neuroendocrine tumors using Cy5-¹¹¹In-DTPA-Tyr3-octreotate [156], prostate cancer using PSMA I&F [157] and breast cancer using ¹¹¹In-DTPA-trastuzumab IRDye800 (Wang2015)). There, some have recently even entered first-in-human studies (e.g., clear cell renal cell carcinoma using ¹¹¹In-DOTA-girentuximab IRDye800 [30] or prostate cancer using ⁶⁴Ga-PSMA-914 [29]).

Superparamagnetic iron-oxide nanoparticles (SPIONs) were introduced in 2013 as an alternative to radiolabelled colloids and blue dye in sLNB [158]. Clinically, their use has been restricted to SLNB (e.g., breast [159], vulva [160], prostate [161], penis cancer [162], and also melanoma [163]), or occult-lesion localization even in a laparoscopic setup [164]. As with radioactive tracers, SPIONs do allow for preoperative imaging and mapping of the sentinel nodes using MRI, where they usually show a black taint such that a degree of bare-eye guidance is possible. So far, the limitations in our view are in the lack of miniaturized detectors that can work in combination with surgical robots and slow tracer clearance from the body [165]. Interestingly, more and more groups are nevertheless actively working on bringing up tumor-specific SPIONs [166], which might offer alternatives for the future.

The concept of using pulsed light in tissue while imaging the resulting ultrasonic vibrations to discern differences in tissue-absorption was introduced in 2007 for diagnosing breast cancer [167, 168]. Along with improvements of the technology itself, a significant breakthrough was achieved by applying fluorophores to increase tissue contrast in 2012 [169]. This technique, currently known as multispectral optoacoustic tomography (MSOT), has the main advantage of an improved tissue-penetration (up to 5 cm [92]) with comparison to fluorescence imaging and the fact that the resulting images are tomographic. Applications have been reported for lymphatic imaging [170], breast cancer detection [171], characterization of non-melanoma skin cancers [172], among others. Current research focuses on miniaturizing the devices to make them suitable for laparoscopic surgery and developing AI-powered signal processing algorithms to improve the interpretability generated images (e.g., [173, 174]).

One of the promising technologies in the molecular imaging realm is Raman spectrometry, a technique that is based on the fact that different molecules, and with that different tissue types, have different Raman scattering spectra. This modality does not necessarily require a tracer. Its first applications in the operating room were as early as 2006 to evaluate resection margins in breast cancer [175]. Raman spectrometry can be applied to various indications like the detection of parathyroid adenoma [176] and the evaluation of lymph node metastasis in breast cancer [177], as well as the definition of resection margins in oral cancer [178, 179], glioma [180, 181], follicular thyroid cancer [182], and prostate cancer [94]. Given that Raman spectrometers can be easily miniaturized, porting them to a robotic setup is easy, as reported in the last reference. Like in MSOT, AI also has contributed significantly in particular by coping for illumination interference [183, 184] opening thus alternatives to generalize the use of this technology [185] and even beat fluorescence imaging in particular indications [186]. Using similar optical settings as with fluorescence imaging, tissue penetration is expected to be low (< 1 cm) making this a superficial technology.