Continuous lung region segmentation from endoscopic images for intra-operative navigation
Introduction
In recent decades, computer-based intraoperative navigation has played an increasingly important role in minimally invasive surgeries. However, intraoperative ultrasound, Computed tomography (CT), C-Arm, and related technologies are expensive and potentially time-consuming in endoscopic surgeries. Therefore, current research efforts are focused on 3D intraoperative organ models using preoperative CT volume data [1], [2], [3], [4]. A problem is that for organs such as the lungs, which deform considerably during surgery, it is difficult to obtain reliable intraoperative 3D information based on the preoperative CT data. Thus, image processing methods that analyze intraoperative endoscopic images have been examined for their utility in providing intraoperative information on the organ of interest. For example, Wang et al. reconstructed the 3D shapes of organs by tracking the object in successive endoscopic images [5]. Similarly, the method of Lin et al. provided valuable guidance for computer-assisted minimally invasive surgery by tracking the distinctive features of sequential endoscopic images [6]. Compared with preoperative CT data, intraoperative endoscopic images are more intuitive and timely. Therefore, endoscope-based intelligent navigation system, especially those combined with preoperative 3D models, are particularly promising. Nonetheless, in practical terms, the performance of these systems is often compromised by noise from medical equipment and the background. If the lung region of each endoscopic image could be segmented accurately, the influence of noise could be avoided. Furthermore, the boundary of the segmentation can provide valuable information on the intraoperative shape of the lungs. In general, lung region segmentation is both important and necessary in computer-assisted intraoperative navigation.
Till now many successful approaches have been proposed for image segmentation, including automatic and user-interactive methods [7]. Considerable success in the image processing field has been achieved with automatic segmentation methods such as K-means, Region-growing, Thresholding, and Level-set [7], [8], [9], [10]. However, while automatic techniques are convenient, their performance in clinical applications is difficult to guarantee [11]. User-interactive segmentation algorithms such as Graph-cut, GrabCut, GrowCut, and Snakes require users to provide seeds of the background and foreground [12], [13], [14], [15], [16], and their performances may thus be more reliable than fully automatic methods [10]. These segmentation techniques are also widely used to process medical images. For example, Yoon et al. applied Snakes to endoscopic image segmentation [15], [16]. However, during thoracoscopic surgery, image blur often occurs, and only a partial lung region is shown on the screen. Under these circumstances, the performance of Snakes is highly degraded because the method is sensitive to noise and the initial boundaries [10]. The Chan and Vese model which is an improvement of Level-set, was introduced also for endoscopic image segmentation [8]. This model depends on gradient information. Nevertheless, in lung endoscopic images, the gradient of the boundary between foreground and background is not especially distinct. The use of color components to segment endoscopic images fails if the color difference between the organ and background is small [17], [18]. Nosrati et al. achieved multi-organ segmentation based on preoperative 3D model [19], but this approach is difficult to apply to lung segmentation because the intraoperative shape of the lung differs from its preoperative one. As a successful user-interactive segmentation technique, GrabCut and its variants were introduced to process endoscopic images [11], [20]. Despite the high performance of these methods, they require user interaction during segmentation. For example, although our previous GrabCut-based algorithm can track lung parts during endoscopic video filming, it requires users to provide a precise manual drawing of the initial boundary, which is rather time-consuming [21].
Thus, current researches can not guarantee both the efficiency and accuracy required in clinical intraoperative navigation systems. To address this issue, we propose a novel algorithm for sequentially segmenting lung region from endoscopic images. The proposed algorithm includes both segmentation and tracking parts. In the segmentation part, a new user-interactive segmentation method (Active-masking) is devised to obtain the precise boundary of the object from a masking quickly made by the user (user quick mark), which makes the user interaction easier and faster. In the tracking part, a novel technique (Mask-updating) is utilized to guide GrabCut to perform continuous segmentation of time-varying organ images without user interaction. The segmentation part provides initial information for the tracking part, in addition to correcting boundaries if tracking is lost. In contrast to other image segmentation methods, the proposed approach has three advantages: (1) It combines automatic and user-interactive segmentation skills seamlessly to guarantee high accuracy and speed. (2) It uses color, edge, boundary, and motion information for lung segmentation by combining GrabCut and optical flow based on a new framework. (3) It shortens user interaction time by introducing a boundary refinement method. In experiments described herein, the proposed algorithm was compared with previously developed algorithms by using several clinical video scenes of thoracoscopic surgeries, in which the ground truth of each frame was carefully labelled by skilled members.
Section snippets
GrabCut
GrabCut is a user-interactive image segmentation technique based on Graph cut [13]. In Graph cut, each pixel in the target image is processed as a node in a graph, the whole graph is established by adding two extra nodes, referred to as “source” and “sink” nodes. Each edge connecting two nodes has an energy. The energy of the pixel-to-pixel edge is calculated based on the difference in the intensity of the two pixels, whereas the energy of the pixel-source (or pixel-sink) edge is computed based
Proposed framework
This section describes the proposed framework shown in Fig. 1. It achieves continuous lung segmentation using the following steps:
- •
Step 1: User-interactive segmentation from the user quick mark that roughly separates the foreground from the background. Precise segmentation is then achieved using the proposed Active-masking method.
- •
Step 2: Automatic tracking of time-varying lung regions with an initial boundary. With the accurate segmentation of the first frame, the proposed Mask-updating
Experiments
This section describes the details of experiments designed to evaluate the proposed approach, based on the use of carefully assessed, manually labelled ground truths, and comparisons with previously developed algorithms. Lung endoscopic videos from two different patients are used as the test data in the experiments. To show the effectiveness of each compared algorithm, we applied Overlap which is estimated as follows [24]: T specifies the area of the ground truth, and R denotes the segmented
Discussion
With respect to user-interactive segmentation, GrabCut relies on the probability based on color components and the smoothness of edges. Cutting with the minimum energy means labelling each pixel to retain its close fit to the GMMs, and assigning different labels to strongly different neighborhood pixels. However, GrabCut ignores an important fact, i.e., the user interaction, which implies the domain of possible boundaries. Compared with GrabCut, the proposed Active-masking integrates GMMs,
Conclusion
This paper proposes a novel algorithm for semi-automatic continuous lung segmentation from endoscopic images. It described a new user-interactive segmentation technique Active-masking for refining user input. By effectively combining GrabCut with optical flow, it enables automatic tracking. As such, it introduces a new framework that provides efficient segmentation for all frames while only requiring user interaction for several key frames.
In general, compared with previous methods, the
Conflict of interest
The authors have no conflict of interest to declare.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all participants involved in the study.
Acknowledgment
This research was funded by a JSPS Grant-in-Aid for Scientific Research (B) (15H03032) and the Center of Innovation (COI) Program from the Japan Science and Technology Agency (JST).
References (27)
- et al.
Object segmentation using graph cuts based active contours
Comput. Vis. Image Underst.
(2007) - et al.
Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching
Phys. Med. Biol.
(2011) - et al.
Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures
Int. J. Comput. Assisted Radiol. Surg.
(2009) - et al.
Systematic review of the use of pre-operative simulation and navigation for hepatectomy: current status and future perspectives
J. Hepatobiliary Pancreat. Sci.
(2015) - et al.
Physics-based interactive volume manipulation for sharing surgical process
IEEE Trans. Inf. Technol. Biomed.
(2010) - et al.
Anatomical reconstruction from endoscopic images: toward quantitative endoscopy
Am. J. Rhinol.
(2008) - et al.
Efficient vessel feature detection for endoscopic image analysis
IEEE Trans. Biomed. Eng.
(2015) - et al.
A comparative study on classification of image segmentation methods with a focus on graph based techniques
Int. J. Res. Eng. Technol.
(2014) - et al.
A segmentation model and application to endoscopic images
- et al.
Supervised learning of edges and object boundaries
A comparative study on medical image segmentation methods
Appl. Med. Inf.
Tumor segmentation from endoscopic images using GrowCut method
Graph cuts and efficient N-D image segmentation
Int. J. Comput. Vis.
Cited by (4)
Kernel-based framework to estimate deformations of pneumothorax lung using relative position of anatomical landmarks
2021, Expert Systems with ApplicationsCitation Excerpt :To provide subsurface nodule position from the lung surface information during surgery, it is required to register the preoperative CT image to intraoperative physical lung by the surface information of the lung. Intraoperative segmentation of thoracoscopic camera images (Wu, Nakao, & Matsuda, 2017) and registration methods using the 2D appearance or silhouettes of intraoperative collapsed lung surface as visual cues to register preoperative CT models with intraoperative camera images (Saito, Nakao, Uranishi, & Matsuda, 2015; Nakao, Akira, & Matsuda, 2017) have been reported. Although they achieved the registration from little information that can be obtained intraoperatively, they had a limitation on the registration of depth direction.
Review of surgical robotic systems for keyhole and endoscopic procedures: state of the art and perspectives
2020, Frontiers of MedicineDeep Learning Based Lung Region Segmentation with Data Preprocessing by Generative Adversarial Nets
2020, Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBSSurface deformation analysis of collapsed lungs using model-based shape matching
2019, International Journal of Computer Assisted Radiology and Surgery