Catheter localization in 3D ultrasound using voxel-of-interest-based ConvNets for cardiac intervention

Medical instrument localization in the US image is achieved by classifying the US voxels. Uherčík et al. combined the image intensity with the Frangi filter response as a discriminating feature for voxel classification in needle localization [1, 2]. A recent study combined the Gabor features with Frangi features to localize the catheter in a phantom heart [3]. We previously used extended discriminating features within a multi-definition and multi-scale approach for catheter segmentation on ex vivo datasets [4]. However, these methods are less robust and less efficient when the US image has large variations in a complex anatomical environment. Recently, deep learning, e.g., convolutional neural networks (ConvNets), has shown significant performance improvement in medical image analysis [5]. For US imaging, the ConvNet has been commonly used to classify voxels into different categories. Two different approaches exist for this: voxel-based ConvNet and semantic-based ConvNet. The first approach classifies individual voxels one by one through the local information [6‐9], while the semantic segmentation approach, i.e., fully convolutional networks (FCNs), predicts segmentation masks directly [10]. Although it has shown promising results by making use of the contextual information, the semantic segmentation approach requires a large number of training data and has high computational complexity.

In this paper, we propose a catheter localization method for 3D cardiac ultrasound imaging. As depicted in Fig. 1, our method consists of three main steps. (1) Candidate voxel selection: a Frangi vesselness filter [1] first processes the input US image to coarsely select most of the candidate catheter voxels. The purpose of a voxel-of-interest (VOI) pre-selection procedure is to reduce the number of voxels to be processed by the ConvNet. To address the unstable response distribution in the Frangi-filtered image, resulting from variations in imaging conditions and catheter appearance [4], we introduce an adaptive thresholding method for the VOI selection, which allows to preserve the catheter voxels while omitting most non-catheter voxels. (2) Voxel classification through ConvNet: for each candidate voxel, a 3D neighborhood patch is extracted, and three orthogonal planes are extracted and processed by the ConvNet for voxel classification. In particular, we propose a simplified triplanar-based ConvNet, called Share-ConvNet, which reduces the computation complexity by sharing a single ConvNet for all orthogonal slices. We also combine two-stage training with a weighted loss function to improve the performance of the ConvNet. (3) Catheter localization: for the classified voxels, a cubic spline-based catheter model is fitted to localize the catheter.

Our contributions are threefold when compared to our preliminary work [9]. First, we employ a vesselness-based filter to coarsely select the candidate voxels to reduce the computation load for the ConvNet. With an adaptive thresholding strategy, most catheter voxels are preserved for further processing. Second, we proposed a Share-ConvNet for voxel classification in 3D US, which is in-depth compared with the existing methods. Third, we collect ex vivo datasets within challenging conditions, to thoroughly test the proposed method for catheter localization. The paper is structured as follows. Our approach is described in “Methods” section. The datasets and experimental results are presented in “Datasets and experimental results” section. Finally, “Conclusion” section concludes the paper with discussions.

In our method, we use Frangi vesselness filtering to select the candidate catheter voxels from 3D US, which enables to dramatically reduce the number of samples to be classified by the ConvNet (typically a reduction from \(\sim 10^6\) to \(\sim 10^4\)). From our previous study [4], this simple selection resulted into a high false positive rate because of the weak voxel discrimination in noisy and low-quality cardiac 3D images. To address this, we introduce an adaptive thresholding method for the VOI selection. 3D US images are first filtered by a Frangi filter with a pre-defined scale and rescaled to the unit interval [0, 1], so-called \({\mathcal {V}}\). After the filtering, we apply an adaptive thresholding method to \({\mathcal {V}}\) to coarsely select N voxels with the highest vesselness response. The thresholding method is trying to find out the top N possible voxels in \({\mathcal {V}}\). Because the filter response has a large variance in different images, the adaptive tuning of the threshold can gradually select N voxels, by iteratively increasing or decreasing the threshold T based on the image itself. The pseudocode is described by Algorithm 1. Based on the pre-selected voxels in 3D US with remaining voxels being around N, the 3D patches are extracted and processed to generate the three orthogonal slices of each voxel for the ConvNet. In our experiment, the initial threshold is empirically set to be \(T=0.3\). Value N is empirically selected to balance the efficiency of ConvNet classification and classification performance. More details are shown in “Voxel-of-interest selection” section.

For voxel-wise classification of volumetric data, the 3D local information is processed by ConvNet to classify the voxels. The straightforward way is to classify the voxel based on its 3D neighborhoods. For each candidate voxel located at the center of a 3D cube, the cube is processed by a 3D-ConvNet [6], as shown in Fig. 2a. However, when using a 3D data cube as input, this approach includes too many parameters in the network, which hampers the efficiency of the voxel-wise classification in 3D US volumes. To preserve the 3D information and yet reduce the convolution operations, especially going from 3D operation to 2D operation, the multi-slice-based method was proposed in [11]. To keep the 3D structure information, [11] employed a multi-view cross-section method, which extracts slices from the 3D cube through different angles. Then, each slice will be processed by an individual ConvNet. An example of this method is shown in Fig. 2b, which is called IND-ConvNet. The extracted feature vectors from the slices are concatenated to feed them into fully connected layers (FCs). As for 3D-ConvNet, it processes the information using 3D operations, which leads to too many computations and large execution times. As for IND-ConvNet, although it keeps 3D information by a slicing approach, multiple individual ConvNet branches lead to redundancy, which comes from using a ConvNet for each slice. Because of these redundancies in the networks, 3D-ConvNet and IND-ConvNet are sub-optimal choices in terms of application and computation time.

In this work, we attempt to propose a simplified method to classify the voxels. We follow the slice-based strategy. Instead of training ConvNet for each slice, we propose to train one shared ConvNet for all slices. All feature vectors from the shared ConvNets are concatenated to form a longer feature vector for classification. We call it Share-ConvNet, which is shown in Fig. 3b. There is a similar structure called RGB-ConvNet [7], as shown in Fig. 3a. It extracts three orthogonal slices from the principal directions of the 3D cube, which are then reorganized into RGB channels. However, this introduces a limitation: the spatial information between each slice is processed rigidly by convolutional filters at the first stage of the network. With shallow processing by ConvNet, only low-level features are processed and this simple strategy cannot fully exploit the spatial relationships. Alternatively, our proposed Share-ConvNet can exploit the spatial correlation in high-level feature space. Based on the binary selection of candidate voxels during the pre-selection, a 3D cube is obtained for each candidate voxel located at the center of the cube. We extract a cube of size \(25\times {25}\times {25}\) voxels, which is larger than a typical catheter diameter of 4–6 voxels in 3D cardiac US. Then, three orthogonal planes passing through the center point of the cube are sliced as the input for the ConvNet (Fig. 3b).

For training with medical images, the class imbalance is the most challenging issue. In our case, the ratio of catheter voxels vs. non-catheter voxels is commonly less than 1/1000. As a consequence and to fully exploit image information, we perform a two-step training when training the ConvNets. First, the number of imbalanced voxels in training images are re-sampled on non-catheter voxels to obtain the same amount as catheter voxels. These balanced samples train the ConvNets. Then, the training images are validated on the trained models to select the falsely classified voxels, which are used to update the networks for finer optimization [8, 9]. Specifically, unlike the diagram in Fig. 1, the training process is applied in the whole US image rather than the VOI processed one. This update step reduces the class imbalance by dropping out the easiest sample points (so-called two-stage training). The parameters of networks are learned by minimizing the cross-entropy, using the Adam optimizer for faster convergence. During the two-step training, the cross-entropy is characterized into a different form to balance the class distribution. In the first training stage, the cross-entropy is characterized in a standard format. However, during the updating, the function is redefined as weighted cross-entropy. These different entropies avoid the bias in the updating stage, which occurs due to the number of false positives being usually 5 to 10 times larger than the positive training samples in the second stage. As a result of the weighted cross-entropy, the networks tend to preserve more catheter voxels than discarding them after the classification. The weighted cross-entropy is formulated in Eq. (1), where the y indicates the label of the sample, while \({\hat{p}}\) is the class probability of the sample, and parameter w is the sample class ratio among the training samples. During the training, the dropout is used to avoid overfitting with 50% probability in FCs together with an L2 regularization with \(10^{-5}\) strength. The initial learning rate is set to be 0.001 and rescaled by a factor 0.2 after every 5 epochs. Meanwhile, to generalize the network in orientation and image intensity variation, data augmentation techniques like rotation, mirroring, contrast and brightness transformations are additionally applied. The mini-batch size is 128, and the total training epoch is 20 which are around 25k in the first training, while iterations in the second training are around 100k.

The classified volume may include some outliers, which are generated from the blurry tissue boundaries or catheter-like anatomical structures. To robustly localize the catheter, we employ our previously designed SPD-RANSAC method to fit a pre-defined catheter model [12]. A curved cylinder models the catheter with a fixed radius, which is set to be three voxels in this paper. To robustly localize the catheter, the classified volume, so-called dense volume, is processed by connectivity analysis to generate clusters. Then, the cluster skeletons are extracted to generate the sparse volume. During the fitting stage, three control points are automatically and randomly selected from the sparse domain and ordered in orientation by principal components analysis. The reordered points ensure the cubic spline fitting passes the points in sequential order, which generates the catheter-model skeleton which is shown in Fig. 4. The localized skeleton with the highest number of inliers in the dense volume is adopted as the fitted catheter. The inliers are determined by its Euclidean distances to the skeleton.

In this study, we have collected 4 ex vivo datasets on 4 isolated pig hearts, which resemble the human heart (1 heart for 1 dataset). Table 1 summarizes these datasets. Dataset 1 was collected with a Philips CX-50 machine, while the rest was collected by a Philips EPIQ-7 US machine. Dataset 4 was recorded by TTE (Transthoracic Echocardiogram), which explains the larger voxel size, while other datasets were recorded by TEE (Transesophageal Echocardiography). During the recording, all the images were tuned to have the best visual perception. However, due to equipment variations, the US parameters were different for each dataset. Moreover, to make sure the images in each dataset are independent from each other, we changed the relative position between the heart and US probe to obtain a different appearance of the heart in each captured image. Furthermore, we extracted the catheter and re-inserted it into the heart chambers to make the images independent, i.e., 1 session for 1 image. All datasets were re-sampled to obtain an isotropic spatial resolution and were annotated manually by experts. Examples of three cases are shown in Fig. 5, which compares the recordings on phantom heart, pig heart and human heart. Compared to the phantom heart and human heart, the captured pig heart images are more complex. Compared to the phantom data, real pig tissue has more complex anatomical structures, which makes it hard to distinguish between the catheter and tissue. When compared to the real human heart, the chambers of the pig heart are collapsed due to the dead tissue, which leads to a small free space within the heart. Moreover, the human heart image, which is shown here, has a larger field-of-view than the pig heart recordings, as the data were collected for Transcatheter Aortic Valve Implantation (TAVI) operation. To fully make use of the limited datasets for deep learning, we performed threefold cross-validation on all collected images.

To reduce the number of voxels for classification, we applied the Frangi vesselness filter to select the candidate voxels. However, it cannot filter out the catheter voxels from tissue and background with a pre-defined scale due to too many false positives [4]. In our method, we first apply the Frangi filter with scale size equal to 2.5 to filter out the most tubular-like structures. Then, Frangi responses are rescaled to the unity interval, which maps response into a probability-like range.

To evaluate the performance of thresholding, we employ three metrics: Recall (the remaining catheter voxels versus ground-truth catheter voxels), Ratio (thresholded voxels versus all voxels, to evaluate the voxel preserving ability) and their fusion score (mimic \(F_1\) score by replacing Precision by Ratio to evaluate a joint threshold performance), which enables to show the preservation performance of catheter voxels and removes non-catheter voxels. The metrics are defined in Eq. (2), where TP is true positive, FN is false negative, TV is remaining voxels after the threshold, while AV is all voxels, giving the specification:The performances of adaptive thresholding are shown in Fig. 6, where the voxel threshold N ranges from 10k to 190k voxels with step size 10k. The values are obtained by averaging of all the testing volumes through threefold cross-validation. It can be observed that the adaptive thresholding method provides a more stable voxel distribution, i.e., a smaller fraction of the whole pyramid area while keeping a higher recall. As a result, the proposed thresholding method provides a better selection for Voxel-of-Interest. However, this pre-selection leads to a drop in Recall. As a consequence, in the following step, a ConvNet with high Recall for voxel classification is needed.

Based on voxel classification, the model fitting is applied to the binary images to localize the catheter (its skeleton and end-points) and remove the outliers. We employ the following metrics to measure the model fitting performance: skeleton-based metrics, Volumetric Similarity (VS) and Average Hausdorff Distance (AHD) [13]. More specifically, skeleton-based metrics include two specific types: (1) end-points error (EE) characterized by the average distance between corresponding end-points on the detected catheter and the end-points on the annotation; (2) skeleton error (SE): the average distance between 5 equally sampled points on the detected skeleton and the ground-truth skeleton. Skeleton error has more robust performance than EE. This performance difference is explained by analyzing the difficult cases. For example, sometimes the catheter tip is attaching to the tissue so that it is hard to distinguish the tip from the tissue in B-mode imaging, as shown in Fig. 7a. In such case, the EE metric will give a higher error but SE has inherently better accuracy. However, EE would be more informative than SE, because correctly localizing the tip of the catheter can facilitate the success of the intervention.

We here compare the catheter localization performance based on MF-AdaB, Share-ConvNet, VOI-90k-Share-ConvNet, and VOI-190k-Share-ConvNet. The localization performances are shown in Table 6, which are the average of threefold cross-validation with five times fitting in each volume. The table shows that our proposed Share-ConvNet method achieves a better performance with a lower position error, smaller than the diameter of the catheter. Furthermore, the results show that the VOI-based ConvNet can boost the localization precision in terms of the lowest error. When comparing the results in Tables 5 and 6, VOI provides lower \(F_2\) score, but better localization accuracy. This is because VOI provides a higher Precision performance so that a better sparse volume can be achieved. The model fitting relies on the SPD model fitting where fewer outliers would make randomly control points selection more stable. Moreover, with a sacrifice of \(F_2\) score through VOI selection, we achieved \(10 \times \) faster voxel-based classification, which has shown the trade-off between classification accuracy and efficiency. The whole chain based on VOI-90k-Share-ConvNet takes around 10 s (Frangi filtering: 1.5 s, VOI selection: 0.3 s, ConvNet: 6.8 s and SPD-RANSAC: 1.9 s).