Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy

Image-based monocular 3D reconstruction has drastically improved over the last decade with learning algorithms and improved computational power [6]. Nonetheless, 3D reconstruction in endoscopy and laparoscopy remains a difficult and open problem because endoluminal tissue is view dependent and reflective with limited texture which makes sequential image matching very challenging. Supervised approaches, on the other hand, remain infeasible as the colonoscope cannot easily integrate additional sensors, because it must be thin, flexible and has to encompass channels for water, air and instruments. This means that ground truth data that can train learning-based algorithms cannot be obtained using standard equipment and insufflation during colonoscopy deforms and invalidates anatomical data obtained from a computer tomography (CT) scan of the colon.

Approaches to 3D reconstruction of the colon and other parts of the gastrointestinal tract have traditionally not been data driven and have focused on modelling cues and characteristic geometry [7]. Both Structure-from-Motion (SfM) and Shape-from-Shading (SfS) as well as their combination have been considered alongside surface modelling [8]. More recently, learning methods have emerged using training data from a simulator that provides ground truth camera motion [9], or learning correspondences that are used to learn camera motion in an unsupervised manner [10]. The use of synthetic data generated from an anatomical CT scan or data from photorealistic phantoms has been explored to leverage data-driven approaches [11, 12]. Such approaches are trained on rendered images instead of real colonoscopy images and therefore require training of unsupervised transformer networks that translate between the two different domains. A fully unsupervised approach trained on sequential frames was shown to successfully predict relative depth information that can be used for polyp detection [4]. A self-supervised method, which embeds a depth prediction neural network in an SfM approach, trained on sequential images as well as sparse 3D points and relative camera poses obtained from SfM has also recently been proposed [13].

Generative adversarial networks (GANs) [14] learn the probability distribution over a dataset. A standard GAN consists of two players: the generator, which learns to generate new, realistic looking instances, and the discriminator, which learns to distinguish the generator’s fake data from real data. Competing with one another, both players gradually improve, until the generator’s outputs are indistinguishable from the underlying real data.

Let \(y = {\mathcal {G}}(z|\theta _g)\) describe the output of the generator function with random input \(z\sim p_z\), where \({\mathcal {G}}\) is a convolutional neural network (CNN) with parameters \(\theta _g\) and z is random noise. Further, let \(p = {\mathcal {D}}(y|\theta _d)\) be the scalar output of the discriminator \({\mathcal {D}}\) with some input y and CNN parameters \(\theta _d\), describing how probable it is that y is drawn from the real distribution. Then, the generator’s goal is to become so good at fooling the discriminator that the discriminator assigns high values \(p = {\mathcal {D}}( {\mathcal {G}}(z|\theta _g)|\theta _d)\) to the outputs of the generator. Simultaneously, the discriminator is trained to assign small values to the outputs y of the generator, while assigning high values to real inputs \(x\sim p_\mathrm{data}\). That is, generator and discriminator play a minimax game yielding at their equilibrium a generatorthat ideally learns the distribution over the input data. For a proof of convergence, we refer to [14].

In the case of conditional GANs [15], the discriminator and the generator not only observe y or z, respectively, but also the condition c. Recently, different versions of cGANs have been proposed. Some condition on semantic labels [16], others, like pix2pix [17], learns to transform images between domains by conditioning on entire images. Given a set of correspondences (c, x) between real images and their depth maps, one can compare the output of \({\mathcal {G}}\) to its actual label by measuring the \(L_1\)-error. This error can be used to train \({\mathcal {G}}\) to generate outputs that not only look realistic, but are as close as possible to the real label. The loss functions are thenand \({\mathcal {G}}^* = \arg \min _{\mathcal {G}}\max _{\mathcal {D}} {\mathcal {L}}_\mathrm{cGAN}({\mathcal {G}},{\mathcal {D}}) + \lambda {\mathcal {L}}_{L_1}({\mathcal {G}}) \), where \(\lambda \) is a regularizing weight.

While this approach requires explicit input–output pairs for training, other models can be trained on unpaired examples [18].

We propose to train a cGAN to translate real colonoscopy images to depth maps. Although cGANs have been used for depth prediction in other environments [19], previous approaches required the availability of training data consisting of RGB–depth pairs. Because training data are difficult to obtain during endoscopy, we avoid the necessity of paired colonoscopy images and corresponding depth maps altogether. Instead, we propose to train a pix2pix network on paired simulated data and unlabelled real images. Our approach leverages the availability of real images improving the generalization capabilities of the network while avoiding explicit domain translation networks. These networks are error-prone, due to the lack of an application-specific loss function, and are trained independently of the depth prediction network precluding latter from back-propagating its gradients to the former. A cGAN, on the other hand, allows optimization of domain adaption and depth prediction simultaneously and is independent of camera, lighting and patient. Additionally, we publish the, to the best of our knowledge, first dataset of synthetic colonoscopy images with corresponding depth here: http://​cmic.​cs.​ucl.​ac.​uk/​ColonoscopyDepth​.

GANs allow to learn the distribution over the shape of the colon, and while the texture of the synthetically generated data will differ from real data, the shape of a colon model generated from a CT mesh will be consistent. Therefore, learning a distribution over shapes observed during colonoscopy, the generative model can be trained to predict coherent depth maps, that is, even if it is exposed to unseen textures. Discriminative models, on the other hand, learn a deterministic mapping, which can fail once the input deviates strongly from the training data.

Let c equal an input RGB image, and x equal its corresponding depth map. Then, y is the predicted depth that the generator produces. A scheme of pix2pix is shown in Fig. 3a. The original version is solely trained on image–depth pairs (x, c), and instead of inputting a random vector z drawn from a normal distribution, dropout layers with dropout probability 0.5 are used during training and inference to simulate noise. The generator is composed of an encoder–decoder architecture with skip connections between each layer i and \(L+1-i\) where L is the number of layers. Combining several levels of down-scaling enables the network to learn different degrees of details, with the first layer learning a general representation, and subsequent layers learning more local feature representations. The architecture can be found in [21]; however, we removed batch normalization layers because we noted those to cause inconsistent results and an exploding generator loss.

We train and compare three models: (i) the original pix2pix model without batchnorm layers; (ii) pix2pix with our training modification which, for simplicity, we refer to as extended pix2pix, although we would like to emphasize that is not an extension to the model itself but only to the training; and (iii) a discriminative model that has the same architecture as the generator in the original pix2pix network. We measure the absolute \(L_1\) distance \(||\mathbf{x }-\mathbf{y }||_1 \), the relative error \(||\frac{\mathbf{x }-\mathbf{y }}{\mathbf{x }}||_1\) and the root-mean-squared error RMSE = \(\sqrt{\frac{1}{N}\sum _n(x_n-y_n)^2}\) between ground truth depth x and predicted depth y. Results on all three models are shown in Table 1.

We show that even on synthetic data the generative models lead to better results. The superiority of the extended pix2pix over the standard pix2pix on synthetic data is counter-intuitive but can be explained through the facts that the extension has a better ability to generalize and that hyper-parameters, which are crucial to the performance, were chosen on the extended model. Experimenting with input transformation, we find that logarithmically scaling the input depth and remapping it before the loss calculation has a beneficial result on the prediction. One reason could be that generators, in general, work less well on extreme values. Example predictions from each of the nine subsets of the testing set are shown in Fig. 4. One can observe that errors are especially high for regions with a larger distance from the scope. The average maximum pixel-wise \(L_1\)-error per image is 2.18 cm and is mainly found along folds with a step in depth.

We test the proposed training modification on real images without any preprocessing [22]. From Fig. 5, one can see the model detects minute details and, under qualitative inspection, predicts sensible depths. Critically, polyps and folds are correctly located. Even exceedingly small polyps, such as depicted in the centre of Fig. 5j, are correctly reconstructed. The images also reveal that reflections cause no inconsistencies nor do blood vessels or air bubbles, and hence, the model is largely invariant to texture. We further project some of the examples into 3D space and observe that the projected shapes are plausible (Fig. 6). From the images, one can see steps in depth along folds, the elevated nature of the polyps, and relative uniformity of the scale of the lumen across similar scenes.

We also compare the results using the new training paradigm with those from the standard version of pix2pix. Figure 7 shows that the depth maps generated by the standard version are uneven, less detailed and appear to be blemished. Further, we juxtapose our prediction to those of Mahmood and Durr [12] in Fig. 8. One can observe that our method appears to better reflect the overall tubular shape, the general depth, and the position and the magnitude of the lumen, although notably this is purely a qualitative statement and no full comparison is possible as we ran our method of images extracted from the pdf of their paper. Lastly, we show three out of five scenes (out of the 44 different testing scenes, each showing several frames) on which the extended pix2pix model fails in Fig. 9. The failures indicate that the model has learned to predict tubular shapes.

We trained the models on half of the original dataset and the additional dataset made from the phantom. The number of epochs was reduced to forty as the network starts to over-fit to the shape of the patient CTC. However, the synthetic dataset based on the CT of the phantom is too uniform to suffice as the sole training set. For the extended version of pix2pix, we used unpaired frames acquired from the da Vinci as real input images. Due to the homogeneity of the validation set, we pick three representative frames for validation. Results are shown in Table 2. Although the phantom is obstructed by markers, the network nevertheless recognizes the phantom’s shape (Fig. 10). Although the scale is inaccurate in the centre of the phantom, the overall scale is correctly identified. The \(L_1\) loss between ground truth and prediction is 1.42 cm and also accounts for the markers obstructing the image and the reconstruction error discussed in “Data generation” section. The RMSE of the world-to-Unity transformation alone is on average 0.12 cm. For comparison, we train the original pix2pix and its discriminative version in the same manner as our proposed model and report the result on the same frame in Fig. 10. The \(L_1\) distance between ground truth and prediction for the two models is 1.94 cm and 1.97 cm, respectively. This reflects a more than 37% greater error when compared with the extended pix2pix model.