Deep learning for sensitive detection of Helicobacter Pylori in gastric biopsies

Modified Giemsa and regular H&E stained slides were scanned using a NanoZoomer S360 (Hamamatsu Photonics) whole-slide scanning device at 40X magnification, as well as a DP200 slide scanner (ROCHE Diagnostics) at 40X. The slides were then evaluated for image quality and included within the validation cohort, if at least 50% of the tissue allowed a distinction of cell types. The evaluated tissue had to include at least 50% of gastric tissue to be included. Therefore, biopsies that primarily included intestine or esophageal tissue were not included.

Our approach consists of first localizing areas of H. pylori presence (herein referred as H. pylori hot spots) using image processing, and then cropping the hot spots into 224 × 224 patches and classifying them with a deep neural network. H. pylori typically exists inside and around glandular structures that can be described as white regions image regions inside the gastric tissue (Fig. 1b). To localize these regions, we downscaled the slide by a scale of × 32 in each axis, applied Otsu thresholding on the saturation channel in the HSV color space, then performed erode/dilate morphological operations to create a mask with the white regions. Then we would find the contours of these regions and crop them into non-overlapping patches of 224 × 224. Regions that had an area smaller than 32 pixels were discarded. A typical slide had hundreds to thousands of H. pylori patch candidates. These were then filtered by a classification network to only keep patches that contained H. pylori.

To classify the patches, we trained a compact VGG-style deep neural network [18]. The same network was trained on both Giemsa and H&E slides, in order to improve the generalization of the network on slides with stain variation. The network had 9 convolutional blocks followed by 3 fully connected layers. The first two convolutional layers are pointwise convolutions that help the network generalize on multiple stains [20]. The next two layers had kernel sizes of 7 × 7 and 5 × 5, that increased the receptive field, and the rest had kernel sizes of 3 × 3. The capacity of the network was reduced by scaling the number of features in every layer. The last convolutional layer had the output of only 32 channels. The first two fully connected layers had 1024 channels each, and the last layer has two outputs. We used 50% dropout between the fully connected layers, and batch normalization layers after the ReLU nonlinearity layers in the convolutional part of the network. In addition, we used standard cross entropy as the loss function and weighed the categories by their proportions in the dataset. As an optimizer, we used Adam with default parameters. We used a batch size of × 32, that was split among 4 1080ti GPUs, using the PyTorch framework. To help generalize among different stains, we used aggressive color augmentation. The brightness, contrast, saturation, and hue of every image was randomly jittered with strong portions. We also randomly flipped and rotated the image, then applied random grayscale and local elastic transformations.

Although the network performs classification, we applied a technique to help with the localization of individual H. pylori bodies. We first applied the SmoothGrad technique and averaged the gradients of the category score with respect to noisy input image pixels [21]. The gradient image is then passed to a ReLU gate, to keep only positive gradients. The gradient image was then used as a threshold, and input image pixels that had lower gradients were masked out, in order to only retain pixels that were important for the network decision. The threshold was then set to the 99.8% gradient percentile to keep the top instances. Then, the gradient image was dilated, and we drew contours over connected components to highlight areas that had H. pylori bodies with high confidence.

An overview of the training dataset can be found in Table 1. Overall, 191 H&E and 286 Giemsa stained slides were used, with a total of 2629 tiles containing for Giemsa 790 and H&E. About 4241 and 1533 tiles without H. pylori -like bacterial structures were used for Giemsa and H&E, respectively.

Using the strategy of training an initial model to classify H. pylori hot spots (Fig. 1b), we annotated the whole slide image (WSI), avoiding the need to manually annotate H. pylori regions in slides. First, the H. pylori candidate localization algorithm was applied on every slide, creating thousands of crops on average. Crops from slides lacking H. pylori were automatically labeled as “not H. pylori.” Crops from slides containing H. pylori were labeled in a gallery by pathologists. Since the number of H. pylori may be scarce even in slides that do contain H. pylori, there was a high level of variance between categories that made the annotation challenging. We used an active learning approach, where the dataset was constructed in steps of labeling few hundred crops each time, then a model was trained, and the remaining unlabeled crops were ordered from high to low according to their H. pylori category score. In the next step, the labeling was done on high scoring H. pylori crops, dramatically increasing the rate of encountered H. pylori crops and making the dataset labeling time efficient. This process was stopped when the rate of appearing H. pylori crops was very low, after a few thousand labeled images. The dataset contains more Giemsa slides than H&E slides. Using active learning, we were able to reduce the reviewed crops from nearly a hundred thousand crops (the total number of extracted candidates) to several thousand.

We used a decision support algorithm approach of showing the top ranked 39 tiles (224 × 224 pixels) of an WSI to an experienced pathologist. We defined H. pylori presence, if at least two bacterial-like and spiral structures were present within an image. In total, the area of the presented fields reflected less than 1% of the area of the whole slide tissue area (Fig. 2b, c).

In a subset of samples with conflicting microscopic diagnosis, despite clear visible H. pylori-like structures, or inconclusive H. pylori status, additional 46 samples were analyzed using PCR. H. pylori was screened with the RIDA GENE® Helicobacter pylori assay (r-biopharm, Darmstadt, Germany). At least one area on a hematoxylin–eosin stained slide was selected by an experienced pathologist (RB, AQ, SK) and DNA was extracted from corresponding unstained 10 µm thick slides of formalin-fixed, paraffin-embedded tissue by manual micro-dissection. Isolation was performed semi-automatically using the Maxwell® 16 FFPE Plus Tissue LEV DNA Purification Kit on the Maxwell® 16 System (Promega, Mannheim, GER) following the manufacturer’s protocols. The RIDA GENE® Helicobacter pylori assay was performed with 5 µl of DNA for each sample independent of DNA concentration and fragmentation on a CFX96 Touch™ Real-Time PCR Detection System (BIO-RAD, Hercules, California) according to the manufacturer’s protocol. Analyses were performed with the corresponding software. As amplification control, an internal control DNA (1 µl) was added to each PCR-Mix of all samples. All samples had to show a positive amplification of the internal control DNA (Additional file 1: Figure S2A). For a reliable detection of H. pylori and the Clarithromycin resistance positive (red curve) and negative controls (black curves) were included into each run (Fig. 2b). Additional file 1: Figure S2C shows the amplification curves for the detection of the clarithromycin resistance. The fluorescence signal for a resistance to clarithromycin had to be more than 20% of the fluorescence signal of the positive control. To determine the sensitivity of this assay, a serial dilution (1:2) of the positive control was performed (Additional file 1: Figure S2D). Discrepant cases were validated with the GenoTypeHelicoDR assay (Hain Lifesciences GmbH, Nehren, Germany) according to the manufacturer’s protocol with 5 µl of DNA for each sample. Immunohistochemical staining for H. pylori was performed on full tissue sections using the BOND MAX from Leica (Leica, Germany) according to the manufacturer’s protocol, and a polyclonal anti-H. pylori antibody from Cell Marque™. All stained sections were evaluated by an experienced pathologist (AQ, RB).

Symptoms of gastritis regularly require an endoscopic evaluation of the stomach. While several diagnostic tests can be applied, histo-morphological assessment of the tissue can be considered a standard procedure in Western countries (Fig. 1a) [4‐10]. Clinically, the subtype of bacterial-associated gastritis (Type-B-Gastritis) is linked to infection with H. pylori. Usually, modified stains, such as Giemsa stains, help visualize these bacterial structures for histo-morphological assessment. In addition, H&E stains are performed to evaluate morphological abnormalities.

Therefore, we generated a decision support algorithm that would highlight regions of H. pylori presence on both modified Giemsa stains and regular H&E stains. We applied a combination of image processing and deep learning to extract regions of H. pylori presence, as well as to classify those regions, using a convolutional neural network (CNN) (Fig. 1b). Due to the fact this hybrid approach limits the amount of image information that are processed by a CNN-thereby saving computational resources–it allows the WSI analysis in high resolution to be done within seconds on regular clients. Typically, H. pylori resides in areas that can be described as a white background from an image processing point of view. Thus, we applied a pre-processing step to detect these candidate regions, using a combination of thresholding- and morphological operations on a low-resolution image. We decided to extract all candidate regions and annotate those patches in a gallery as the annotation of H. pylori containing areas is necessary, but time consuming and challenging on WSI (Fig. 1b). We applied an active learning approach on the annotated data, which represents an imbalanced data pool. The images were sorted by their H. pylori scores, allowing high-scoring images to be presented in priority.

In order to overcome the stain variation of histological specimens, we applied aggressive color augmentation on the extracted images for better performance and generalizability. Images were converted to a grayscale with 10% probability and heavy color jittering. In addition, we trained the same network for both H&E and Giemsa stains to further improve generalization across color- and stain variations.

Having generated the final model, we further aimed to understand the decisions of the network. We visualized and highlighted areas in the image that were significant for the network’s output (Fig. 2a). Having shown that the algorithm detects H. pylori bacterial structures on both modified Giemsa stains and regular H&E stains, we validated the algorithm using an independent cohort of gastric biopsies of cases that had not been used for training purpose. Initially, we validated the decision support technology against microscopic diagnosis.