Artificial Intelligence Techniques for Automatic Detection of Peri-implant Marginal Bone Remodeling in Intraoral Radiographs

As mentioned before, YOLOv3 is proposed to detect and locate the screwed part and the crown part, which are the two types of objects to be detected. It also allows us to determine the relative position of both parts with the aim of identifying whether the implant belongs to the upper or lower jaw. Some approaches based on convolutional neural networks (CNNs) can be found in [28]. Lee and Jeong [29] applied a pretrained deep CNN architecture (GoogLeNet Inception-v3) to classify different models of the implant body in one of the three classes studied, including periapical radiographs. This is also the line proposed in [30], where VGG-16 and VGG-19 models were compared against the model of the authors themselves. In this regard, the main drawback of these approaches is to determine the location of the screw for its automatic classification. Once the region of interest is defined, their bounding boxes can be supplied to the network.

Thus, YOLOv3 is trained with these two kinds of objects so that during the detection process, both parts can be differentiated and isolated. This means that two classes are to be considered, i.e., they guide the definition of the three loss functions involved (box, object, and class), particularly the one belonging to the classification loss.

The procedure defines the DL part introduced in Fig. 1, and it is established as follows, where Fig. 2 serves as a guide for its description:

Regarding both images in Fig. 2, extending it to other images in the set, we can see that some parts of the screws are not visible in the two types of intraoral radiographs. Although this fact does not prevent further processing, it is not a problem when the percentage of bone resorption is to be calculated with respect to the total length of the screw. The length in millimeters of the implant is always known. Once the X-ray acquisition device is calibrated by establishing the correspondence between millimeters and pixels (Materials section), the relative and absolute degree of deterioration is immediate. Obviously, the best situation to determine the percentage is to visualize the full length of the screw without occlusions, although the proposed approach was designed to detect critical points where significantly deteriorated areas are identified without such requirements and, therefore, even with missing parts.

Once objects have been detected, the next step is focused on the screws to identify their boundaries and to fit the best line (curve), to end with the computation of the percentage relating to bone remodeling. This is the IU process in Fig. 1, which is illustrated based on the periapical image displayed in Fig. 3a. Figure 3b displays the detected objects by YOLOv3. Here, all crowns and screws have been detected with their confidence scores, i.e., all above 0.5. In this case, implants belonged to the lower jaw, according to the process described above.

To validate a crown bounding box, it must be spatially located above or below a bounding box associated with a screw, depending on whether it is a lower or upper jaw implant. This is verified by extending the bounding box of the screw toward the bounding box of the crown in such a way that an overlap, with respect to the crown bounding box, of more than 30% is achieved.

Sixteen peaks are identified in the Hough polar space, which determine the corresponding m and d parameters associated with 16 straight lines adjusted to the edges. The goal is to fit several piecewise line segments for the best description of the curved edges. This minimizes the effect of some adverse structures, such as edge protrusions seen at the top of the edges, which is detrimental when trying to fit a unique line. Segments with slopes tending toward the horizontal are excluded since the target edges to be described are oriented toward the vertical. For this purpose, segments with slopes less than 30° are discarded. Figure 5b displays the segments extracted and associated with the four edges. Then, we proceed to isolate groups of segments to fit a unique line on the cloud of points that define the groupings of the segments. Figure 5c displays the grouping of such lines; each group is identified by applying region labeling [35]. Each labeled region is expanded horizontally by a morphological dilation operation with a structural element of 1 × 5. Each labeled and expanded region is intersected with the edges belonging to the bounding boxes, in this example, the ones displayed in Fig. 5a. This allows the isolation of labeled pixel alignments, as displayed in Fig. 5c, on which groups of labels are identified; in this example, four groups (1, 2, 3 and 4) are marked. For each group of labels, two least squared regression-based polynomial settings are applied. The first adjusts a straight line, as displayed in Fig. 5d, and the second is to adjust a curve, i.e., a polynomial of degree equal to or greater than two [36]. In all our experiments, we verified that a degree of two suffices (this is justified according to the experimental results explained in the Results and Discussion section). This means that the polynomial is defined as follows: x = a + by + cy², where the independent variable is expressed as y and a, b and c are the coefficients estimated by regression. For each grouping of pixels, the points with the maximum and minimum values of component y are identified, y_iM and y_im, where i represents the number of the group. Figure 5e displays the four curves adjusted for the four edges available together with y_1M, y_4M, y_1m and y_4m for illustrative purposes.

For each Cum vector, we compute three points at three samples at which the profile changes abruptly according to the statistical mean value of Cum. For illustrative purposes, Fig. 6 displays the profile (blue line) corresponding to Cum computed in the curved line of Group 3 in Fig. 5. From the starting point, identified as zero y-position at the profile, Cum values are positioned at levels above the value 100 and higher until reaching the abrupt change that occurs at the y-position of 155 (y_3p), just where a significant decrease in intensity levels appears, which corresponds to the starting point of tissue degradation. This is the first critical point for this curved line, located at y_3f = y_3M – y_3p with x_3f = a + by_3f + c(y_3f)². At this point, we compute five Cum R values toward the direction of the end point, and these values are averaged to obtain R_av, which is a reference intensity level.

We continue exploring the curved line only until entry into the crown is detected. Continuing with the example of the same curved line, such input is detected when the average intensity profile provided by the new positions of Cum from the critical point reaches high intensity values corresponding to the radiographic image of the crown. During the previous process, Cum values were computed with the average intensity levels of the left part (L), but now, the computation is based on the right part (R), guaranteeing full entry into the crown. Therefore, after five explorations with R values, all greater than R_av, we determine that the crown is entered, and therefore, the identification and position of the second critical point, whose y-coordinate is taken as that corresponding to the first intensity change in the five explorations considered. In the example, it is y_3s with x_3s = a + by_3s + c(y_3s)². Figure 7a displays the eight critical points detected on this periapical radiograph. The first critical points are marked with cyan asterisks and the second ones with blue asterisks. The piece of each curved line between both critical points determines the degree of bone resorption, and its length with respect to the length of the corresponding curved line determines the percentage of bone resorption. The position of the second critical point can be confirmed by verifying that its position is inside a bounding box defining crowns. Although this verification has not been necessary in our experiments, it is possible that it should be considered in the future. Figure 7b displays the six values, in pixels, obtained for this image, where we can see that the maximum and minimum levels achieve 20% and 6% bone resorption, matching qualitatively with what is visually observed.

As mentioned in the Materials section, from the 2920 images, we use 1460 images during the training phase of the YOLOv3 model (1314 for training and 146 for validation, i.e., 90% and 10%, respectively), and the remaining 1460 images are used for testing. All of them are conveniently labeled with the Image Labeler Application of MATLAB [26] to obtain the network model parameters derived from the training and the metrics described below during classification.

Intersection over union (IoU), i.e., the Jaccard index [37], is the basic metric used for determining the degree of matching between the predicted and ground-truth bounding boxes associated with each object (screw, crown). From the IoU and considering an objectness score greater than 0.5, the well-known mAP@0.5 metric is used for assessing the YOLOv3 performance. Table 2 displays the mAP@0.5 values obtained in our experiments with the testing data separated by screws and crowns in periapical and bitewing radiographs, distinguishing between upper and lower implants.

This validation is only related to the process of detecting clinical structures of interest (crowns and screws) within the DL process (Fig. 1); therefore, its relevance is restricted to such a process but not from a strictly clinical point of view. However, it is worth noting that any failure at this stage will have a negative impact on the clinical bone resorption computation.

Higher mAP@0.5 values indicate better performance. Therefore, from the results in Table 2, we can see that the best performance is achieved in screws as lower jaw implants, particularly in periapical and not so much in bitewing. Regarding crowns, it is also in the lower jaw where the best results are obtained. This situation is determined exclusively by the type of images, which contain a greater distribution and distinction of the lower jaw parts versus the upper jaw parts. Regarding the best performance of screws against crowns, we verified that this is because they appear more separated in the images than crowns that are much more grouped in both lower and upper jaws, being more difficult to distinguish them at the individual level. These results correspond with those obtained in [8] and, therefore, in the same range of values. The aim is to identify the clinically relevant parts (crowns and screws) to guide subsequent IU processing without the need to achieve maximum precision at the DL stage, as this responsibility is entrusted to IU processing. Therefore, achieving values such as those obtained in [7] is considered acceptable.

The number of labeled objects apparently represents a relatively low number in the context of DL. However, two things concerning the proposed approach should be considered in this regard:

During the testing process of the DL approach (classification phase with YOLOv3), the following considerations need to be made based on the illustrative examples displayed in Fig. 9:

We designed a specific testing strategy to assess the performance in adjusting lines along the screws. To analyze this fit, it is unnecessary to distinguish between the upper and lower jaw or between the anterior and posterior teeth, as only the fit along the edges of each screw is of interest due to contrast differences, regardless of their position, as there is no dependence in this respect.

With this purpose, the clinical specialist authors, using the interactive MATLAB [26] function that captures pixel coordinates on the images, manually select at least n ≥ 10 points on each edge, obtaining the corresponding coordinates (x_i,y_i), which are labeled as ground truth once they are agreed upon by all specialists. A polynomial of degree d, \(\hat{y} = p_{1} x^{d} + p_{2} x^{d - 1} + ... + p_{d} x + p_{d + 1} ,\) is adjusted by applying least squares regression with the ground-truth points, with d varying from 1 to 3. A grade higher than three is not necessary, as the curvature of the screw edges is relatively smooth. The polynomial that best fits the selected points is determined by measuring the mean squared error (MSE) that determines differences between the predicted point values and the ground-truth points. The lower the MSE is, the better the model’s predictive accuracy.

In our experiments, with the 1394 images finally used for testing, we used N = 3206 screw edges by setting d = 1, 2 and 3 and computing the MSE for each edge. The best fit is selected with the minimum MSE value, which is cataloged as the ground truth for the corresponding screw edge. We verified that 7% of polynomials have been with d = 1 and 92% with d = 2. Only a portion of 1% corresponds to polynomials with d = 3. These results justify the choice of degree two during the segmentation process explained in the Materials and Methods section. Table 3 summarizes the results derived from the experiments to determine the degree of the polynomial as a function of the minimum value of MSE.

Now, we compute a matching score for each screw edge (ms_e) by comparing the ground-truth line (G) and the one adjusted (A) by the proposed procedure with d = 2 and N = 3206. In this regard, following the vertical direction of the image and, therefore, along the y-coordinates, we determine the minimum (Y_m) and maximum (Y_M) y-coordinates of the overlap between G and A. From Y_m to Y_M and for each y-coordinate (Y_i), we obtain the corresponding x-coordinate value for G (X_Gi) and A (X_Ai) and compute the matching score as follows:

Expression (3) represents the average deviation values between the X-coordinates for each edge (e). Considering the N tested edges, we compute the statistical average (m_e) and standard deviation (σ_e), both related to m_se, obtaining 2.75 and 1.01 pixels, respectively.

To determine the validity and significance of the results for the proposed approach, we designed a testing strategy based on the one-sample statistical Student’s t distribution (t test), [39, 40] which is useful when a small number of samples is involved. The t test is recommended against the z test (which assumes a normal or Gaussian distribution) when the standard deviation of the population from which the samples are collected is unknown, as is the case in our experiments. The testing strategy is focused on each edge to be detected. The samples are defined through the statistical variable \(x_{e} \equiv X_{Gi} - X_{Ai} ,\) i.e., each sample in x_e is the difference between the X-coordinates between the detected points (A) and the ground-truth points (G). The length of each edge (L_e) is defined by the difference L_e = Y_M – Y_m, which is the number of samples in x_e. In our experiments, the L_e values are, on average, approximately 285, i.e., the dimensions of x_e. The degree of freedom for this test [38, 39] is df = L_e –1, i.e., df > 30, which is a value accepted by the scientific statistical community.

The best performance is obtained when x_e values are null, a perfect match between the fitted edge line (A) and the corresponding ground-truth line (G). This defines Hypothesis H₁ in the t test and alternative Hypothesis H₀ (x_e values differ from zero). To measure the performance of these hypotheses, we use the p value [41] as the probability of obtaining results from the test at least as extreme as the observed results, assuming H₀ is true. Low p values indicate that the extreme observed outcome is very unlikely under H₀ (rejection of nonmatches) and specified by the significance level (α) set to 0.05 in our experiments, which is a typical value [42].

We tested the N edges individually by computing the p value for each edge, obtaining an averaged p_p-value of 0.0106 over N, i.e., the degree of rejection of H₀ is high. Only 2.43% of the N edges tested were not rejected. The estimated standard deviation averaged with the N edges (σ_p), is 0.981 and, therefore, on the same order of magnitude as σ_e obtained above. According to these results, we can conclude that the deviations obtained against the ground truth are acceptable in terms of resolutions computed in millimeters. Table 4 summarizes the statistical results obtained for the N edges with a polynomial degree of d = 2.

The main errors in the process of line fitting along the screws come mainly from edge extraction due to the low contrast in intensity levels between the implant and the surrounding tissues. This causes the edges to appear broken with discontinuities and branching, resulting in incorrect fits. Figure 10 displays an illustrative example of this issue. Indeed, analyzing the image in Fig. 8a, reproduced in Fig. 10a on its original representation, an apparent lack of contrast between the implant and the surrounding tissues can be seen. Figure 10b displays the resulting edges after applying Step 1 in detecting the implant boundary section. The bottom part displays the borders inside the bounding box once it has been expanded by 10%, according to Step 2 in detecting the implant boundary section.

The left edge of the screw appears correctly segmented, but the right side shows a clear anomaly. The result of this effect is that the left edge appears correctly detected, Fig. 11a yellow line, or at least with minimal deviations from its corresponding ground truth, Fig. 11b green line, while the detection of the right edge, Fig. 11c yellow line, appears erroneously detected with respect to its ground truth, Fig. 11d green line. In conclusion, it is worth mentioning that the null hypothesis was rejected in the first case and not rejected in the second, which allows us to clearly identify the source and origin of the errors made in the adjustment of the lines to the screws.

To address this problem, the 1460 images dedicated to testing the IU process have been all enhanced by applying a linear transformation to each input image so that the intensity values in the range [m-σ, m + σ] are mapped to the range [0, 255], where m and σ are the mean intensity value of the input image and the standard deviation, respectively. Continuing with the illustrative example above, Fig. 12a displays the enhanced image; in (b) the edges extracted with Canny, where both edges were correctly detected; in (c) the right edge correctly detected. With this new test, an improvement of approximately 9% was achieved in all the results shown in Table 4 compared to the results obtained with the same images without enhancement. The degree of rejection of H₀ has improved, with 2.19% of the N edges tested not rejected. The Canny edge extractor (containing smoothing, gradient, nonmaximum suppression and hysteresis thresholding) is sufficiently robust against noise and other artifacts in the images, outperforming other edge extractors, hence its usefulness in this type of image. Furthermore, it is not affected by the relative arrangement of the implants in the upper or lower jaw.

The last analysis is focused on the measurement of the error obtained when computing the percentages of bone resorption based on the identification of the significant points. Again, the clinical authors, using the interactive function from MATLAB [26] that captures pixel coordinates on the images, manually mark and agree on the two significant points (ground-truth points) on each ground-truth edge. Traversing the edge between the two manually marked critical points, we compute the piece of each ground-truth line and its length. The ratio between this length and the full length of the corresponding ground-truth line determines what is considered the correct percentage of bone resorption. This process is identical to the one proposed for the IU method, described in detecting implant boundary sections, but now in relation to the ground truth. Therefore, for each fitted line to each edge of the screws, we obtain two percentage values, one (A) provided by the proposed IU method and another (G) from the ground truth. The difference between the two percentages is defined as the error: E = A − G.

The 1394 images used for testing, without enhancement, are grouped into ten batches, with approximately 140 images per batch and a number of screw edges determined by the number of implants with their edges, making up a number of E percentage differences varying approximately from 340 to 400. If a significant point is not detected with the IU approach, the corresponding error associated with the ground truth is not computed. Therefore, for each batch, we build the variable x_b of observations with E as samples. On this variable x_b, we also apply a t test and define Hypothesis H₁ as follows: “the mean of the differences between the percentages are zero, i.e., both percentages A and G are identical, null error E”. The alternative hypothesis is H₀, expressing that A and G are different with a nonzero error E. We compute the p value for each batch and obtain an averaged p_b-value of 0.0213 over the ten batches, also with the significance level (α) set to 0.05. The estimated statistical mean (m_b) and standard deviation (σ_b) of the errors (E) averaged over the ten batches were 2.63 and 1.28 pixels, respectively. Table 5 summarizes the statistical results of the bone resorption process. Because the p_b-value is less than 0.05, the test is statistically significant. Therefore, from a clinical point of view, we can infer that our automatic approach is able to accurately detect bone loss due to peri-implantitis when it exists and as a direct consequence of an implant. This allows specialists to be aware of the problem, which is particularly relevant in the early stages for possible prevention or in later stages for tracking and therapy. Additionally, considering that the average lengths of implants are approximately 10 mm and that with a radiograph of the type used in our experiments, this same length is equivalent to approximately 160 pixels in the vertical axis, the average error represented by m_b is approximately 0.17 mm with σ_b of approximately 0.08 mm. Thus, according to the study in [43], if the average annual bone loss is approximately 0.4 mm, with these errors, we are in a very good position for early bone loss detection. With respect to the strategies [8] and [20] mentioned in the introduction and Table 1, with the same purpose, it should be noted that they both use deep learning–based object detectors, Mask R-CNN and Faster R-CNNN, to extract the key points from the feature maps in convolutional layers, while we also use an object detector (YOLOv3) in the DL part only to roughly locate the crowns and screws and then the process described in IU for the precise identification of the significant points. This is why the identification of the required points of interest from the feature maps provided by the convolutional layers was not successful in our radiographs.

During the detection of the significant points, we identified two clear sources of error. When there is no or very little apparent level of bone resorption and when the transition in the intensity profile is not abrupt but gradual, both cases were related to the process described in Step 3 of detecting the implant boundary section. The first source of error appeared when bone resorption was nonexistent (or normal with low impact), and the procedure simply failed to detect the significant points; see Fig. 8a as an illustrative example of this situation. This is because according to the procedure described in detecting the implant boundary section, Step 3, no abrupt changes in intensity levels appeared due to tissue degradation, and therefore, no change in slope in the intensity profile is identified, as illustrated in Fig. 6. In these cases, the significant points are not identified without affecting rejection or acceptance of the null Hypothesis H₀. From a clinical point of view, this is not very problematic due to the absence of bone loss.

The second case arises due to smooth transitions of the intensity level along both edges in the damaged area up to the dark area. This is because the tooth structure, including the tissue, is three-dimensional (3D) and the image is two-dimensional (2D), i.e., this results in a loss of existing information in the 3D structure when mapped onto the 2D image. The heads of arrows in Fig. 13a illustrate this situation. The identification of the significant point corresponding to the transition is incorrectly identified, Fig. 13b, against what is a correct detection (ground truth), Fig. 13c. If the same enhancement is applied to the image, as previously indicated in Fig. 13d, the doubtful transition zone disappears, but a larger displacement than the real one is produced, as shown by the double arrow between Figures (c) and (d), also causing an erroneous detection of the critical point. During testing, these misdetections clearly contribute to the nonrejection of the null Hypothesis H₀. From a clinical point of view, given that there is bone loss, there is detection even if the extent of degradation is not exactly what is desired. The method for correcting this error is to place the patient’s jaw in the correct position in front of the capture device so that the 3D part is aligned in the image.