Evaluating the accuracy of automated cephalometric analysis based on artificial intelligence

Cephalometric analysis is essential in every stage of orthodontic treatment. However, manual analysis has low consistency between repetitions and takes a great deal of time. With the rapid development of technology, computer vision allows machines to understand and work with images for cephalometric landmark detection after being trained with a given dataset. In contrast to conventional manual methods, AI can assess images within seconds, helping users save time and increasing the efficiency of repetitive work [11, 15]. According to our study, humans take an average of nearly 80 times as long as the AI program to complete the task. Using an automatic program saves a great deal of time and frees up orthodontists for other tasks. Additionally, the accuracy of cephalometric landmark digitization depends largely on the operator. The results of tracings by different operators or even repeated tracings by the same operator are likely to be significantly different [22]. In contrast to computer-assisted manual tracings, the results obtained from the same automatic program are fairly stable, showing its superiority in terms of repeatability [15]. Nevertheless, whether the accuracy of manual and AI cephalometric analyses is acceptable is always an important concern. To date, growing numbers of studies have focused on the accuracy of the automated cephalometric software, which is more accessible to the practicing orthodontist than in-house algorithms [16‐20]. Therefore, our study is practical and meaningful for both the users of automatic programs and the engineers who seek to improve the algorithms.

The results regarding accuracy in landmark detection are consistent with published studies [9, 10]. The most accurately defined points, such as Prn, U1 tip and Pg, are on the edges of anatomical structures with sharp margins on CBCT images. Automatic programs are capable of identifying them with a high degree of precision similar to that of humans. The MREs of the U1 root (4.14 ± 1.33 mm), Go (3.08 ± 1.67 mm), L1 root (2.85 ± 0.93 mm), N (2.63 ± 2.31 mm), S (2.61 ± 1.88 mm), Me (2.41 ± 0.65 mm), D (2.27 ± 0.86 mm) and P points (2.24 ± 1.14 mm) exceeded the average value (2.07 ± 1.35 mm). The great deviation of the U1 root and L1 root points may be due to the indistinct outlines of the incisor roots [10, 23]. The recognition of incisor roots is obstructed by other dental roots when patients have anterior tooth crowding or overbite malocclusion, which is also a challenge for orthodontists in clinical work. The mixed density of incisor roots on imaging makes them difficult to identify by manual or automatic analysis. The reason for the significant errors in the Go, Me and P points is probably the asymmetrical structure of the lower mandible margin and external auditory canal, which varies over a wide range in the population. Midpoints of bilateral structures are chosen when the left and right sides cannot overlap, while asymmetrical structures increase the complexity of the region. Localizing these points relies heavily on the personal judgement of the operator. Obstruction by adjacent anatomical structures may also affect the localization of the Go point, which increases the difficulty of detection. As for the S, N and D points, the relevant structures are sometimes vague and quite ambiguous in RLCs, which makes them difficult for automatic programs to locate. The distance in landmark position between manual localization and automatic localization is interpreted as follows: <2 mm is correct, and < 4 mm is acceptable [24]. In this study, the average SDR as defined by MRE within 1 mm, 2 mm, 2.5 mm, 3 and 4 mm were 18.82%, 58.58%, 71.70%, 82.04% and 91.39%, respectively. Thus, more than half of the points were localized correctly, and more than 90% of the points were localized acceptably well.

The scatter plots (Fig. 2) indicate the distribution of landmark localization error by distance and direction. The 95% confidence ellipse shows a two-dimensional expansion of the Bland-Altman plot to observe the correlation between the x- and y-axis errors in the shape of the ellipse [15]. Points such as Or, UL and LL were distributed along the x-axis, which implied mainly horizontal bias in automatic detection. Similarly, points such as N, B, Gn and Pg’ illustrated mainly vertical bias. The anatomical positions of cephalometric landmarks are subject to bias in both the x and y coordinates. Points with considerable horizontal bias or vertical bias are usually ambiguous to locate in the corresponding direction to a certain extent. The results of AI detection are consistent with those of manual detection.

For cephalometric measurements, the rate of consistency within the 95% limits of agreement was between 91.76% and 98.82% (Table 3), which indicated high consistency in these measurements. Also, the Bland-Altman plots (Fig. 3) visualize the consistency between paired measurements. The results essentially corroborate those of a previous study [17]. The range between the limits of agreement is wider for dental measurements than for skeletal or soft tissue measurements. This conclusion is consistent with our conclusion regarding landmark identification. There were a total of 8 measurements that were both statistically significant (P<0.05) and had mean differences above 2 mm or 2°. Dental measurements have more discrepancy in comparison with manual tracing. Soft tissue measurements are basically equivalent in the two methods. Since the RLCs selected in this study were all from people who had not yet undergone orthodontic treatment, the precision of dental measurements might increase if the automatic program were applied to post-treatment images.

Generally, the consistency of AI cephalometric measurements is almost at an acceptable level, considering that it is nearly 80 times faster than manual cephalometric analysis. The main results that were of interest to us proved fairly good precision and consistency with manual analysis. Only a few measurements showed significant bias. Thus, human supervision is still necessary for automatic landmark identification. The solutions for increasing the accuracy and efficiency of automatic cephalometric analysis may be to improve the quality of the image like sharpening and algorithm approving and to implement an outlier detection and feedback system [23]. Orthodontists would need to spend only a short time checking the results of automatic localization, which could save them a great deal of time during clinical work.

Despite its strengths, this study also has several limitations. We used initial diagnosis RLCs to assess the automatic program. Diverse and complex intraoral conditions, such as overlapping teeth, overjet and overbite, are complex and make it difficult for AI to recognize relevant landmarks and obtain measurements. More detailed classification of image samples would increase the comprehensiveness of the conclusions. In addition, we selected fewer soft tissue landmarks than skeletal or dental landmarks, and the same was true of soft tissue measurements versus skeletal and dental measurements. The results would be more reliable if more soft tissue indicators were included.