Automated spinopelvic measurements on radiographs with artificial intelligence: a multi-reader study

This study was approved by the institutional review board (approval number 18–399). Informed consent was waived due to the retrospective and non-interventional nature of the study. All procedures were conducted in accordance with the Declaration of Helsinki (as revised in 2013).

All imaging data were queried retrospectively from our institutional Picture and Archiving System (PACS) for patients receiving lateral radiographs of the whole spine between September 2012 and May 2019. At our institution, these are acquired with the EOS System (EOS Imaging, Paris, France), where a vertically moving X-ray source allows for distortion-free images in a standing position under weight-bearing conditions [25], without the need for stitching conventional images and with very low radiation dose [26].

Inclusion criteria were: i) lateral images of the full spine in a standing position, and ii) acquisition ranging cranial from the external auditory canal and caudal to the femoral heads. Exclusion criteria were: ii) incomplete capturing of the spine, ii) motion artifacts during image acquisition, and iii) severe spinal deformities, defined by the presence of hemivertebrae or moderate to severe scoliosis (Cobb angle ≥ 25°), [27] which would cause superimposition artifacts on lateral images. Otherwise, cases with foreign material, e.g., from spinal instrumentation or hip replacement, and vertebral fractures were all included to represent clinical reality.

This resulted in a set of 295 images for algorithm development, randomly split into 80% training data and 20% for internal validation and optimization. Another randomly selected test set of 65 images was completely withheld from training and used for subsequent performance evaluation of the algorithm.

Five frequently used clinically relevant spinopelvic measurements on lateral radiographs were used for the current study, as defined in the following. The thoracic kyphosis (TK) of the thoracic spine is theoretically considered the angle between T1 and T12, which is in practice often impossible to delineate due to the superposition of the humeral heads [1], and therefore in clinical routine and most previous studies alternatively measured from the superior endplate of T4 to the inferior endplate of T12 (normal: 30–50°) [28], which we adopted for our study. More agreement exists on the lumbar lordosis (LL), measured from the superior endplate of L1 to the superior endplate of S1 and denoted as negative values (normal: –60 to –50°) [28], the sacral slope (SS), measured between the superior endplate of S1 and a horizontal line (normal: 35–45°) [28], and the sagittal vertical axis (SVA), measured as the horizontal distance from the plumb line of the center of C7 to the superior endpoint of S1, with positive values in front of S1, and negative values behind (normal: ± 20 mm) [28].

Collected training data were annotated for anatomical landmarks by two extensively trained annotators on a dedicated internal platform. Annotations included the four corners of each vertebra (C2-S1, 94 labels per case, total of ~ 27.730 labels), and all labels were reviewed by a radiologist with strong experience in orthopedic radiographs. The annotated images were augmented manifold (including rotation, translation, cropping, resizing, blurring, and distortion) and used to train a two-staged neural network. First, a region-based convolutional neural network (R-CNN) was used to automatically detect the whole spine centers from the images in a single shot. Then, the original image was sliced into small areas containing the vertebrae based on the detected spinal centers, and a second R-CNN was used to automatically detect the four vertebral corners. Finally, the predicted landmarks were post-processed with geometric formulae to automatically calculate the above-mentioned spinopelvic angles and measurements and displayed over the input images to allow visual inspection of the results.

Six readers independently measured the validation data set manually, after receiving a verbal and written introduction with all the specifications for the measurements. Readers included three radiologists (B.F.H, B.O.S., and J.Rue., with 3, 6, and 3 years of experience; referred to as “Readers R1-R3”) and three orthopedic surgeons (A.C.-K., S.W., and W.H., with 3, 8 and 5 years of experience; referred to as “Readers S1-S3”). As we found that even the most experienced reader would occasionally over- or underestimate a single reading (Supplemental Fig. S2), the median of the readers was used as the consensual reference standard. For an even number of values (from six readers), the median is defined as the arithmetic mean of the two middle values, when ordered from lowest to highest. This further implies that no single reader alone could establish the reference standard for any given case.

Quantitative measurements were expressed as mean and standard deviation (± SD); while, categorical variables were expressed as counts and percentages. Normal distribution of the measurements was confirmed with histogram plots (Supplemental Fig. S1). Performance of the AI algorithm and the readers was evaluated using the mean absolute error (MAE) with standard deviation (SD), Pearson’s correlation (r) with the reference standard, Bland–Altman plots for mean difference visualization, and cumulative distribution function (CDF) to evaluate performance up to clinically relevant thresholds and calculate a normalized area under the curve (AUC) for further quantification. Intraclass correlation coefficient (ICC) was calculated based on a two-way random-effects model with absolute agreement [29]. Agreement was defined as previously described: ICC ≥ 0.90 as excellent, 0.90–0.75 as good, 0.75–0.50 as moderate, and ≤ 0.50 as poor [30].

To test for any significant differences between AI and human readers, analysis of variance (ANOVA) of the errors was performed with Tukey’s post hoc test, if needed. All analysis and visualizations were performed in Python (Version 3.12, Python Software Foundation), with recent SciPy, statsmodels, and seaborn libraries. Two-sided significance testing was conducted with an α of 5% (p < 0.05) (Fig. 1).

The evaluation cohort consisted of 65 patients (39 female; 60.0%), from all different age groups (mean age: 47.8 [± 24.1] years, range: 7–85 years), with a wide range of (pathologic) spinopelvic measurements. Over a third of patients had vertebral fractures (23; 35.4%), and about a quarter of patients had previously undergone spinal instrumentation (16; 24.6%), reflecting the broad spectrum of clinical reality (Table 1).

All 65 cases were successfully processed by the AI (100.0% success rate), including special cases with vertebral fractures, deformities, or spinal instrumentation (examples in Fig. 2). AI analysis took a maximum of 1 s per case; while, human readers took an average of 139 s per case (range: 86–231 s). Bland–Altman plots showed a low mean difference for all automatic measurements (TK: −0.57; LL: 0.51; SS: 0.59; SVA: 0.39), with no proportional bias (Fig. 3A).

All AI-based results had an excellent correlation with the reference standard (ICC: 0.92–1.00; all p < 0.001), with the highest result for sagittal vertical axis (SVA), and the lowest result for thoracic kyphosis (TK), but for every measurement, the ICC was within the range of the six human readers (Table 2, Fig. 3B). Further, the mean absolute error (MAE) was also within the range of the readers for all four measurements, with the smallest errors for sagittal vertical axis (AI: 2.44 mm [± 3.93]; readers: 1.22–2.79 mm), followed by thoracic kyphosis (AI: 3.14° [± 4.24]; readers: 1.97–3.87°), lumbar lordosis (AI: 3.71° [± 4.68]; readers: 1.86–5.88°), and sacral slope (AI: 4.56° [± 6.10]; readers: 2.20–4.76°) (Table 2).

All these errors were below the threshold of < 5°, which has been shown previously as the normal inter-reader variability and is generally regarded as acceptable [8, 31].

Cumulative distribution function (CDF) enabled more detailed analysis, by incrementally plotting the percentage of cases (y-axis) below a certain absolute error (x-axis), thereby allowing clinically relevant thresholds to be read and calculating a normalized area under the curve (AUC) for quantitative comparison (Fig. 4A). This revealed on a per case analysis, that an error < 5° was achieved in 84.6% of cases for SS, in 70.8% for LL, and in 68.8% for TK; while, a higher threshold of < 10° (as also utilized in previous studies [18]) was reached in well over 90% of cases for all measurements (SS: 96.9%; LL: 90.8%; TK: 98.4%).

Analysis of variance (ANOVA) showed no significant difference between the errors of the AI and the human readers for SS (p = 0.054) and SVA (p = 0.064), but for TK (p < 0.001) and LL (p < 0.005). However, post hoc Tukey’s test revealed no significant differences between AI and any of the human readers, only for some readers between each other (TK: R3 vs. R2, S2, and S3; LL: S3 vs. S1, R2, and R3; all p < 0.05; full results in Supplemental Table S1) (Fig. 4B).