Clinical evaluation of fully automated thigh muscle and adipose tissue segmentation using a U-Net deep learning architecture in context of osteoarthritic knee pain

Thigh muscle deficits [1, 2] and accumulation of (local) adipose tissue [3‐5] are important pathophysiological events in the context of the clinical science and management of musculoskeletal diseases such as knee osteoarthritis (OA) [1]. Muscles play an essential role in stabilizing the joints [1, 6], while excessive adipose tissue may induce a chronic inflammatory state by producing adipokines and inflammatory cytokines. Both are suggested to be involved in cartilage degradation, synovial inflammation, and bone erosion [3, 4]. Magnetic resonance imaging (MRI)-based analysis is increasingly used to study the association between thigh muscle and adipose tissue composition with knee OA [7‐12]. Further, it has permitted to investigate the impact of training interventions on thigh tissue composition [13], as well as on functional and clinical outcomes of knee OA [5, 14]. Yet, evaluation of thigh muscle morphology and adipose tissue composition requires image segmentation, with the time needed for manual segmentations of thigh muscle and adipose tissue cross-sectional areas (CSAs), precluding the analysis of large databases and image repositories such as the Osteoarthritis Initiative (OAI) [15].

There exist several semi-automated [16‐21] and fully automated [22‐25] tools for thigh tissue volume and CSA segmentation to overcome the challenges in capturing the complex morphology and texture of thigh muscle and adipose tissue that are complicated by considerable inter-subject variability (Fig. 1) and potentially also artefacts as intensity distortions. Imperfections in MRI systems and interactions between the imaged subject and the electromagnetic field cause the sensitivity and hence, the image intensity scale to vary over the image.

Therefore, in previous published studies mainly continuous methods [18, 19, 24, 26, 27] and/or applied intensity inhomogeneity correction prior to segmentation were applied [19, 21, 22, 24]. Several of the discrete methods used k-means [19, 27], fuzzy c-means clustering [24, 28] for adipose tissue classification and focused on atlas-based segmentation methods for the segmentation [18, 24, 25, 27] of individual muscle heads or lean muscle tissue from whole MRIs. Similar segmentation techniques have also been applied to other body tissues [29‐31]. With more data becoming available and recent advances in machine learning and computing infrastructure, segmentation techniques based on deep convolutional neural networks (CNN) are emerging as the new state-of-the-art [32, 33]. For this reason, CNNs are recently examined for musculoskeletal tissue segmentation of the knee joint [34‐36] and thigh muscle MRIs by Ahmad et al. [37] and our group [38]. Ahmad et al. explored five pre-trained fully convolutional networks (FCN) with initiated weights for transfer learning for the quadriceps muscle (including the femoral bone and the medulla). The authors reached high dice similarity coefficients (DSC) of 0.95. While in the above paper [37], the FCN-8s showed the most accurate results, a modified 2D U-Net architecture achieved an ever better performance, when applied to segmentation of cardiac MRI data [39]. While, in general, these CNN methods show great potential for thigh muscle and adipose tissue segmentation, particularly in large clinical image repositories; it is important to demonstrate whether clinical observation can be reproduced using segmentations generated and thus to “clinically” validate the methodology developed. More specifically, our study aims to close this important gap between medical imaging innovation and its clinical application by reproducing a clinical effect observed in a previous published study and by comparing effect sizes. A fully automated segmentation method that is clinically validated will enable segmentation of large imaging repositories such as the OAI, where the MRIs of several thousand patients can be analyzed and used for the development of imaging biomarkers and ultimately the resulting diagnosis and treatment of diseases.

The aim of the current study was therefore: (i) to determine the agreement between a fully automated thigh muscle and adipose tissue segmentation method based on a 2D U-Net technique vs. manual segmentation and (ii) to test whether a previously published clinical study can be reproduced using the CNN algorithm [40] that has shown that patients display lower quadriceps CSAs in limbs with frequently painful knees compared with pain-free contralateral limbs. The latter “clinical evaluation” is particularly important to show the value of the fully automated algorithm for detecting small differences between groups and within subjects over time.

Figure 3 shows examples of different segmentations and some of the variability observed in the training dataset.

The agreement between manual and U-Net segmentation on the technical evaluation set (segmented by the same readers who’s segmentations had been used in the training and testing of the U-Net) was consistently high for all segmented structures (overall DSC (mean ± SD): 0.96 ± 0.02, overall ASSD: 0.004 ± 0.001, overall HD: 0.022 ± 0.001, Table 1). The DSC agreement was particularly high for the SCF, the quadriceps, the hamstrings, the femoral bone circumference, and the sartorius, and somewhat lower for femoral medulla, adductors, and IMF (Table 1).

The agreement between manual and U-Net segmentation, using training and evaluation data from different readers was also very high with an overall DSC of 0.94 ± 0.04 and overall ASS of 0.006 ± 0.004 cm² and an overall HD of 0.085 ± 0.097 cm² (Table 2). Yet, the measures of similarity were slightly lower than for the technical evaluation obtained from data from the same reader (Table 1).

In the technical evaluation (Fig. 4), the Bland–Altman analysis applied to data from the 1st reader showed a high agreement between the automated and manual segmentation results with a difference (absolute values in cm² (percent values relative to size of structure)) of − 1.1 cm² (− 0.8%) for the SCF, + 0.8 cm² (+ 2.1%) for the quadriceps, + 0.3 cm2 (− 1.0%) for the hamstring, + 0.2 cm² (+ 1.5%) for the adductors, and + 0.02 cm² (+ 0.7%) for the Sartorius and -0.5 cm2 (− 3.0%) IMF CSAs (Fig. 4). When applied to data from the 2nd reader, the Bland–Altman analysis showed good agreement between the automated and manual segmentation results with a difference (absolute values in cm² ( percent values relative to size of structure) of − 1.2 cm² (1.5%) for the SCF, − 1.0 cm² (− 2.0%) for the quadriceps, − 0.45 cm² (− 1.3%) for the hamstring,  + 0.2 cm² (+ 1.5%) for the adductors, and + 0.02 cm² (+ 0.7%) for the Sartorius CSAs (Fig. 4). Some systematic deviations between the automated and the manual segmentation methods were observed with + 2.3 cm² (+ 14.6%) for the IMF CSAs (Fig. 4).

In the clinical evaluation study (Fig. 5), painful knees displayed significantly lower quadriceps CSAs for analyses performed with both segmentation methods (manual:  − 5.7 ± 7.9%, p < 0.001, effect size: 0.69; fully automated: − 5.6 ± 7.6%, p < 0.001, effect size: 0.73) than painless contralateral knees (Table 3). The CSAs of the hamstrings and adductors, in contrast, did not show any significant side differences using either segmentation method (Table 3). No statistically significant differences were observed for the other muscle groups, the SCF or the IMF between both segmentation techniques, but as can be appreciated by Table 3, the mean values and standard deviations on either side were very similar between both methods. Values of IMF CSAs obtained from the automated method tended, however, to be somewhat (approx. 15%) smaller than the CSAs obtained from manual segmentation (Table 3).

The aim of the current study was to evaluate a rapid fully automated U-Net segmentation method for thigh muscle CSA segmentation from MRI that is suitable the analysis of large imaging databases in clinical trials. For this purpose, we evaluated the agreement between the fully automated segmentations and previously performed manual segmentations using data from the reader that performed the manual segmentations used to train the U-Net and using data from a second reader that were not part of the training or validation set. In a second step, the current U-Net segmentation method was able to reproduce the results from a previous clinical study, in which we had observed that the quadriceps of limbs with frequently painful knees shows lower CSAs compared with contralateral knees without knee pain.

The results from the current study showed high agreement (DSCs > 0.95) between the fully automated U-Net vs. manual segmentation approach for SCF, quadriceps, hamstrings, and femoral bone segmentations, independent of whether the algorithm was compared to the segmentations of the same or a different reader. The agreement for adductors, medulla, and sartorius was still high, but slightly lower (DSCs > 0.91) and in a similar range for both readers. The agreement for IMF was still good (DSCs > 0.90), when the U-Net was applied to segmentations from the same reader and was notably lower (DSCs > 0.80), when the U-Net was applied to segmentations from a different reader. This was consistent with the outcome of the Bland–Altman plots: the fully automated method showed a good agreement with the manual segmentations from both readers for most of the structures. Only the measurement of the IMF CSAs showed a considerable bias, when the U-Net was applied to data from a different reader. As outlined previously, the method presented here is fully automated and not dependent on a specific reader. The difference between automated and manual segmentation observed here, however, is well in the range of the interobserver variability of manual segmentation reported in previous studies [45]. When applied to the data from the clinical study with unilateral pain subjects, the proposed fully automated algorithm detected similar side differences in quadriceps CSAs, but with substantially less time needed for the analysis (< 1 s) than for current (semi-) automated (3–6 min) or manual segmentation techniques (60–90 min) depending on the reader and the image quality.

Prescott et al. used a numerical analysis-based level set approach and reported DSCs of 0.69 ± 0.16 (vastus medialis) − 0.82 ± 0.08 (vastus lateralis) in the individual quadriceps heads [17]. Trotter et al. focused on the individual quadriceps heads as well, reaching a DSC of 0.87 ± 0.11 for the fully automated multi-atlas framework [18]. Baudin et al. reported an average DSC of 0.86 ± 0.07 for individual thigh muscle heads combining a statistical shape atlas with a random walks graph [26]. Andrews et al. presented a probabilistic shape model framework and reported a mean DSC of 0.81 ± 0.07 for the segmentation of all individual thigh muscle heads [23]. Yang et al. used a voxel classifier combined with morphological operations in four contrast Dixon MR images. The authors reached a DSC of 0.96 ± 0.03 for the SCF, 0.80 ± 0.03 for the IMF and 0.97 ± 0.01 for the combined thigh muscles [22]. Karlsson et al. based their work on a multi-atlas segmentation approach for the muscle tissue segmentation from the whole body and reached a true positive classification from 0.93 ± 0.01 to 0.93 ± 0.03 [24]. Orgiu et al. introduced a discrimination of muscle and adipose tissue from T1-weighted MRIs of the thigh using a fuzzy c-mean algorithm and morphologic operators reporting a mean sensitivity above 96%, mean relative area difference of 1.8%, 2.7%, and 2.5%, respectively [28]. A first attempt for quadriceps MRI segmentation based on deep neural networks was undertaken by Ahmad et al. [37]. The authors explored five pre-trained deep learning models FCN-AlexNet, FCN-32s, FCN-16s, FCN- 8s with initiated weights for transfer learning and PSPNet with two different optimizations as Stochastic Gradient Descent and ADAM for quadriceps (including femoral bone and medulla as one large segmentation label), where the FCN-8s showed combined with the ADAM with quick processing time for inferencing as the best all-around deep learning model with a DSC of 0.95 for the quadriceps and processing time of 0.117s per image. In our study, we obtained an average DSC of 0.98 for the quadriceps, both when using evaluation data segmented by the same or by another reader (in relation to the training dataset) and hence, were able to improve upon this previous approach. Also, we included other and far more complex thigh MRI structures, such as the other muscle groups and IMF, and reached an overall DSC of 0.96 for an evaluation dataset from the same reader and an overall DSC of 0.94 for that from the a different reader, improving upon current state of the art.

More importantly, in the previous approaches [17, 18, 22-24, 26, 28, 37] the performance was not evaluated in the setting of a clinical study. The ability to reproduce relatively small side differences in the quadriceps muscle shown previously is promising for the application of the automated method in future studies, in particular for muscle and adipose tissue of the thigh that are in focus in knee OA [2, 13].

A potential limitation of the study is that the proposed fully automated segmentation method was trained only for a particular anatomical location (33% level of the femoral bone: distal–proximal) and not for other CSAs or a volumetric analysis. However, muscle CSAs acquired at the 33% level were shown to be strongly correlated with 3D muscle volume [46] and found to be sensitive to longitudinal change or cross-sectional differences in several clinical studies [12, 47, 48]. In addition, a longitudinal reduction of CSAs acquired at 33% of the femoral length was shown to be associated with muscle strength loss in patients with concurrent increase in KOA pain [10].

Another potential limitation is that the fully automated method showed a bias toward manual segmentation for the IMF that was greater when the method was applied to data from a 2nd reader (14.6%, DSC: 0.80), whose segmentations were not part of the training and validation set, than when applied to data from the 1st reader (− 3.0%, DSC: 0.90) whose data were used to train the U-Net. This difference between two readers is consistent with results from previous studies that reported an interobserver variability between manual segmentations of two different readers of 18.4/ 14.7% for the IMF, before/after quality control, respectively [45]. Yet, the DSC observed for IMF using data from the same reader compares quite favorable to the literature, while the DSC achieved with data from the 2nd reader is still comparable with the best achieved results of 0.80 in a study of Yang et al. using DIXON MRIs [22]. Further, since the observed effect was systematic and therefore similar for all patients, measuring (side) differences or longitudinal change in the IMF may not be strongly affected. Yet, future studies will have to establish the sensitivity to change for IMF and SCF, for instance during weight gain or loss.

The strength of the current study was that it not only assessed the agreement between manual and automated segmentation, but it also showed that the results of a clinical study could be reproduced using this new method.

Our novel approach of muscle segmentation based on a U-Net is shown to be accurate and can thus be applied to fully automated evaluation of large datasets considerably faster (< 1 s) than for current (semi-) automated (3–6 min) or manual segmentation techniques (60–90 min). More importantly, the effect shown in a clinical study that knees with unilateral frequent pain demonstrate lower CSAs of the quadriceps (but not of other thigh muscles) compared with contralateral knees without knee pain was reproduced and showed a comparable effect size to that of manual segmentation.