Classification and treatment of proximal humerus fractures: inter-observer reliability and agreement across imaging modalities and experience

Proximal humeral fractures (PHFs) comprise 5% of all fractures in adults and are the third most common fracture in adults over 65 years old [1]. In 1970, Charles Neer II created a classification system for fractures of the proximal humerus, which is widely utilized [2, 3]. However, over the past 2 decades the reliability of Neer's system has been challenged, as multiple studies have reported low inter-observer agreement when attempting to classify PHFs using Neer's system [4‐17] or recommending subsequent treatment [18]. Neer's classification is not alone in this quandary, as many studies have similarly reported disagreement in classification schemes for other types of fractures [19‐21]. It has been postulated that the low levels of agreement is not a limitation of the classification systems itself but rather the surgeons' inability to accurately interpret the images. In fact, Neer himself has rebutted that experience and suboptimal imaging are likely responsible for the lack of agreement in his system [22].

Although some authors have evaluated the effect on inter-observer agreement by adding advanced imaging such as CT scans and three-dimensional (3D) reconstructions, [4, 10, 14, 15, 23‐25] the results have been inconclusive, and none have addressed all of these modalities in terms of both classification and treatment recommendations as a function of physician experience. Thus, to the best of our knowledge, this is the first study to evaluate the inter-observer agreement of multiple imaging modalities: X-ray, CT, and 3D reconstructions on both the classification as well as treatment of proximal humerus fractures in a single study. The secondary study goal was to observe the effect of stratifying agreement based on fracture severity and surgeon experience.

Sixteen proximal humerus fractures were selected and classified by the senior author as four 2-part fractures, eight 3-part fractures, and four 4-part fractures. Each of the 16 fractures had an X-ray (anteroposterior and a scapular-Y lateral), a CT scan, and a 3D CT reconstruction, which resulted in a total of 48 standardized image sets. All images were taken between 2003-2008 at the same institution and drawn from the same PACS system.

After providing a brief review of the Neer classification system, each observer was presented the same 48 image-sets by PowerPoint in random order. They were blinded to any patient demographic information, mechanism of injury, or associated morbidities. Each observer was asked only two questions per set of images: (1) to classify the fracture using the Neer classification, and (2) to determine their treatment of choice. Treatment options were standardized to four choices: non-operative, open reduction internal fixation, hemiarthroplasty or total shoulder arthroplasty. No case demographics were provided.

The observers included orthopedists of varying experience: 8 board-certified attending surgeons (consisting of 4 general orthopedists and 4 upper extremity specialists), 4 senior residents, and 4 junior residents. A general orthopedist was defined as a surgeon practicing all aspects of orthopaedic surgery including the surgical management of PHFs. An upper extremity specialist was defined as a surgeon with fellowship training and a practice focus on the upper extremity whose practice includes the surgical management of PHFs.

In the past two decades, the validity and reproducibility of fracture classification systems has come under greater scrutiny, which has sparked much debate in the orthopedic literature [29‐34]. As a result, subsequent studies have examined the utility of advanced imaging techniques in improving inter-observer agreement, but the results have been inconclusive [4, 10, 14, 15, 22, 25]. In the beginning of this debate, a few studies have concluded that CT scan or three-dimensional reconstruction add very little in pre-operative assessment [4, 10, 14, 15, 22]; however, a study published recently by Brunner et al. has challenged this assertion by showing a consistent increase in inter-observer agreement through the use of stereo-visualization and real 3D imaging [25].

In our series, we examined the effect of advanced imaging, including 3D reconstructions, on fracture classification and treatment and found that inter-observer agreement was less than ideal for both classification and treatment among orthopedic surgeons, which is consistent with the majority of reports in the literature [4‐20, 22]. In a recent review, "slight" to "moderate" agreement has been reported in almost all major studies regarding inter-observer reliability in PHFs [8], and our results indicate the same despite the addition of advanced imaging in the form of 3D reconstructions. However, it should be noted that the comparison of kappa coefficients across studies should be done with caution, as factors such as bias, prevalence, and marginal distributions influence kappa values and can vary at different institutions [35]. Thus, our second goal was to compare overall inter-observer agreement only within our institution and to observe the effect sub-dividing our results by fracture type and observer experience. In doing this, we observed two major trends in our data: 1) the only significant improvement in agreement with advanced imaging was among upper extremity surgeons and 2) the only benefit of advanced imaging was among all users in attempting to classify 4 part fractures. No benefit was witnessed with advanced imaging in order to improve inter-observer agreement on treatment.

Our study showed that the greatest inter-observer agreement was among upper extremity surgeons with 3D reconstruction. Furthermore, none of the other groups of observers had significantly improved kappa scores with the addition of advanced imaging, which may suggest that experience enhances inter-observer agreement in our study. Reports in the literature are nearly split regarding the role of experience. Kristiansen et al. was the first to suggest that low experience accounted for low agreement [17]. Then, Sidor et al. argued against experience by concluding that the three attending physicians had the same agreement as the residents; however, the group of attending physicians was heterogeneous and not all were orthopedic surgeons [11]. Siebenrock et al. studied only shoulder specialists and found that inter-observer agreement with plain films still landed in the "fair" to "moderate" range; they suggested that experience did not improve the kappa score when compared to other studies, but they did not compare the specialists to a control group [13]. Sallay et al. was the first article to refute experience by sorting observers into groups. They measured agreement with both X-ray and 3D reconstructions, but their technology for 3D reconstruction was an earlier version and had lower resolution than in the present study (Figure 4) [10]. On the other hand, a few studies have supported the role of experience. Brorson et al. showed significantly higher confidence intervals in specialists when compared to residents and fellows, and although not explicitly stated as a study aim, Bernstein et al. showed higher absolute kappa values among attending surgeons when compared to residents [4, 7, 8]. As a response to the challenge of the 4-type classification system, Neer commented that inter-observer variability is likely the combination of "suboptimal quality of current imaging and inexperienced interpreters [22]." Our study agrees with Neer's interpretation, as our highest and most significant agreement was observed in both our most experienced observers and most advanced imaging modality. This may suggest that the greatest benefit of advanced imaging is to the upper extremity surgeon; however, despite the improved trend, overall agreement is still less than ideal and therefore not recommended.

4-part fractures showed the greatest inter-observer agreement among all observers for both classification and treatment. The data in this category also trended significantly, as 3D reconstruction was stronger than CT scan, which was stronger than X-ray. However, treatment of 4 part-fractures was most agreeable with CT scan. All other subdivisions of fracture type did not show significant improvement with advanced imaging. These data may suggest that complex 4-part fracture classification could be improved by 3D reconstruction. A few studies have corroborated this assertion: Mora-Guix et al. showed that despite the little overall value of CT imaging, it did improve identification of number of fragments [23]. Additionally, Brien et al. cited that the largest point of contention in their inter-observer study was agreeing upon 4-part fractures, and the surgeons would benefit from CT scans in that regard [5].

The literature regarding inter-observer agreement of fracture classifications appears to converge on the following paradigm: low inter-observer agreement is largely caused by compromised interpretation of the imaging, which is caused by imprecise measurements of the pathoanatomy. Moreover, Neer described patient, procedural, and clinical variability as causes of these imprecise measurements [22], and a few studies have improved precision through education [6, 8, 16]. Important to remember is that "the 4-segment classification is not a radiographic system but is a pathoanatomic classification of fracture displacement [22]." Our study and others have underscored the difficulty in categorizing a 3D concept with 3D images displayed on a 2D screen. Perhaps further studies with experienced users of advanced technology or stereo-visualization need to be evaluated for observer agreement possibly with correlation to intra-operative findings.

There were several study limitations. First, it should be understood that these conclusions are based on an experimental model; thus the distribution of Neer Fractures is not reflective of that which would be experienced in a clinical setting. Furthermore, the observers were not privileged to any patient demographic information, which certainly influences a treating surgeon's decision algorithm. Further studies evaluating agreement of treatment based on a more complete clinical picture would have a broader application. The number of cases (sixteen) presented to the observers was a limitation, and a power analysis was not performed in the selection of this number; however, it was chosen to provide an adequate breadth of cases without resulting in observer fatigue, which might have confounded the results. Additionally, the observers were not able to combine or manipulate images, as they might in a clinical setting. Gonimeters or rulers were also not provided but have been shown to be ignored in clinical setting even when available [11]. The image sets were pre-selected, which imparts a selection bias. Also, treatment comparisons are inherently biased by the observer's comfort level with a procedure and by their experience with the fracture classification, which also may have changed if they were given the opportunity to combine modalities. Observers may have also been limited by their specific experience with 3D technology.

In examining the inter-observer agreement with kappa values for X-ray, CT scan, and 3D reconstruction for fracture classification and treatment, we conclude that although 3D reconstruction showed a slight improvement in the inter-observer agreement for fracture classification among specialized upper extremity surgeons, overall all imaging modalities yielded low inter-observer agreement for both classification and treatment.