Quantifying cervical spondylosis: reliability testing of a coherent CT-based scoring system

Patients were examined in a Philips Brilliance 64-slice CT scanner. A special study protocol was designed since the patients included were also eligible to participate in a related study at the same institution. The study CT protocol was extended craniocaudally to include the clivus and sternal tip. The expanded FOV was compensated with a low radiation dose profile with CTDI_vol of around 3.8.

Patients aged > 18 years that were admitted to the emergency department at Södersjukhuset, Stockholm, Sweden for neck pain after a motor vehicle accident were included in the study. Those requiring medical imaging in the emergency setting according to the Canadian C-spine rules [23] underwent a CT of the cervical spine. The patients whom the examining physician deemed not to require medical imaging were later contacted by the research team and offered to participate in the study. If they accepted, they were also examined with a CT of the cervical spine. All patients had been contacted and gave their informed consent prior to the CT scan.

The statistical software package SPSS 22 (SPSS Inc., Chicago, IL) was used for analysis of both intra-rater and inter-rater agreement. An additional SPSS macro was used to compute the inter-rater agreement level for categorical data between multiple observers [24]. The Kappa values were considered significant if p < 0.05. The intra-rater agreement was assessed with intra-class correlation coefficient (ICC).

The interpretation guideline established by Landis & Koch [25] was used to evaluate the strength of inter-observer agreement using the kappa statistic.

Kappa values of > 0.40 were considered representing clinically acceptable level of strength of agreement for a scoring system [26].

The standards for strength of intra-rater agreement proposed by Fleiss et al. [27] were used as a basis for interpretation of the magnitude of the ICC values obtained.

Figures 6, 7 and 8 illustrate the variance in degeneration scores as assessed by the three raters regarding discs, facet joints and total degeneration, respectively (Figs. 6, 7 and 8).

As shown in Figs. 6, 7 and 8, the senior orthopedic surgeon had a consistent tendency to assign the variables a more severe degree of degeneration than both the radiologists. A statistical analysis of the agreement in scorings by the three raters generated kappa values for the variables (Table 2). Anterior osteophytes generated the highest kappa value, i.e. the raters tended to rate the presence and size of anterior osteophytes similarly. Regarding facet joint degeneration, in terms of joint space narrowing the raters were closest to agreement.

To test the intra-rater reliability and thus reproducibility of the scoring system, the senior orthopedic surgeon and the junior radiologist repeated their assessments of the 20 patients after a minimum of 3 months (Table 3). The analysis showed excellent agreement between the two assessments for the total degeneration and at least fair agreement for all parameters with ICC spanning from 0.538 to 0.820.

The kappa value for the overall degree of degeneration showed a substantial agreement. However, this value represents the agreement between the raters when adding their degeneration scores on the separate variables and then dividing the subjects into three separate categories (no degeneration, moderate degeneration or severe degeneration). When grouped together, the rate of disagreement on the separate variables is masked and the agreement when only three categories exist is presumably consequently higher than it would be if more than three categories of degeneration were eligible. This becomes apparent when analyzing the separate variables, where kappa values are considerably lower. In two of the variables the null-hypothesis could not be rejected (endplate sclerosis and facet joint osteophytes). The only variable where the strength of agreement was substantial was anterior osteophytes. This variable is weighted to contribute less to the disc degeneration score than height loss, which only reached a moderate strength of agreement. The fact that agreement for the total level of degeneration was higher than for the separate variable could be explained by compensation mechanisms of the individual rater. For example, a borderline case of facet joint osteophytes could have been neglected with a compensatory affirmation of borderline irregularity of the articular surface.

Our study showed similar results with those of the study of Walraevens et al. [16] concerning facet joint degeneration, even though our classification criteria differed, with low strength of agreement on osteophytes and irregularity of the articular surface but slightly higher strength of agreement for joint space narrowing.

However, when applying the radiograph-based scoring system for disc degeneration to CT there seemed to have been a slight loss of reliability compared to Walraevens et al. [16]. They showed “good” or “excellent” agreement on the disc degeneration variables apart from endplate sclerosis which was low in both studies, whereas our results ranged from “moderate” to “substantial” with a slightly lower level of agreement overall. However, the trend is clear; assessing endplate sclerosis, facet joint osteophytosis and irregularity of facet joint articular surfaces is more complex than the three other variables.

Considering a cut-off limit of 0.40 for strength of agreement, which is arbitrarily set, many of our obtained kappa values indicate an acceptable or good level of agreement. However, several Kappa values were below 0.40. There are a few reasons for the relatively low values that must be considered. First, the relatively small sample size could have affected the level of agreement. Another factor might have been the multi-segment assessment. Determining the spinal segment with the highest level of degeneration is an assessment by itself. It is plausible that the raters were in fact reviewing different segments and consequently assessing them differently. Lack of training among the raters might also affect the level of agreement. In this material, the raters deliberately had no joint training session of the scoring system prior to the assessment procedure. This was to simulate a clinical setting to a high extent.

The goal of developing a scoring system that is easily applicable and experience- and discipline independent is of importance. However, we believe minor modifications could be done to improve the scoring system while still keeping it user-friendly. For example, one source of disagreement on the height loss-variable may have been presence of endplate compression affecting the disc height.

The ICC-values obtained all indicated fair, good or excellent intra-rater agreement, with total degeneration scores having the strongest agreement for both raters. However, the confidence intervals were large and the true ICC-values thus hard to discern. They are interpreted to originate from the variation between examiners using an ordinal scale on a relatively small material. Only two of the raters participated in the intra-rater reliability part of the study. As in the inter-rater analysis, the agreement of the total degeneration score was higher when summing disc degeneration and facet joint degeneration scores.

In comparison with other scoring scales in the field the agreement is regarded equivalent. Considering inter-rater reliability of the assessment of disc degeneration, previous scales vary from 0.41–0.78 [16, 28, 29] intra-rater reliability of the discs vary from 0.71–0.86 [16, 29]. In the material reviewed, the inter-rater agreement for facet joint degeneration the agreement varied from 0.43–0.49 [15, 16] and the intra-rater agreement from 0.57–0.72 [15, 16]. When comparison is made, one must consider the different radiologic modalities that are used in previous materials.

In summary, our results indicate a well acceptable level of agreement regarding both inter-rater and intra-rater reliability of a CT based scoring system, especially addressing facet joint degeneration and overall degeneration. The findings enable a role for this scoring system in both future research and clinical practice. However, when analyzing individual parameters in the scores, the agreements were lower than in the total scores. Hence, we recommend the system to be clinically applied in its aggregated form to assess disc degeneration, facet joint degeneration and overall degeneration.

This study has a few limitations. First, the sample size is rather small and for wide clinical implication, future studies with larger material are required to confirm the results.

Second, the study population in this this material consists exclusively of post-traumatic patients. This makes it less representable for the general population and is neither to be considered an asymptomatic cohort nor a cohort with non-specific neck pain. We welcome further investigations in a different clinical setting to validate the scoring system.