Background
Methods
Design
Imaging
Coding of MRI narrative reports
Inter-rater reproducibility
Intra-rater reproducibility (test-retest)
Data Analysis
Results
Pathoanatomic category | Reliability index (95%CI) | Coding round 1 | Coding round 2 | Coding round 3 | Coding round 4 |
---|---|---|---|---|---|
Intervertebral disc degeneration | Prevalence*
Kappa
Percentage agreement | 71.7% (50.2-93.1)
0.66 (0.54-0.77)
80.8% (73.8-87.9) | 94.0 (92.0-96.0)
0.94 (0.84-1.00)
95.8 (92.3-99.4) | 80.0 (80.0-80.0)
n/a
98.2 (95.8-100.0) | 83.3 (80.1-86.6)
0.91 (0.80-1.00)
97.5 (94.7-100.0) |
Intervertebral disc bulge | Prevalence
Kappa
Percentage agreement | 73.3% (70.1-76.6)
0.91 (0.78-1.00)
95.0% (94.7-100.0) | 87.0 (83.1-90.9)
0.86 (0.76-0.90)
92.5 (87.7-97.2) | 70.0 (70.0-70.0)
1.00 (0.87-1.00)
100.0 | 48.3 (45.1-51.6)
0.89 (0.76-1.00)
99.2 (97.5-100.0) |
Intervertebral disc herniation | Prevalence
Kappa
Percentage agreement | 40.0% (34.3-45.7)
0.91 (0.73-1.00)
97.5% (94.7-100.0) | 61.0 (59.0-63.0)
1.00 (0.85-1.00)
100.0 | 40.0 (40.0-40.0)
0.98 (0.87-1.00)
96.5 (93.1-99.9) | 45.0 (45.0-45.0)
1.00 (0.82-1.00)
100.0 |
Nerve root compromise | Prevalence
Kappa
Percentage agreement | 13.3% (4.7-22.0)
n/a
97.5% (94.7-100.0) | 42.0 (31.9-52.1)
0.74 (0.49-1.00)
94.2 (90.0-98.4) | 31.7 (23.0-40.3)
0.63 (0.45-0.81)
93.0 (88.3-97.7) | 18.3 (15.1-21.6) 1.00 (0.75-1.00) 99.2 (97.5-100.0) |
Modic change type 1 | Prevalence
Kappa
Percentage agreement | 26.7% (23.4-29.9)
0.86 (0.68-1.00)
95.0% (91.1-98.1) | 18.0 (15.6-20.4)
0.86 (0.60-1.00)
98.3 (96.0-100.0) | 20.0 (20.0-20.0)
1.00 (0.82-1.00)
100.0 | 18.3 (15.1-21.6)
0.86 (0.60-1.00)
100.0 |
Modic change type 2 | Prevalence
Kappa
Percentage agreement | 25.0% (19.3-30.7)
0.76 (0.51-1.00)
97.5% (94.7-100.0) | 10.0 (5.6-14.4)
n/a
98.3 (96.0-100.0) | 20.0 (20.0-20.0)
1.00 (0.82-1.00)
100.0 | 5.0 (5.0-5.0)
n/a
100.0 |
Spondylolisthesis | Prevalence
Kappa
Percentage agreement | 20.0% (20.0-20.0)
n/a
97.5% (94.7-100.0) | 19.0 (17.0-21.0)
1.00 (0.75-1.00)
100.0 | 10.0 (10.0-10.0)
1.00 (0.82-1.00)
100.0 | 0.0 (0.0-0.0)
n/a
100.0 |
Stenosis | Prevalence
Kappa
Percentage agreement | 6.7% (3.4-9.9)
n/a
97.5% (94.7-100.0) | 23.0 (20.6-25.4)
0.84 (0.59-1.00)
99.2 (97.5-100.0) | 25.0 (25.0-25.0)
0.95 (0.74-1.00)
99.1 (97.4-100.0) | 0.0 (0.0-0.0)
n/a
100.0 |
Scoliosis | Prevalence
Kappa
Percentage agreement | 25.0% (25.0-25.0)
1.00 (0.82-1.00)
100.0% | 6.0 (4.0-8.0)
n/a
95.0 (91.1-98.9) | 5.0 (5.0-5.0)
n/a
99.1(97.4-100.0) | 5.0 (5.0-5.0)
n/a
100.0 |
Osteophytes | Prevalence
Kappa
Percentage agreement | 5.0% (5.0-5.0)
n/a
100.0% | 1.0 (0.0-3.0)
n/a
100.0 | 10.0 (10.0-10.0)
n/a
99.1 (97.4-100.0) | 10.0 (10.0-10.0)
1.00 (0.75-1.00)
99.2 (97.5-100.0) |
Facet joint arthrosis | Prevalence
Kappa
Percentage agreement | 0.0% (0.0)
n/a
100.0% | 14.0 (12.0-16.0)
0.68 (0.42-0.93)
95.0 (91.1-98.9) | 20.0 (20.0-20.0)
1.00 (0.82-1.00)
100.0 | 5.0 (5.0-5.0)
n/a
100.0 |
Endplate irregularities | Prevalence
Kappa
Percentage agreement |
#
| 1.0 (0.0-3.0)
n/a
99.2 (97.5-100.0) | 25.0 (19.3-30.7)
1.00 (0.82-1.00)
98.2 (95.8-100.0) | 18.3 (11.8-24.9)
0.74 (0.56-0.92)
97.5 (94.7-100.0) |
Red flags | Prevalence (%)
Kappa
Percentage agreement |
#
| 0.0 (0.0-0.0) n/a 100.0 | 8.3 (1.8-14.9)
n/a
98.2 (95.8-100.0) | 0.0 (0.0-0.0)
n/a
100.0 |
High intensity zone | Prevalence (%)
Kappa
Percentage agreement |
#
| 15.0 (11.9-18.1)
0.84 (0.59-1.00)
99.2 (97.5-100.0) | 26.7 (20.1-33.2)
0.85 (0.67-1.00)
98.2 (95.8-100.0) | 25.0 (25.0-25.0)
1.00 (0.82-1.00)
100.0 |
Paired comparison | Percentage agreement (95% CI) | Kappa (95% CI, p value) |
---|---|---|
Highly trained rater 1 × Trainee clinician 1 | 98.2% (97.5%-98.8%) | 0.87 (0.82-0.91) |
Highly trained rater 1 × Trainee clinician 2 | 97.8% (97.2%-98.5%) | 0.85 (0.79-0.89) |
Highly trained rater 1 × Trainee clinician 3 | 97.5% (96.8%-98.2%) | 0.83 (0.77-0.87) |
Mean
|
97.8% (97.0%-98.7%)
| |
Highly trained rater 2 × Trainee clinician 1 | 98.1% (97.4%-98.7%) | 0.87 (0.82-0.90) |
Highly trained rater 2 × Trainee clinician 2 | 97.5% (96.8%-98.2%) | 0.83 (0.77-0.87) |
Highly trained rater 2 × Trainee clinician 3 | 98.6% (98.1%-99.2%) | 0.91 (0.86-0.94) |
Mean
|
98.1% (96.7%-99.5%)
|
Reproducibility during the formation of the coding rules
Reproducibility in inexperienced clinicians trained with the final set of coding rules
Pathoanatomic category | Reliability index (95%CI) | Inter-rater Coding round 1 | Inter-rater Coding round 2 | Test-retest (intra-rater) |
---|---|---|---|---|
Intervertebral disc degeneration | Prevalence*
Kappa
Percentage agreement | 90.0% (90.0-90.0)
0.99 (0.87-1.00)
99.3% (99.0-100.0) | 90.0% (90.0-90.0)
0.93 (0.82-1.00)
96.7% (95.0-99.0) | -
0.88 to 1.00
97.3% (94.0-100.0) |
Intervertebral disc bulge | Prevalence
Kappa
Percentage agreement | 90.0% (90.0-90.0)
0.96 (0.84-1.00)
98.0% (97.0-99.0) | 85.0% (79.3-90.7)
0.93 (0.80-1.00)
96.7% (95.0-99.0) | -
0.79 to 0.96
94.3% (91.0-97.0) |
Intervertebral disc herniation | Prevalence
Kappa
Percentage agreement | 58.3% (55.1-61.6)
0.94 (0.80-1.00)
98.7% (98.0-99.0) | 58.3% (55.1-61.6)
0.94 (0.80-1.00)
98.7% (98.0-100.0) | -
0.86 to 0.91
97.3% (97.0-98.0) |
Nerve root compromise | Prevalence
Kappa
Percentage agreement | 53.3% (50.1-56.6)
0.76 (0.58-0.94)
96.0% (94.0-98.0) | 55.0% (49.3-60.7)
0.81 (0.63-0.99)
97.3% (96.0-98.0) | -
0.61 to 0.86
95.3% (94.0-98.0) |
Modic change type 1 | Prevalence
Kappa
Percentage agreement | 20.0% (20.0-20.0)
1.00 (0.82-1.00)
100.0% (100.0-100.0) | 21.7% (18.4-24.9)
0.93 (0.75-1.00)
99.3% (99.0-100.0) | -
0.89 to 1.00
99.7% (99.0-100.0) |
Modic change type 2 | Prevalence
Kappa
Percentage agreement | 10.0% (10.0-10.0)
1.00 (0.75-1.00)
100.0% (100.0-100.0) | 8.3% (5.1-11.6)
n/a
99.3% (99.0-100.0) | -
1.00 to 1.00
99.7% (99.0-100.0) |
Spondylolisthesis | Prevalence
Kappa
Percentage agreement | 20.0% (20.0-20.0)
1.00 (0.75-1.00)
100.0% (100.0-100.0) | 20.0% (20.0-20.0)
1.00 (0.75-1.00)
100.0% (100.0-100.0) | -
1.00 to 1.00
100.0% (100.0-100.0) |
Stenosis | Prevalence
Kappa
Percentage agreement | 16.7% (13.2-19.9)
n/a
99.3% (99.0-100.0) | 15.0% (15.0-15.0)
n/a
100.0% (100.0-100.0) | -
n/a
99.7% (99.0-100.0) |
Scoliosis | Prevalence
Kappa
Percentage agreement | 5.0% (5.0-5.0)
n/a
100.0% (100.0-100.0) | 5.0% (5.0-5.0)
n/a
100.0% (100.0-100.0) | -
n/a
100.0% (100.0-100.0) |
Osteophytes | Prevalence
Kappa
Percentage agreement | 0.0% (0.0-0.0)
n/a
100.0% (100.0-100.0) | 0.0% (0.0-0.0)
n/a
100.0% (100.0-100.0) | -
n/a
100.0% (100.0-100.0) |
Facet joint arthrosis | Prevalence
Kappa
Percentage agreement | 15.0% (15.0-15.0)
1.00 (0.82-1.00)
100.0% (100.0-100.0) | 15.0% (15.0-15.0)
0.92 (0.74-1.00)
98.7% (98.0-100.0) | -
0.88 to 1.00
99.3% (98.0-100.0) |
Endplate irregularities | Prevalence
Kappa
Percentage agreement | 5.0% (5.0-5.0)
n/a
100.0% (100.0-100.0) | 1.7% (0.0-4.9)
n/a
99.3% (99.0-100.0) | -
n/a
99.3% (99.0-100.0) |
Red flags | Prevalence
Kappa
Percentage agreement | 0.0% (0.0-0.0)
n/a
100.0% (100.0-100.0) | 0.0% (0.0-0.0)
n/a
100.0% (100.0-100.0) | -
n/a
100.0% (100.0-100.0) |
High intensity zone | Prevalence
Kappa
Percentage agreement (%) | 15.0% (15.0-15.0)
1.00 (0.75-1.00)
100.0% (100.0-100.0) | 15.0% (15.0-15.0)
1.00 (0.75-1.00)
100.0% (100.0-100.0) | -
n/a
100.0% (100.0-100.0) |