Standardized image evaluation in patients with idiopathic normal pressure hydrocephalus: consistency and reproducibility

Descriptive statistics are presented as medians and interquartile ranges. The Mann-Whitney U test and χ² test when appropriate were used to investigate differences between groups, and the significance level was set at p < 0.05. Inter- and intra-rater reliability was calculated using Cronbach’s alpha, interclass correlation, and kappa statistics, including weighted kappa and prevalence-adjusted bias-adjusted kappa. The kappa values were interpreted according to Landis and Koch: 0.01–0.20 slight agreement; 0.21–0.40 fair agreement; 0.41–0.6 moderate agreement; 0.61–0.80 substantial agreement; and 0.81–1.00 almost perfect agreement [25]. Bland-Altman plots illustrated the differences between measurements. Statistical analyses were performed using SPSS version 25 (IBM Corp, Armonk, NY, USA).

Table 2 shows the inter-rater agreement for CT and MRI assessment of the seven radiological features associated with iNPH. Notably, inter-rater agreement was substantial to almost perfect for assessment of Evans’ index (concordant readings for CT and MRI in 97% of the cases); for assessment of the size of the temporal horns (concordant readings for CT and MRI in 83% of the cases); for assessment of narrow sulci (concordant readings in 86% of the cases for CT and in 89% of the cases for MRI); and for assessment of enlarged Sylvian fissures (concordant readings for CT and MRI in 91% of the cases). There was almost perfect inter-rater agreement for assessment of focally enlarged sulci for CT (concordant readings in 91% of the cases) and fair inter-rater agreement for MRI (concordant readings in 74% of the cases). Inter-rater agreement was fair to substantial for assessment of the callosal angle (concordant readings in 74% of the cases for CT and in 83% of the cases for MRI). The inter-rater agreement was fair for assessment of periventricular white matter changes (concordant readings in 60% of the cases for CT and in 57% of the cases for MRI).

Intra-rater agreement (test-retest reliability) was equally distributed for both modalities, with higher values for continuous data (Table 2). Figure 2 shows examples of comparable CT and MRI scans in the same patient in whom the assessment remained the same, independent of the modality. When the results were compared for different modalities for the same reader, the intra-rater agreement was lower for white matter changes and for focally enlarged sulci. Callosal angle assessment showed almost perfect intra-rater agreement for both CT and MRI, and intra-rater agreement was only slightly lower when the two modalities were compared for the same reader (Table 2). However, when comparing the two readers, the inter-rater agreement was fair to moderate, and Bland-Altman plots (Fig. 3) showed a systematic difference between the readers, with reader B having a mean score that was lower by 10°.

When summarizing the parameters of the iNPH Radscale score, the median score was 9 (IQR 8–10) for both readers and modalities. The intra-rater reliability varied from moderate to almost perfect, and the inter-rater reliability, as assessed by weighted kappa, was substantial (Table 3). The intra-rater agreement, as assessed by Cronbach’s alpha, had high values that ranged from 0.79 to 1.0. Comparing readers, Cronbach’s alpha was marginally lower, ranging from 0.79 to 0.85 (Table 3).

With standardized projections, the measurements can be performed in a more uniform way and with high reliability, as shown in this study for Evans’ index and temporal horns. Evans’ index is a robust measurement that showed almost perfect agreement between readers, both in this study and in previous studies (Table 1). Furthermore, the concordance between CT and MRI assessment was high, with an ICC over 0.80.

Measurements of the temporal horns showed higher inter-rater agreement in this study than in earlier publications, especially for CT [16, 19]. Thin sections, proper alignment, and a consensus on where to measure are important factors that increase agreement. Very few studies report the agreement for temporal horns as measured in millimeters, as in the present study, but more often this is reported as mild, moderate, and severe dilation. In this study, the substantial to almost perfect agreement for temporal horns was stable both between modalities and between readers.

The assessment of the callosal angle was robust in terms of test-retest reliability and consistency between CT and MRI. The inter-rater reliability is in line with that reported by Bao et al. [16] but lower than that reported by Miskin et al. [17] and Reinard et al. [19]. However, in Miskin et al., coronal reformatting was performed beforehand and supplied to all readers. In the present study, the two readers individually angulated the coronal reformat perpendicular to the commissure plane, which could affect agreement. The systematic 10° difference between the readers (Fig. 3) is a good example of the importance of harmonization between readers. Not only is the alignment perpendicular to the commissure plane of great importance but also is the placement of the tangent along the ventricular roof. When the callosal angle is small, the curvature of the ventricular roof is more pronounced, thereby making precise measurement of the callosal angle more difficult (Fig. 4).

The assessment of periventricular white matter changes varied to the same extent as in previous studies (Table 1). It was expected that the periventricular white matter changes would have low agreement between modalities, since MRI has a higher sensitivity for detecting white matter changes. However, the clinical relevance of this parameter is uncertain, since white matter changes should not exclude patients from shunting [26]. Furthermore, although periventricular white matter changes are a supporting feature of iNPH according to the diagnostic guidelines, this is also a common finding in elderly patients with vascular disease that is not restricted to iNPH. In the clinical setting, changes over time in the same patient are compared with the same modality of examination, i.e., MRI or CT, to limit discrepancies.

For the three sulci-related parameters, the intra- and inter-rater agreement was substantial for CT and MRI, with one exception. Reader A had lower intra-rater agreement for focally enlarged sulci on MRI than on CT as well as when CT was compared with MRI, which is interpreted as an expression of inexperience. The scale was first developed on CT, and Reader A was more used to assessing that modality. In a patient with borderline compression of sulci, the lower resolution of CT blurs the contours of the sulci, potentially overestimating the narrowing and therefore increasing the likelihood of concordance between readers. MRI defines the border between CSF and gray matter, which hypothetically could make two readers differ in their scoring. Features that involve more subjective assessments require distinct predefined rules to achieve high reliability. Narrow sulci and focally enlarged sulci should be assessed both in the most superior transverse slices and in the coronal view. Widening of the Sylvian fissures is usually evident in the coronal plane but should also be assessed in sagittal images, where the comparison with surrounding sulci is facilitated.

We also compared the ratings for the seven features with the total scores on the iNPH Radscale [12]. The two investigators had the same median and range for both modalities, indicating concordant assessments. The results however demonstrated less consistent reliability for the iNPH Radscale than for the separate parameters (Table 3). The agreement was moderate to almost perfect for one reader using the same modality (test-retest reliability), whereas inter-rater agreement was moderate to substantial. This may be due in part to the fact that the iNPH Radscale has 13 levels, whereas each radiological parameter has either two or three levels. The likelihood of deciding on the exact same score will therefore decrease and thus also decrease the absolute agreement. However, the test-retest reliability and interrater agreement according to Cronbach’s alpha were still acceptable for both CT and MRI, exceeding the rule-of-thumb threshold level of 0.7 [18].

There was better agreement for the callosal angle with continuous measurements than with the iNPH Radscale score. This may be because many subjects had a callosal angle of about 60°, which is the cutoff between one or two points. A difference of just 1° between the two readers, for example, 59 vs. 60 degrees, can lead to a different score.

The underestimation of periventricular white matter changes on CT is a potential problem when comparing different modalities. According to the diagnostic guidelines, periventricular edema is a supportive feature of iNPH and does not alter the treatment plan. Thus, it may be excluded in a future revision of the iNPH Radscale, at least for CT.

Agreement studies have a blinded protocol, so one cannot compare modalities or present and previous readings. This contrasts with the clinical setting, in which the comparison of images is a key strategy of high-quality reporting by the radiologist. Therefore, some of the discrepancies found in this study would probably be less important in the daily clinical routine.

During the patient selection process, 297 patients were excluded because of the inclusion criteria, i.e., a short time interval between investigations and the use of thin sections or data that were reformatted in three planes. However, if there was a longer time interval between the examinations, the morphology might have changed due to disease progression, which would have affected the results. Likewise, lower image quality would have influenced the accuracy of the measurements and visual scoring.

The sample included scans suited for multiplanar reconstruction (MR 3D and CT in 1-mm sections) as well as coronal reformats saved to PACS from the scanner. The coronal reformat was then re-angulated perpendicular to the commissure plane using multiplanar reconstruction, to correctly measure the callosal angle. The thin sections are of course preferable for correct alignment, but the coronal reformat realigned is still an improvement. The exclusion would have been too big if 3D sequence for MR and 1-mm sections for CT had been mandatory.

Due to the selected sample population, the prevalence of radiological changes was high. Indeed, a more diverse sample with a corresponding number of positive and negative cases would have simplified the calculations. Skewed samples, which are common in clinical materials, complicate the interpretation of kappa and therefore require compensation in the statistical method. Since the kappa values were possibly unreliable [27], they were completed with prevalence-adjusted bias-adjusted kappa for dichotomous categorical data [28]. On the other hand, the sample was representative of shunted iNPH patients.

Imaging is important for the diagnosis of iNPH, and early detection of the disease facilitates prompt treatment. With corresponding reliability for CT and MRI, when it comes to reporting signs of iNPH, radiologists and neurologists can be confident in the assessment independent of modality. For detection of iNPH, CT is a suitable first-line investigation because of its wide availability and low cost, whereas MRI can confirm the diagnosis and rule out other causes of ventriculomegaly. The iNPH Radscale includes seven iNPH-associated features and can aid the structured reporting. Other scales and scoring systems have been suggested, i.e., the iNPH probability calculator for diagnosis and the DESH score for selection of shunt candidates [17, 29]. The iNPH probability calculator includes Evans’ index and callosal angle and has been proven to discriminate Alzheimer’s disease from iNPH. However, the DESH features are not included, with the risk of false negative results. DESH score is constructed for predicting shunt response, which is important to separate from diagnostic purposes.

Reporting a standardized measurement that is assessed in an arbitrary way could potentially mislead an investigation. If it is not performed correctly, such a measurement should not be reported at all. A standardized measurement procedure can be time-consuming and requires patience, but hopefully it produces reproducible data and eliminates reader-dependent/subjective reports.