Reliability of the freehand region-of-interest method in quantitative cerebral diffusion tensor imaging

Diffusion tensor imaging (DTI) is a magnetic resonance imaging (MRI) technique that has become a popular tool for central nervous system imaging [1, 2]. DTI is based on the diffusion characteristics of water molecules, which, in turn, reflect the histological structure of the tissue [3]. Diffusion data can be used to calculate several quantitative parameters, such as fractional anisotropy (FA), the apparent diffusion coefficient (ADC), and axial (AD) and radial (RD) diffusivity. FA indicates the degree of diffusion anisotropy. The diffusion is generally strongest in the orientation parallel to the nerve tracts. The ADC expresses the mean diffusion in each direction. AD can be considered to be modulated by the axonal integrity [4, 5], and its changes can thus reflect the degree of axonal degeneration [6]. RD, on the other hand, is modulated by axonal myelination [4, 5].

Several studies on different neurological diseases have utilized these DTI indices as biomarkers of white matter integrity [7‐11]. Significant age-related changes in the integrity of white matter have also been found in healthy volunteers [12‐17].

Chronic white matter diseases as well as normal aging, causes a decrease in FA values while RD values tend to increase [18‐25]. A strong relationship has also been found between the changes in AD and axonal injury [4]. Moreover, ADC values may temporarily decrease in the acute phase of cerebrovascular accidents, but, in the chronic phase, they usually increase [26, 27].

The imaging process includes several steps between acquisition and the final parametric result, and each step is susceptible to different pitfall sources [28, 29]. Specifically, low resolution, a low signal-to-noise ratio (SNR), and a variety of different types of artifacts can reduce the image quality [30‐33]. In particular, the single-shot echo-planar technique used in diffusion imaging can cause severe image distortions because of the long echo trains that are used in the sequence. The consequence of these susceptibility artifacts are geometric distortions at the interfaces between soft tissue and air at the base of the skull [34]. In addition, B₀ inhomogeneities cause a decrease in the efficiency of fat-saturation pulses [34]. Protons in water and fat have a different Larmor frequency, which leads to fat misregistration in single-shot echo-planar imaging. All of the above-mentioned pitfalls and artifacts also have a detrimental effect on the reliability of parametric results.

Post-processing and analysis methods can be selected according to whether individual or group results are required. The histogram [35], region-of-interest (ROI), and quantitative tractography methods [36] are suitable for both individual- and group-level analysis. In addition, the tract-based spatial statistics (TBSS) method [37] is an option for group analysis. Nowadays, different methods are often used concomitantly, giving additional value to the accuracy of the results [38, 39].

The ROI method is still a highly valid method when measuring individual subjects. While laborious, time-consuming, and observer-dependent, it however, is the most readily available method in commercial clinically approved software. The method can be used to evaluate the focal areas of brain parenchyma of a single subject and it enables leaving artifacts outside the area of measurement. The low or moderate repeatability of the method as well as its high intra- and inter-observer variation have been considered its cons [40].

The main objective of this study was to investigate the reliability of the freehand ROI method, by intra- and inter-observer variation and repeatability measurements. The aim was also to examine the effects of different parameters (FA, ADC, AD and RD) and artifacts on the reliability of the results. In addition, the effects of age on white matter changes were studied in group comparisons.

The data quality was excellent in most cases. In some of the cases, artifacts were detected in the cerebral peduncle, corona radiata, and centrum semiovale (Table 1 and Fig. 2). Significant eddy current artefacts did not occur.

The mean SNR values (± SD) for all regions was 27.7 ± 4.2: the pyramidal tract 30.5 ± 4.2, frontobasal area 24.1 ± 4.7, corpus callosum 25.4 ± 0.3, and thalamus 28.0 ± 4.2

FA values are considered to reflect the integrity of the white matter. Although not in itself a specific parameter in a diagnostic sense, it provides indirect information about myelination, fiber packing density, and fiber orientation [46]. It is well-known that FA values vary widely at different anatomic levels of the brain [12, 13, 40, 45, 47]. Specifically, Lee et al. [12] reported that regional FA values varied from 0.21 in deep gray matter (putamen) to 0.81 in tightly packed parallel white matter tract bundles, such as the genu of the corpus callosum. The corresponding results in this study were 0.32 for deep gray matter (thalamus) and 0.86 for the genu of the corpus callosum. Regions with coherently oriented fibers, such as the cerebral peduncle, internal capsule, and corpus callosum exhibited higher anisotropy than regions with less coherence, such as the centrum semiovale and other subcortical regions [48]. Due to the vast regional variability of FA, possible anatomical mismatches should be taken into account in inter-observer and intergroup comparisons [47]. The ADC values, on the other hand, exhibit less regional variation [13]. In our study, the ADC mean values varied between 0.7–0.8 × 10⁻³ mm²/s, and in other similar studies the variation was 0.7 to 0.9 × 10⁻³ mm²/s [45, 49‐51]. In the frontobasal area, compared to other white matter regions demonstrated lower FA and AD values and higher ADC and RD values. The FA values were in line with a tractography study by Deng et al. [52], where a mean FA value of 0.41 (profile 0.3 to 0.52) was found in the uncinate fasciculus and 0.54 (profile 0.40 to 0.68) in the forceps minor. In our study, the corresponding FA values were 0.57 and 0.51. The results of Lieberman et al. [53] were also similar to ours in the uncinate fasciculus. The FA and ADC values were almost identical to those found in our previous study (30 subjects) in most of the regions [40]. The biggest difference (14%) between our present and previous study was found in the genu of the corpus callosum. In this region, measurements were previously made on sagittal [40] instead of axial images, like in the present study. In general, the measured quantitative diffusion metrics were well in line with previous studies.

Asymmetry between the hemispheres was found in some of the regions. Pyramidal tracts, such as the posterior limb of the capsula interna and corona radiata, expressed higher FA values and lower ADC and RD values in the left hemisphere. The present results are well in agreement with previous studies [13, 40, 54]. In addition, in the centrum semiovale, asymmetry of the cerebral hemispheres was observed in the ADC value, which was also lower on the left. Some of the observed asymmetry in our study may be attributed to handedness of the volunteers; 39 of the 40 volunteers in our study were right-handed. Corresponding hemispheric differences were obtained for right-handers in another study [54]. Phase artifacts (fat misregistration) could also be a possible explanation in the regions of the corona radiata and centrum semiovale. In the corona radiata, phase artifacts were present in 55% of cases in the left hemisphere but were not present at all in the right hemisphere. Similarly, the centrum semiovale included artifacts in 25% of cases in the left hemisphere and only in 5% in the right hemisphere. The fat misregistration generally raises FA values locally and decreases ADC and RD values. Artifacts can affect the ROIs in the vicinity, even if the visible part of the artifact is cropped out. Hemispheric differences were also found in the frontobasal area. In those regions, the FA values were found to be higher in the right hemisphere, which is in agreement with previous findings [40, 55]. Jahanshad et al. [55] found that the variance in the asymmetry of the frontal lobe is strongly due to genetic factors. In our study, higher FA values were usually found in the right hemisphere of the frontobasal area. Bonekamp et al. [56] reported that small hemispheric differences could be due to slight slice angulation. Therefore, keeping the same slice position and orientation in longitudinal studies is essential [47].

In terms of age-related changes, we found significant differences between the youngest age group (18–30 years) and other age groups (31–40, 41–50, and 51–60 years). Specifically, the FA values were higher and the RD values lower in the frontobasal area in both hemispheres in the youngest age group when compared to the other age groups. For FA, this result has already been published in our previous study [41]. Other studies have also found changes in the frontal regions of the brain caused by aging [16, 17]. In general, several studies have found a negative correlation between age and FA and a positive correlation between age and RD in white matter [21, 22, 57, 59]. These variations may be related to changes in myelination and axon density [17, 58, 60].

In the present study, acceptable intra-observer variability (≤ 10%) was found in six out of nine regions for FA, while three regions had moderate but adequate variation. For ADC and AD, all regions had acceptable variability. For RD, seven out of nine regions had an acceptable or moderate variation and two had high variation (genu and splenium of the corpus callosum). The percent variation of the RD values in the corpus callosum is naturally high, because the mean value is clearly lower than in the other regions. Low RD values are due to the fact that the fibers are tightly packed and parallel to each other. In this case, the variation was not a good indicator for assessing reliability. Overall, the variation results were in line with our previous study [40]. It is noteworthy that the freehand method gives an average of 4% lower variations in the pyramidal regions compared to the circle method [13, 41]. In contrast, in our study, the freehand method gave a slightly higher variation in the corpus callosum than the circle method in previous studies [13, 41]. This may be due to the fact that in our study, ROIs were plotted on the axial image, whereas in previous studies they were plotted on the sagittal image [13, 41]. Thus, in this particular region, it would be better to use the circle method for a sagittal image than the freehand method for an axial image. The inter-observer (n = 15) variability was acceptable or moderate in seven out of nine regions. The inter-observer variabilities are in line with our previous study [40].

The intra-observer repeatability was at a very good level according to the 95% limits of agreement. The results varied according to region, and, with tightly packed white matter tracts, such as the posterior limb of the capsula interna, the difference between the limits was small. Also, the only region of gray matter—the thalamus—was found to be reliable in this analysis. Furthermore, this difference was greater in regions containing crossing fibers, such as the centrum semiovale. Overall, the results were consistent with our previous research [40]. The inter-observer agreement was lower than the intra-observer agreement in all regions, and others have reported similar results [13, 40, 59, 60]. Several studies have shown that inter-observer agreement results have been one-third lower than intra-observer results [59, 60]. Our study further confirms the trend between inter-observer and intra-observer agreements. The uncinate fasciculus was found to be the most reliable region in the inter-observer analyses for FA and RD, while the corona radiata was the most reliable region for ADC and AD.

The intra-observer reliability was high according to the average measures of the ICC analysis. In our study, average ICC refers to the repeatability obtained as the average of two measurements from a single region. Overall, the average ICC results were excellent for all four parameters. The repeatability result was also excellent (above 0.8) in eight out of nine regions for FA and all regions for the ADC. The repeatability of the freehand method was significantly improved compared to our previous study [40]. The average ICC increase was 0.4 (37%) in terms of the FA and ADC parameters.

The higher ICC values were probably due to increased observer experience in selecting a slide, avoiding artifacts and the partial volume effect of border areas. The single intra-observer ICC analysis was, on average, excellent in terms of the ADC and RD parameters and moderate in terms of the FA and AD parameters. Single ICC in our study refers to the repeatability of a single measurement, which can be considered normal practice in clinical measurements. The results showed excellent or moderate repeatability in seven out of nine regions for all DTI parameters. The region with the highest single ICC values was the forceps minor, with excellent reliability for each parameter. Good reliability was also found in the following regions: the uncinate fasciculus, thalamus, and the genu and splenium of the corpus callosum. High reliability in the corpus callosum is consistent with previous studies with the ROI method [45, 61, 62] but also with the TBSS method [38]. Inadequate results (ICC < 0.69) were found in the cerebral peduncle (FA, ADC and RD) and centrum semiovale (AD). The reason for the inferior reliability of the cerebral peduncle was the susceptibility artifact, more specifically the air-cavity. This artifact causes local changes in the results of the parameters. Although efforts were made to avoid distracted areas in the ROI, the effects of the artifact were also reflected in the surrounding areas. The reason for the low reliability of the centrum semiovale in the AD values can be explained by the multitude of crossing fibers in the subcortical white matter. Also, the statistically significant inter-observer results were highly similar to the intra-observer results. The differences between intra- and inter-observer ICC results averaged at less than 5% for the average ICC and less than 10% for the single ICC. The most reliable inter-observer region was found to be the corona radiata, which had the highest value for three different parameters (FA, ADC, and RD). For AD, the highest value was obtained in the splenium of the corpus callosum. The reliability of the measurements is greatly improved if the measurement is repeated at least once or if the result is taken as a mean of the measurements from two different observers.

The SNR measurements showed that the image quality was sufficient for reliable quantitative measurements. In general, the SNR of b = 0 s/mm² should be at least 20 in order to derive reliable FA values [36]. In our study, the SNR was well above 20 in all regions, and the measured SNR values were comparable to other studies [63, 64].

A limitation of this study was that the commercial program did not include eddy current and subject motion corrections. In addition, the used imaging parameters may have not been optimal, especially compared to more recent diffusion imaging, e.g., high angular resolution diffusion imaging (HARDI) using isotropic voxels. Acquisition with higher resolution isotropic voxels and possibly HARDI may give more accurate results [36]. Furthermore, it has been shown that using near 1 mm isotropic voxels gives excellent results in repeatability [65]. In addition, 70% of the inter-observer ICC results were statistically significant. This was a consequence of the small number of samples. The schedule of measurements was limited.

In general, the regions with high reliability and low variation possess some common features. These regions have low anatomical variation and tightly packed fibers with a common orientation [66]. These areas also often have a better SNR, fewer partial volume effects, and are also less affected by “crossing” fibers. In addition, the larger ROI size increases the SNR value and improves the repeatability [66]. When a larger ROI size is used in a limited region, it is likely that there are more percentages of the same voxels between the two measurements than for a smaller ROI. The results of the repeat measurements are thus close to each other.

In future studies, larger samples of carefully collected high-spatial and -angular resolution DTI normal data should be acquired. In those studies, more subjects should be recruited for each age group in order to perform a reliable analysis of the effect of age. In addition, it would be interesting to study how much the reliability of the measurements improve when different methods, such as the ROI, tractography, and TBSS, are used simultaneously.

According to our results, the intra-observer repeatability of the quantitative freehand ROI method can be considered at least adequate. The quantitative freehand ROI method can be considered highly reliable for the average ICC and mostly adequate for the single ICC. The reliability of the single measurements was excellent or moderate in 80% of the regions, including all DTI parameters. In the comparison of parameters, for the single ICCs, most of the repeatability results were excellent in terms of the ADC and RD while only moderate in terms of the FA and AD parameters.

As per our results, the freehand method can be considered highly suitable for research and clinical applications assuming a well-experienced observer. Measurements should be repeated at least once in each region to ensure sufficient reliability of the results. The frontobasal area, such as the uncinate fasciculus and forceps minor, as well as the internal capsule and corona radiata regions of the pyramidal tracts were found to be reliable regions in the repeatability analysis. In addition, the only region of gray matter—the thalamus—was found to be reliable. Therefore, they could be considered as regions which yield the most accurate quantitative ROI measurements in clinical settings. In general, it would be highly beneficial to favor regions with high reliability and repeatability in ROI measurements, if possible. Additionally, special care should be taken in ROI delineation in subjects with image artifacts.

When using the results of healthy adults as a control for patient groups, it should be noted that the results are most reliable on adults less than 30 years of age whose brain white matter does not yet have age-related changes.