Introduction
Mapping brain-wide structural connectivity has become an important goal in neuroscience and is the primary focus of the field of connectomics (Sporns et al.
2005; Sporns
2010). This field aims to construct comprehensive maps of all neuronal elements and their connections in different organisms, including a wide range of mammalian species. Connectomes can provide crucial insights into brain functioning in health and disease as (changes in) structural connections underlie at least part of functional and behavioral phenotypes (Sporns et al.
2005; Bassett and Bullmore
2009; Sporns
2010; Stam
2014).
Unraveling the human connectome is a great challenge (Poldrack and Farah
2015) and its reconstruction would be a fundamental breakthrough in neuroscience (Sporns et al.
2005; Sporns
2010). So far, a complete connectome has been mapped only for
C. elegans (White et al.
1986). Accurate connectome reconstructions at micro-, meso- and macro-scale in other species, including
Drosophila (Chiang et al.
2011), mouse (Oh et al.
2014), rat (Schmitt and Eipert
2012), macaque (Stephan et al.
2001) and human (Van Essen et al.
2012) involve ongoing efforts in large-scale multicenter projects, such as the Human Connectome Project (Van Essen et al.
2012), the BRAIN project (NIH
2014), the Brainnetome Project (Jiang
2013) and the CONNECT project (Assaf et al.
2013).
Diffusion MRI-based tractography has offered exclusive means to map structural connections non-invasively in animal and human brain, and is considered a very useful and powerful technique to unravel the human connectome at meso- and macro-level (Le Bihan et al.
1986; Basser et al.
1994b; Mori et al.
1999; Jones
2008; Jbabdi and Johansen-Berg
2011). However, the anatomical accuracy of diffusion-weighted tractography, which infers neuronal pathways from tissue water diffusion direction, continues to be a matter of controversy, and reconstructing connectomes with both high sensitivity and high specificity remains challenging (Sporns et al.
2005; Bastiani et al.
2012; Jbabdi et al.
2013,
2015; Thomas et al.
2014; Azadbakht et al.
2015; Calabrese et al.
2015; Knösche et al.
2015; Reveley et al.
2015; Donahue et al.
2016). This may lead to biased characterization of connectome topology, particularly in the presence of false positive tract reconstructions (Drakesmith et al.
2015; Zalesky et al.
2016).
The classical approach of diffusion tensor (DT)-based deterministic tractography (Basser et al.
2000) uses the diffusion tensor model (i.e., estimating one principal direction in each voxel) to reconstruct fiber tracts in the brain. This method is particularly hampered by voxels containing multiple fiber orientations such as crossing, bending or fanning fibers (Jbabdi and Johansen-Berg
2011). At standard spatial resolutions around a few mm, multiple fiber orientations affect the diffusivity patterns in up to 90% of white matter voxels in the human brain (Jeurissen et al.
2013). To improve the characterization of the intravoxel diffusion profile and the accuracy of subsequent tractography, high-angular resolution diffusion imaging (HARDI) methods have been developed (e.g., Descoteaux et al.
2007; Jeurissen et al.
2009; Tournier et al.
2011; Tax et al.
2014). For instance, constrained spherical deconvolution (CSD), developed by Tournier et al., models the diffusion profile in voxels with multiple fiber orientations (Tournier et al.
2004,
2007). Instead of estimating one principal fiber direction, CSD estimates multiple fiber directions within a voxel, resulting in a fiber orientation distribution (FOD) that may have multiple peaks (e.g., four in case of two crossing fibers in one voxel) (Tournier et al.
2010,
2012). CSD-based tractography uses second-order integration over fiber orientation distributions, thereby including information from orientations in neighboring voxels (Tournier et al.
2010,
2012). This solves some of the typical tractography errors, but false positive tract reconstructions may still arise from noisy (spurious) peaks or from ambiguous fanning and bending fiber populations (Jbabdi and Johansen-Berg
2011). Recently, a new method, called spherical-deconvolution informed filtering of tractograms (SIFT) has been proposed to reduce local streamline density bias by filtering the tractogram (i.e., selectively removing streamlines) to improve the fit between the overall distribution of streamlines and the FOD in every voxel, estimated from the diffusion-weighted image, which may improve accuracy of connectome reconstruction (Smith et al.
2013,
2015a). Still, additional errors may arise from the conventional method of reconstructing tracts from streamlines that are propagated in a step-wise approach. Consequently, long-distance connections are more difficult to track, as the reconstruction error accumulates with each tracking step (Zalesky and Fornito
2009; Jbabdi et al.
2015; Reveley et al.
2015; Donahue et al.
2016). Instead of tracking streamlines, alternative tractography approaches model tracts at a global rather than voxel level, to solve local reconstruction issues with bending and fanning fiber populations as well as to mitigate the reconstruction error accumulation at longer distances. Global tractography aims to find the full track (i.e., whole brain) configuration that most optimally fits the acquired data (Reisert et al.
2011; Mangin et al.
2013; Christiaens et al.
2015b). Modern methods such as CSD-based and, global tractography and SIFT may significantly improve the performance of brain-wide tractography and contribute to more accurate connectome reconstructions. However, anatomical validation is largely lacking.
Validation studies in mammalian species from which neuronal tracing data is available, offer powerful means to directly compare diffusion-based tract reconstructions against true neuroanatomical connectivity (Dauguet et al.
2007; Dyrby et al.
2007; Gao et al.
2013; Thomas et al.
2014; Calabrese et al.
2015; Chen et al.
2015). This approach has been successfully applied to validate tractography methods and/or to verify reconstruction of specific white matter structures in squirrel monkeys (Gao et al.
2013), mini-pigs (Dyrby et al.
2007) and macaques (Thomas et al.
2014; Reveley et al.
2015). However, validation of whole-brain connectome reconstruction (i.e., structural connectivity between multiple brain regions) from contemporary diffusion-based tractography approaches is currently lacking. Therefore, we compared brain-wide structural connectome reconstruction from DT-based, CSD-based and global tractography, using ultrahigh-resolution diffusion-weighted images of rat brain, against neuroanatomical connectivity measures from tracer studies. We evaluated tractography results across different anatomical sites and distances in the brain and determined how parameter settings affect the connectome reconstruction. In addition, we assessed the effect of SIFT, streamline thresholding and group-based average network thresholding on connectome reconstructions.
Discussion
In this study, we evaluated the accuracy of connectome reconstruction with three different diffusion-based tractography approaches in rat brain. The connectome reconstructions were compared against neuroanatomical tracer data. Despite our ability to track a large variety of brain-wide neuronal fiber connections, we detected a considerable number of false positive and false negative connectivity reconstructions, which increased with increasing fiber lengths. Despite their ability to solve reconstruction issues with crossing fibers, overall performance (i.e., as expressed by the Jaccard index) of CSD-based and global tractography was comparable to conventional DT-based tractography. CSD-based tractography outperformed global tractography for reconstruction of long-distance connections. However, the different tractography approaches presented considerable differences in connectome reconstruction sensitivity and specificity. While DT and GT led to low sensitivity and high specificity, CSD resulted in high sensitivity together with low specificity.
The considerable amount of false positive and false negative connections generated by diffusion-based tractography in rat brain is in line with related studies in macaque brain (Thomas et al.
2014; Azadbakht et al.
2015; Donahue et al.
2016), and mouse brain (Calabrese et al.
2015; Chen et al.
2015), and similar findings have been reported for the human connectome (Maier-Hein et al.
2016). We identified several factors that influence the sensitivity and specificity of connectome reconstruction. The number of streamlines (for CSD- and DT-based tractography) and connection potential (GT) were positively related to sensitivity. Both factors increased the total number of reconstructed connections, leading to a higher true positive rate. Conversely, FA threshold (for DT-based tractography), particle potential (GT), group-based average network thresholding, streamline thresholding and filtering (SIFT) were negatively related to sensitivity. The step size and angle threshold (for CSD- and DT-based tractography) and FOD threshold (for CSD-based tractography) had almost no effect. Improvements in sensitivity were invariably at the expense of specificity and vice versa.
The FA threshold had significantly more effect on connectome reconstruction sensitivity and specificity than the step size and angle threshold, in line with previous research (Azadbakht et al.
2015; Chen et al.
2015). In fact, an increasing FA threshold negatively influenced sensitivity and the Jaccard index, suggesting that an FA threshold should be omitted for this type of datasets and analyses (Azadbakht et al.
2015; Chen et al.
2015). Apparently, the angle thresholds and step sizes—both important factors related to tract curvature—as applied in our study were within plausible neuroanatomical ranges, thereby not affecting the overall connectome reconstructions. However, the angle threshold might still have affected specific pathway reconstructions in the brain (Thomas et al.
2014). Also, further increase (or decrease) of the step size and angle threshold may affect, on their own, or by interaction, subsequent connectome reconstruction.
Interestingly, contrary to the FA threshold (for DT-based tractography), the FOD threshold (for CSD-based tractography) appeared to have no impact on connectome reconstruction sensitivity and specificity. This may be explained by the different natures of DT-based tractography (i.e., deterministic) and CSD-based tractography (i.e., probabilistic) or by the different natures of the FA and FOD thresholds. The FA threshold terminates streamlines in voxels with sub-threshold FA values (i.e., at voxel level), whereas the FOD threshold only prohibits streamline sampling from sub-threshold FOD amplitude peaks within a voxel (i.e., at sub-voxel level), while sampling from other (criteria fulfilling) FOD peaks is still possible. Nevertheless, we expected that an increase in FOD threshold would lead to more rigid delineation of tracts, since streamlines will only follow directions with higher FOD amplitudes (Tournier et al.
2012). However, inspection of our data revealed that many FODs, representing (multiple) crossing fibers, contained multiple high FOD amplitudes. Therefore, to influence the connectome reconstruction sensitivity and specificity, we should apparently increase the FOD thresholds to much higher values (i.e., far beyond the default 0.1 threshold), but this would probably lead to undesirable exclusion of many (truly) crossing fibers. Also, it should be noted that the effects of FOD threshold adjustments may be different across species and datasets.
SIFT filtering has been developed to improve the biological plausibility of final structural connectome reconstructions (Smith et al.
2013,
2015a). However, our study did not reveal an improvement in correspondence between reconstructed connectomes and neuroanatomical connectivity after SIFT correction. As such, our overall results confirm previously reported trade-offs between sensitivity and specificity of tract reconstructions across tractography methods (Bastiani et al.
2012; Thomas et al.
2014; Chen et al.
2015; Donahue et al.
2016). However, we found, as indicated by differing Jaccard indices that the trade-off between reconstruction sensitivity and specificity varies considerably over a range of settings and streamline- or group-based thresholds. This means that a change in the number of false positives through adjustment of tractography parameter settings is not linearly scaled with the change in number of false negatives (and true positives).
Our analyses revealed clear differences in reconstruction sensitivity and specificity dependent on the site (running within or between hemispheres) and length of connections. This is similar to what has previously been reported for macaque brain (Thomas et al.
2014), where reconstruction sensitivity and specificity differed between distinct pathways (and sites) in the brain. The Jaccard index generally increased with higher number of streamlines for intra- and interhemispheric tract reconstructions, despite an increase in false positive tract reconstructions. Our tested DT-based (i.e., deterministic) and CSD-based (i.e., probabilistic) tractography algorithms were affected by connection distance, showing a non-linear decrease in sensitivity and Jaccard index, and a non-linear increase in specificity, with increasing connection distance. CSD-based and global tractography yielded highest Jaccard indices for short-distance connections, which decreased for connections of longer distance. In agreement with previous studies (Zalesky and Fornito
2009; Jbabdi et al.
2015), we found a relatively high rate of false negative reconstructions for connected cortical areas located at large distance from each other, especially for global tractography, which may consequently lead to omissions in identification of important inter-modular hub-node connections. Future studies should focus on elucidation of the exact underlying causes of these variations between and within deterministic and probabilistic tractography approaches. Despite the general trade-off between reconstruction sensitivity and specificity, we observed notable differences between tractography algorithms and parameter settings that should be taken into account for application in connectivity studies and subsequent network analyses. For example, if avoidance of false positives and false negatives are equally important, it may be advisable to choose tractography parameter settings that yield the highest Jaccard index (i.e., a high number of streamlines and a low FA threshold). On the other hand, for modular connectomes such as brain networks, certain network metrics, e.g., clustering coefficient, global efficiency and modularity, have been shown to be more strongly affected by false positives (which tend to arise inter-modularly) than by false negatives (which tend to arise intra-modularly) (Drakesmith et al.
2015; Zalesky et al.
2016). Under such circumstances, an approach that overweighs reconstruction specificity over sensitivity (e.g., with global tractography using a low connection potential, or by applying streamline thresholds to increase specificity), thereby minimizing false positives at the expense of uncovering some true connections, may be preferred (van Wijk et al.
2010; Fornito et al.
2013; Zalesky et al.
2016). Zalesky et al. have recommended that graph-based network analyses should be preferably performed on connectomes with at least about twice the number of false negatives compared to false positives (i.e., the 2:1 rule) (Zalesky et al.
2016). Our data show that specific DT and global tractography parameter settings can satisfy this rule. However, this may be more complicated for CSD, which typically generates high density networks and relatively large numbers of false positives. Lastly, reconstruction sensitivity and specificity may be influenced by streamline thresholding or group-based analysis (i.e., incidence thresholds, which might increase robustness against inter-subject variation), which might be employed to decrease the number of false positive reconstructions to satisfy the suggested 2:1 rule (Zalesky et al.
2016).
It is possible that the mismatch between diffusion-based tractography and neuroanatomical tracers in our (and previous) work may be partly due to incompleteness of neuroanatomical tracer data. For example, the multi-synaptic connections, which are not easily detected with neuronal tracing, might be incorrectly identified as false positives. This actually challenges the concept of neuronal tracing as gold standard method for assessment of neuroanatomical connectivity. It is also conceivable that diffusion tractography findings may guide future neuronal tracing assessments of unknown neuronal pathways. We also did not assess connectivity of subcortical regions, which are less easily co-registerable to the rat brain anatomical atlas, which biases our conclusions towards cortical connectomics. Another point is that we only considered streamline endpoints in determining connectivity between two regions of interest, thereby ignoring the three-dimensional structure of anatomical pathways. It may be argued that this is inappropriate from an anatomical viewpoint, however, this has been the strategy in the majority of structural network studies (which typically focus on network topology and not on exact fiber trajectories). Moreover, a recent study in mice showed that the accuracy of diffusion-based tractography is worse when spatially comparing tractography streamlines with tracer-based connections using 3D co-localization analysis (Calabrese et al.
2015). Lastly, the weights of neuronal connections were not considered in our study. While this was recently done to compare measures of connectivity strength between diffusion tractography and tract tracing data from macaque brain (van den Heuvel et al.
2015; Donahue et al.
2016), most neuroanatomical connections are at least to a certain degree bidirectional and contain nested sub-regions with their own connections and corresponding weights. This complicates the assignment of an absolute or relative measure of connection weight and directionality.
In conclusion, our study shows that conventional deterministic DT-based and more advanced probabilistic CSD-based and global tractography approaches reconstruct the whole-brain connectome with comparable accuracy, be it with considerable amounts of false positive and false negative connections, which strongly depends on the algorithm and parameter settings. Because tractography conditions differ between intra- and interhemispheric connections as well as over different connection distances, algorithms should be chosen purposefully and, if possible, combined to enable most reliable reconstruction of connectomes at large-scale network or whole-brain level. For instance, since global tractography has difficulties with resolving long-distance connections, this long-distance connectivity information could be extracted from different adjacency matrices (based on DT-based tractography). The large number of invalid tract reconstructions we observed in our study, in line with a recent study on human tract reconstructions (Maier-Hein et al.
2016), stresses the relatively ill-posed nature of current tractography approaches and the need for methodological progression. Use of alternative streamline filtering techniques such as SIFT2 (Smith et al.
2015b), inclusion of anatomical constraints (Smith et al.
2012; Lemkaddem et al.
2014) or priors (Yendiki et al.
2011; Christiaens et al.
2014,
2015a), or application of Bayesian connectomics (Hinne et al.
2012; Kasenburg et al.
2016) may lead to further improvement of connectome reconstruction accuracy. Furthermore, the increasing availability of neuronal tracer databases may aid in fine-tuning of diffusion-based tractography settings and applications, which will contribute to the progress of the rising field of connectomics.