Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Deep learning for tomographic image reconstruction

Nature Machine Intelligence volume 2, pages 737–748 (2020)Cite this article

Subjects

Abstract

Deep-learning-based tomographic imaging is an important application of artificial intelligence and a new frontier of machine learning. Deep learning has been widely used in computer vision and image analysis, which deal with existing images, improve these images, and produce features from them. Since 2016, deep learning techniques have been actively researched for tomographic imaging, especially in the context of biomedicine, with impressive results and great potential. Tomographic reconstruction produces images of multi-dimensional structures from externally measured ‘encoded’ data in the form of various tomographic transforms (integrals, harmonics, echoes and so on). In this Review, we provide a general background, highlight representative results with an emphasis on medical imaging, and discuss key issues that need to be addressed in this emerging field. In particular, tomographic imaging is an integral part of modern medicine, and will play a key role in personalized, preventive and precision medicine and make it intelligent, inexpensive and indiscriminate.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Three types of tomographic reconstruction methods.

Molly Freimuth (CT image); Christopher Hardy (MR image); Dawn Fessett (ultrasound image).

Fig. 2: The reconstruction process can be broken up into four essential steps, each of which can either be learnt from the training data or be explicitly defined by the algorithm designer.

Molly Freimuth (CT image); Christopher Hardy (MR images).

Fig. 3: Deep learning reconstruction results for various imaging modalities.

Froedtert & Medical College of Wisconsin (CT images); Christopher Hardy (MR images); Quanzheng Li (PET images).

Fig. 4: Big data acquired from various sources and labelled in different forms.
Fig. 5: Existing mobile/compact imagers and future possibilities for hybrid imaging.

Yan Xi (C-arm); Judy Samuelman (CT scanner); Qiyu Peng (wearable PET device); Alex Panther (MRI scanner); Homer Pien (ultrasound unit).

Similar content being viewed by others

References

  1. Barrett, H., Myers, K., Hoeschen, C., Kupinski, M. & Little, M. Task-based measures of image quality and their relation to radiation dose and patient risk. Phys. Med. Biol. 60, R1 (2015).

    Google Scholar 

  2. Floyd, C. An artificial neural network for SPECT image reconstruction. IEEE Trans. Med. Imaging 10, 485–487 (1991).

    Google Scholar 

  3. Yan, H. & Mao, J. Data truncation artifact reduction in MR imaging using a multilayer neural network. IEEE Trans. Med. Imaging 12, 73–77 (1993).

    Google Scholar 

  4. Wang, G. A perspective on deep imaging. IEEE Access 4, 8914–8924 (2016).

    Google Scholar 

  5. Wang, G., Ye, J. C., Mueller, K. & Fessler, J. A. Image reconstruction is a new frontier of machine learning. IEEE Trans. Med. Imaging 37, 1289–1296 (2018).

    Google Scholar 

  6. McCann, M. T., Jin, K. H. & Unser, M. Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal Process. Mag. 34, 85–95 (2017).

    Google Scholar 

  7. Ravishankar, S., Ye, J. C. & Fessler, J. A. Image reconstruction: from sparsity to data-adaptive methods and machine learning. Proc. IEEE 108, 86–109 (2019).

    Google Scholar 

  8. Sahiner, B. et al. Deep learning in medical imaging and radiation therapy. Med. Phys. 46, e1–e36 (2019).

    Google Scholar 

  9. Gong, K., Berg, E., Cherry, S. R. & Qi, J. Machine learning in PET: from photon detection to quantitative image reconstruction. Proc. IEEE 108, 51–68 (2019).

    Google Scholar 

  10. Nuyts, J., De Man, B., Fessler, J. A., Zbijewski, W. & Beekman, F. J. Modelling the physics in the iterative reconstruction for transmission computed tomography. Phys. Med. Biol. 58, R63 (2013).

    Google Scholar 

  11. Hebert, T. & Leahy, R. A generalized EM algorithm for 3-D bayesian reconstruction from Poisson data using Gibbs priors. IEEE Trans. Med. Imagine 8, 194–202 (1989).

    Google Scholar 

  12. Zaidi, H., Ruest, T., Schoenahl, F. & Montandon, M.-L. Comparative assessment of statistical brain MR image segmentation algorithms and their impact on partial volume correction in PET. Neuroimage 32, 1591–1607 (2006).

    Google Scholar 

  13. Hounsfield, G. N. Computerized transverse axial scanning (tomography): part 1. Description of system. Br. J. Radiol. 46, 1016–1022 (1973).

    Google Scholar 

  14. Shepp, L. A. & Logan, B. F. The Fourier reconstruction of a head section. IEEE Trans. Nucl. Sci. 21, 21–43 (1974).

    Google Scholar 

  15. Kang, E., Min, J. & Ye, J. C. A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Med. Phys. 44, e360–e375 (2017).

    Google Scholar 

  16. Chen, H. et al. Low-dose CT via convolutional neural network. Biomed. Opt. Express 8, 679–694 (2017).

    Google Scholar 

  17. Shan, H. et al. Competitive performance of a modularized deep neural network compared to commercial algorithms for low-dose CT image reconstruction. Nat. Mach. Intell. 1, 269–276 (2019).

    Google Scholar 

  18. Sodickson, D. K. & Manning, W. J. Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays. Magn. Reson. Med. 38, 591–603 (1997).

    Google Scholar 

  19. Pruessmann, K. P., Weiger, M., Scheidegger, M. B. & Boesiger, P. Sense: sensitivity encoding for fast MRI. Magn. Reson. Med. 42, 952–962 (1999).

    Google Scholar 

  20. Griswold, M. A. et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn. Reson. Med. 47, 1202–1210 (2002).

    Google Scholar 

  21. Lustig, M., Donoho, D. & Pauly, J. M. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007).

  22. Brada, R. et al. Towards motion-robust MRI – autonomous motion timing and correction during MR scanning using multi-coil data and a deep-learning neural network. In Proc. ISMRM Twenty Seventh Annual Meeting (ISMRM, 2019).

  23. Hammernik, K. et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2018).

    Google Scholar 

  24. Kwon, K., Kim, D. & Park, H. A parallel MR imaging method using multilayer perceptron. Med. Phys. 44, 6209–6224 (2017).

    Google Scholar 

  25. Sun, J., Li, H. & Xu, Z. Deep ADMM-Net for compressive sensing MRI. In Advances in Neural Information Processing Systems 10–18 (NeurIPS, 2016).

  26. Wang, S. et al. Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th Int. Symp. Biomedical Imaging 514–517 (IEEE, 2016).

  27. Defrise, M. et al. Exact and approximate rebinning algorithms for 3-D PET data. IEEE Trans. Med. Imaging 16, 145–158 (1997).

    Google Scholar 

  28. Fink, M. Time reversal of ultrasonic fields. I. Basic principles. IEEE Trans. Ultrason. Ferroelec. Freq. Control 39, 555–566 (1992).

    Google Scholar 

  29. Jin, K. H., McCann, M. T., Froustey, E. & Unser, M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017).

    MathSciNet  MATH  Google Scholar 

  30. Lee, D., Yoo, J., Tak, S. & Ye, J. Deep residual learning for accelerated MRI using magnitude and phase networks. IEEE Trans. Biomed. Eng. 65, 1985–1995 (2018).

    Google Scholar 

  31. Gong, K., Catana, C., Qi, J. & Li, Q. PET image reconstruction using deep image prior. IEEE Trans. Med. Imaging 38, 1655–1665 (2018).

    Google Scholar 

  32. Qian, H., Rui, X. & Ahn, S. Deep learning models for PET scatter estimations. In 2017 IEEE Nuclear Science Symp. Medical Imaging Conf. 1–5 (IEEE, 2017).

  33. Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy STORM. Nat. Methods 3, 793–796 (2006).

    Google Scholar 

  34. Hess, S. T., Girirajan, T. P. & Mason, M. D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 91, 4258 (2006).

    Google Scholar 

  35. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

    Google Scholar 

  36. Nehme, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. DeepSTORM: super-resolution single-molecule microscopy by deep learning. Optica 5, 458–464 (2018).

  37. Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).

    Google Scholar 

  38. Liu, T. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 3926 (2019).

    Google Scholar 

  39. Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).

    Google Scholar 

  40. Li, S., Deng, M., Lee, J., Sinha, A. & Barbastathis, G. Imaging through glass diffusers using densely connected convolutional networks. Optica 5, 803–813 (2018).

    Google Scholar 

  41. Sun, Y., Xia, Z. & Kamilov, U. S. Efficient and accurate inversion of multiple scattering with deep learning. Opt. Express 26, 14678–14688 (2018).

    Google Scholar 

  42. Lyu, M. et al. Deep-learning-based ghost imaging. Sci. Rep. 7, 17865 (2017).

    Google Scholar 

  43. Rivenson, Y. et al. Deep learning enhanced mobile-phone microscopy. ACS Photon. 5, 2354–2364 (2018).

    Google Scholar 

  44. Xu, M. & Wang, L. V. Photoacoustic imaging in biomedicine. Rev. Sci. Instrum. 77, 041101 (2006).

    Google Scholar 

  45. Zhou, Z. et al. High spatial–temporal resolution reconstruction of plane-wave ultrasound images with a multichannel multiscale convolutional neural network. IEEE Trans. Ultrason. Ferroelec. Freq. Control 65, 1983–1996 (2018).

    Google Scholar 

  46. Antholzer, S., Haltmeier, M. & Schwab, J. Deep learning for photoacoustic tomography from sparse data. Inverse Prob. Sci. Eng. 27, 987–1005 (2019).

    MathSciNet  Google Scholar 

  47. Davoudi, N., Deán-Ben, X. L. & Razansky, D. Deep learning optoacoustic tomography with sparse data. Nat. Mach. Intell. 1, 453–460 (2019).

    Google Scholar 

  48. Wolterink, J. M., Leiner, T., Viergever, M. A. & Išgum, I. Generative adversarial networks for noise reduction in low-dose CT. IEEE Trans. Med. Imaging 36, 2536–2545 (2017).

    Google Scholar 

  49. Yang, G. et al. DAGAN: deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans. Med. Imaging 37, 1310–1321 (2017).

    Google Scholar 

  50. Bubba, T. A. et al. Learning the invisible: a hybrid deep learning-shearlet framework for limited angle computed tomography. Inverse Prob. 35, 064002 (2019).

  51. Kutyniok, G. & Labate, D. Shearlets: Multiscale Analysis for Multivariate Data (Springer, 2012).

  52. Zhang, Y. & Yu, H. Convolutional neural network based metal artifact reduction in X-ray computed tomography. IEEE Trans. Med. Imaging 37, 1370–1381 (2018).

    Google Scholar 

  53. Schlemper, J., Caballero, J., Hajnal, J. V., Price, A. N. & Rueckert, D. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging 37, 491–503 (2018).

    Google Scholar 

  54. Vishnevskiy, V., Walheim, J. & Kozerke, S. Deep variational network for rapid 4D flow MRI reconstruction. Nat. Mach. Intell. 2, 228–235 (2020).

    Google Scholar 

  55. Eo, T. et al. KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magn. Reson. Med. 80, 2188–2201 (2018).

    Google Scholar 

  56. Qin, C. et al. Convolutional recurrent neural networks for dynamic mr image reconstruction. IEEE Trans. Med. Imaging 38, 280–290 (2018).

    Google Scholar 

  57. Vishnevskiy, V., Sanabria, S. J. & Goksel, O. Image reconstruction via variational network for real-time hand-held sound-speed imaging. In International Workshop on Machine Learning for Medical Image Reconstruction (eds Knoll, F. et al.) 120–128 (Springer, 2018).

  58. Adler, J. & Öktem, O. Learned primal-dual reconstruction. IEEE Trans. Med. Imaging 37, 1322–1332 (2018).

    Google Scholar 

  59. Gupta, H., Jin, K. H., Nguyen, H. Q., McCann, M. T. & Unser, M. CNN-based projected gradient descent for consistent CT image reconstruction. IEEE Trans. Med. Imaging 37, 1440–1453 (2018).

  60. Chen, H. et al. LEARN: Learned experts’ assessment-based reconstruction network for sparse-data CT. IEEE Trans. Med. Imaging 37, 1333–1347 (2018).

  61. Cheng, L., Ahn, S., Ross, S. G., Qian, H. & De Man, B. Accelerated iterative image reconstruction using a deep learning based leapfrogging strategy. In Int. Conf. Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 715–720 (Fully 3D, 2017).

  62. Chun, I. Y., Huang, Z., Lim, H. & Fessler, J. Momentum-net: fast and convergent iterative neural network for inverse problems. IEEE Trans. Pattern Anal. Mach. Intell. https://doi.org/10.1109/TPAMI.2020.3012955 (2020).

  63. Aggarwal, H. K., Mani, M. P. & Jacob, M. MoDL: model-based deep learning architecture for inverse problems. IEEE Trans. Med. Imaging 38, 394–405 (2018).

  64. Hauptmann, A. et al. Model-based learning for accelerated, limited-view 3-D photoacoustic tomography. IEEE Trans. Med. Imaging 37, 1382–1393 (2018).

    Google Scholar 

  65. Biswas, S., Aggarwal, H. K. & Jacob, M. Dynamic MRI using model-based deep learning and SToRM priors: MoDL-SToRM. Magn. Res. Med. 82, 485–494 (2019).

    Google Scholar 

  66. Lyu, Q., Shan, H., Xie, Y., Li, D. & Wang, G. Cine cardiac MRI motion artifact reduction using a recurrent neural network. Preprint at https://arxiv.org/abs/2006.12700 (2020).

  67. Gong, K. et al. Iterative PET image reconstruction using convolutional neural network representation. IEEE Trans. Med. Imaging 38, 675–685 (2018).

    Google Scholar 

  68. Wu, D., Kim, K., El Fakhri, G. & Li, Q. Iterative low-dose CT reconstruction with priors trained by artificial neural network. IEEE Trans. Med. Imaging 36, 2479–2486 (2017).

    Google Scholar 

  69. Lim, H., Chun, I. Y., Fessler, J. & Dewaraja, Y. Improved low count quantitative SPECT reconstruction with a trained deep learning based regularizer. J. Nucl. Med. 60, 42 (2019).

    Google Scholar 

  70. Gao, Y. et al. A feasibility study of extracting tissue textures from a previous full-dose CT database as prior knowledge for Bayesian reconstruction of current low-dose CT images. IEEE Trans. Med. Imaging 38, 1981–1992 (2019).

    Google Scholar 

  71. Buzzard, G. T., Chan, S. H., Sreehari, S. & Bouman, C. A. Plug-and-play unplugged: optimization-free reconstruction using consensus equilibrium. SIAM J. Imaging Sci. 11, 2001–2020 (2018).

  72. Liu, J. et al. RARE: Image reconstruction using deep priors learned without ground truth. IEEE J. Selected Topics Signal Process. 14, 1088–1099 (2020).

  73. Wu, Z. et al. SIMBA: Scalable inversion in optical tomography using deep denoising priors. IEEE J. Selected Topics Signal Process. 14, 1163–1175 (2020).

  74. Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep image prior. In Proc. IEEE Conf. Computer Vision and Pattern Recognition 9446–9454 (IEEE, 2018).

  75. Jin, K. H., Gupta, H., Yerly, J., Stuber, M. & Unser, M. Time-dependent deep image prior for dynamic MRI. Preprint at https://arxiv.org/abs/1910.01684 (2019).

  76. Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R. & Rosen, M. S. Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018).

    Google Scholar 

  77. Ye, D. H., Buzzard, G. T., Ruby, M. & Bouman, C. A. Deep back projection for sparse-view CT reconstruction. In 2018 IEEE Global Conf. Signal and Information Processing (IEEE, 2018).

  78. Li, Y., Li, K., Zhang, C., Montoya, J. & Chen, G.-H. Learning to reconstruct computed tomography images directly from sinogram data under a variety of data acquisition conditions. IEEE Trans. Med. Imaging 38, 2469–2481 (2019).

    Google Scholar 

  79. Fu, L. & De Man, B. A hierarchical approach to deep learning and its application to tomographic reconstruction. In 15th Int. Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 1107202 (International Society for Optics and Photonics, 2019).

  80. Whiteley, W., Luk, W. K. & Gregor, J. DirectPET: full-size neural network PET reconstruction from sinogram data. J. Med. Imaging 7, 032503 (2020).

  81. Häggström, I., Schmidtlein, C. R., Campanella, G. & Fuchs, T. J. DeepPET: a deep encoder–decoder network for directly solving the PET image reconstruction inverse problem. Med. Image Anal. 54, 253–262 (2019).

  82. Würfl, T. et al. Deep learning computed tomography: learning projection-domain weights from image domain in limited angle problems. IEEE Trans. Med. Imaging 37, 1454–1463 (2018).

  83. Han, Y. & Ye, J. C. One network to solve all ROIs: deep learning CT for any ROI using differentiated backprojection. Med. Phys. 46, e855–e872 (2019).

  84. Han, Y., Kim, J. & Ye, J. C. Differentiated backprojection domain deep learning for conebeam artifact removal. IEEE Trans. Med. Imaging 39, 3571–3582 (2020).

    Google Scholar 

  85. Claus, B. E., Jin, Y., Gjesteby, L. A., Wang, G. & De Man, B. Metal-artifact reduction using deep-learning based sinogram completion: initial results. In Proc. 14th Int. Meeting Fully Three-Dimensional Image Reconstruction Radiology Nuclear Medicine 631–634 (Fully 3D, 2017).

  86. Ghani, M. U. & Karl, W. C. Fast enhanced CT metal artifact reduction using data domain deep learning. IEEE Trans. Comput. Imaging 6, 181–193 (2019).

    Google Scholar 

  87. De Man, Q. et al. A two-dimensional feasibility study of deep learning-based feature detection and characterization directly from CT sinograms. Med. Phys. 46, e790–e800 (2019).

    Google Scholar 

  88. Han, Y., Sunwoo, L. & Ye, J. C. k-space deep learning for accelerated MRI. IEEE Trans. Med. Imaging 39, 377–386 (2019).

    Google Scholar 

  89. Lee, J., Han, Y., Ryu, J.-K., Park, J.-Y. & Ye, J. C. k-space deep learning for reference-free EPI ghost correction. Magn. Reson. Med. 82, 2299–2313 (2019).

    Google Scholar 

  90. Akçakaya, M., Moeller, S., Weingärtner, S. & Uğurbil, K. Scan-specific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: database-free deep learning for fast imaging. Magn. Res. Med. 81, 439–453 (2019).

    Google Scholar 

  91. Pramanik, A., Aggarwal, H. & Jacob, M. Deep generalization of structured low-rank algorithms (Deep-SLR). IEEE Trans. Med. Imaging https://doi.org/10.1109/TMI.2020.3014581 (2020).

  92. Luchies, A. C. & Byram, B. C. Deep neural networks for ultrasound beamforming. IEEE Trans. Med. Imaging 37, 2010–2021 (2018).

    Google Scholar 

  93. Gasse, M. et al. High-quality plane wave compounding using convolutional neural networks. IEEE Trans. Ultrason. Ferroelec. Freq. Control 64, 1637–1639 (2017).

    Google Scholar 

  94. Khan, S., Huh, J. & Ye, J. C. Adaptive and compressive beamforming using deep learning for medical ultrasound. IEEE Trans. Ultrason. Ferroelec. Freq. Control 67, 1558–1572 (2020).

    Google Scholar 

  95. Luijten, B. et al. Deep learning for fast adaptive beamforming. In 2019 IEEE Int. Conf. Acoustics, Speech and Signal Processing 1333–1337 (IEEE, 2019).

  96. Yoon, Y. H., Khan, S., Huh, J. & Ye, J. C. Efficient B-mode ultrasound image reconstruction from sub-sampled RF data using deep learning. IEEE Trans. Med. Imaging 38, 325–336 (2018).

    Google Scholar 

  97. Kang, E., Koo, H. J., Yang, D. H., Seo, J. B. & Ye, J. C. Cycle-consistent adversarial denoising network for multiphase coronary CT angiography. Med. Phys. 46, 550–562 (2019).

    Google Scholar 

  98. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. IEEE Int. Conf. Computer Vision 2223–2232 (IEEE, 2017).

  99. Nguyen, T., Xue, Y., Li, Y., Tian, L. & Nehmetallah, G. Deep learning approach for Fourier ptychography microscopy. Opt. Express 26, 26470–26484 (2018).

    Google Scholar 

  100. Lee, S., Han, S., Salama, P., Dunn, K. W. & Delp, E. J. Three dimensional blind image deconvolution for fluorescence microscopy using generative adversarial networks. In 2019 IEEE 16th Int. Symp. Biomedical Imaging 538–542 (IEEE, 2019).

  101. Villani, C. Optimal Transport: Old and New Vol. 338 (Springer, 2008).

  102. Lim, S., Lee, S.-E., Chang, S. & Ye, J. C. CycleGAN with a blur kernel for deconvolution microscopy: Optimal transport geometry. IEEE Trans. Comput. Imaging 6, 1127–1138 (2020).

    MathSciNet  Google Scholar 

  103. Li, M., Shan, H., Pryshchep, S., Lopez, M. M. & Wang, G. Deep adversarial network for super stimulated emission depletion imaging. J. Nanophoton. 14, 016009 (2020).

    Google Scholar 

  104. Liu, F., Jang, H., Kijowski, R., Bradshaw, T. & McMillan, A. B. Deep learning MR imaging–based attenuation correction for PET/MR imaging. Radiology 286, 676–684 (2018).

    Google Scholar 

  105. Hwang, D. et al. Improving the accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. J. Nucl. Med. 59, 1624–1629 (2018).

    Google Scholar 

  106. Dar, S. U. et al. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans. Med. Imaging 38, 2375–2388 (2019).

    Google Scholar 

  107. Lee, D., Moon, W.-J. & Ye, J. C. Assessing the importance of magnetic resonance contrasts using collaborative generative adversarial networks. Nat. Mach. Intell. 2, 34–42 (2020).

    Google Scholar 

  108. Lehtinen, J. et al. Noise2Noise: learning image restoration without clean data. In Int. Conf. Machine Learning 2965–2974 (PMLR, 2018).

  109. Yaman, B. et al. Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data. In 2020 IEEE 17th Int. Symp. Biomedical Imaging 921–925 (IEEE, 2020).

  110. Wu, D., Ren, H. & Li, Q. Self-supervised dynamic CT perfusion image denoising with deep neural networks. IEEE Trans. Radiation Plasma Med. Sci. https://doi.org/10.1109/TRPMS.2020.2996566 (2020).

  111. Wu, D., Gong, K., Kim, K., Li, X. & Li, Q. Consensus neural network for medical imaging denoising with only noisy training samples. In Int. Conf. Medical Image Computing and Computer-Assisted Intervention 741–749 (Springer, 2019).

  112. Donoho, D. L. Compressed sensing. IEEE Trans. Inform. Theory 52, 1289–1306 (2006).

    MathSciNet  MATH  Google Scholar 

  113. Ye, J. C., Han, Y. & Cha, E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J. Imaging Sci. 11, 991–1048 (2018).

  114. Ye, J. C. & Sung, W. K. Understanding geometry of encoder-decoder CNNs. In Proc. 36th Int. Conf. Machine Learning Vol. 97 (eds Chaudhuri, K. & Salakhutdinov, R.) 7064–7073 (PMLR, 2019).

  115. NIH Image Database (NIH, 2020); https://www.nihlibrary.nih.gov/resources/subject-guides/web-search-thinking-beyond-google/images

  116. McCollough, C. H. et al. Low-dose CT for the detection and classification of metastatic liver lesions: results of the 2016 low dose CT grand challenge. Med. Phys. 44, e339–e352 (2017).

  117. Gaillard, F. & Murphy, A. Imaging data sets (artificial intelligence). Radiopaedia https://radiopaedia.org/articles/imaging-data-sets-artificial-intelligence?lang=us (2020).

  118. Xu, X. G., Taranenko, V., Zhang, J. & Shi, C. A boundary-representation method for designing whole-body radiation dosimetry models: pregnant females at the ends of three gestational periods—RPI-P3, -P6 and -P9. Phys. Med. Biol. 52, 7023 (2007).

    Google Scholar 

  119. Segars, W. P., Sturgeon, G., Mendonca, S., Grimes, J. & Tsui, B. M. 4D XCAT phantom for multimodality imaging research. Med. Phys. 37, 4902–4915 (2010).

    Google Scholar 

  120. Krishna, A. & Mueller, K. Medical CT image generation with style. In 15th Int. Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 1107234 (International Society for Optics and Photonics, 2019).

  121. De Man, B. et al. CatSim: a new computer assisted tomography simulation environment. In Medical Imaging 2007: Physics of Medical Imaging Vol. 6510 65102G (International Society for Optics and Photonics, 2007).

  122. Jia, X., Yan, H., Cerviño, L., Folkerts, M. & Jiang, S. B. A GPU tool for efficient, accurate, and realistic simulation of cone beam CT projections. Med. Phys. 39, 7368–7378 (2012).

    Google Scholar 

  123. Abadi, E. et al. DukeSim: a realistic, rapid, and scanner-specific simulation framework in computed tomography. IEEE Trans. Med. Imaging 38, 1457–1465 (2018).

  124. De Man, B. et al. XCIST (X-ray-based Cancer Imaging Simulation Toolkit): a new open-source project (ITCR grant). In Institute for Mathematics and its Applications Workshop on Computational Imaging (IMA, 2019).

  125. Sharma, D. et al. In silico imaging tools from the VICTRE clinical trial. Med. Phys. 46, 3924–3928 (2019).

    Google Scholar 

  126. Parisi, G. I., Kemker, R., Part, J. L., Kanan, C. & Wermter, S. Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019).

    Google Scholar 

  127. Konečný, J. et al. Federated learning: Strategies for improving communication efficiency. Preprint at https://arxiv.org/abs/1610.05492 (2016).

  128. McMahan, H. B. et al. Communication-efficient learning of deep networks from decentralized data. Preprint at https://arxiv.org/abs/1602.05629 (2016).

  129. Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void-learning denoising from single noisy images. In Proc. IEEE Conf. Computer Vision and Pattern Recognition 2129–2137 (IEEE, 2019.

  130. Liao, H., Lin, W.-A., Zhou, S. K. & Luo, J. ADN: artifact disentanglement network for unsupervised metal artifact reduction. IEEE Trans. Med. Imaging 39, 634–643 (2019).

  131. Locatello, F. et al. Challenging common assumptions in the unsupervised learning of disentangled representations. In Int. Conf. Machine Learning 4114–4124 (PMLR, 2019).

  132. Gjesteby, L., Xi, Y., Kalra, M. K., Yang, Q. & Wang, G. Hybrid imaging system for simultaneous spiral MR and X-ray (MRX) scans. IEEE Access 5, 1050–1061 (2016).

    Google Scholar 

  133. Dineley, J. Tackling the silent crisis in cancer care. Lindau Nobel Laureate Meetings https://www.lindau-nobel.org/blog-tackling-the-silent-crisis-in-cancer-care-with-innovation/ (2018).

  134. Antun, V., Renna, F., Poon, C., Adcock, B. & Hansen, A. C. On instabilities of deep learning in image reconstruction and the potential costs of AI. Proc. Natl Acad. Sci. USA https://doi.org/10.1073/pnas.1907377117 (2020).

  135. Ozdag, M. Adversarial attacks and defenses against deep neural networks: a survey. Procedia Comp. Sci. 140, 152–161 (2018).

    Google Scholar 

  136. Wu, W. et al. Stabilizing deep tomographic reconstruction networks. Preprint at https://arxiv.org/abs/2008.01846 (2020).

  137. Park, S. H. & Han, K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286, 800–809 (2018).

    Google Scholar 

  138. Arrieta, A. B. et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inform. Fusion 58, 82–115 (2020).

  139. Fan, F., Xiong, J. & Wang, G. On interpretability of artificial neural networks. Preprint at https://arxiv.org/abs/2001.02522 (2020).

  140. Geis, J. R. et al. Ethics of artificial intelligence in radiology: summary of the joint European and North American multisociety statement. Radiology 293, 436–440 (2019).

  141. Awad, E. et al. The Moral Machine experiment. Nature 563, 59–64 (2018).

    Google Scholar 

  142. Wang, G., Zhang, Y., Ye, X. & Mou, X. Machine Learning for Tomographic Imaging (IOP, 2019).

  143. Poirot, M. G. et al. Physics-informed deep learning for dual-energy computed tomography image processing. Sci. Rep. 9, 17709 (2019).

    Google Scholar 

  144. Kang, E., Chang, W., Yoo, J. & Ye, J. C. Deep convolutional framelet denosing for low-dose CT via wavelet residual network. IEEE Trans. Med. Imaging 37, 1358–1369 (2018).

    Google Scholar 

  145. Ouyang, J., Chen, K. T., Gong, E., Pauly, J., Zaharchuk, G. & Ultra-low-dose, P. E. T. reconstruction using generative adversarial network with feature matching and task-specific perceptual loss. Med. Phys. 46, 3555–3564 (2019).

    Google Scholar 

  146. Chen, K. T. et al. Ultra-low-dose 18F-florbetaben amyloid PET imaging using deep learning with multi-contrast MRI inputs. Radiology 290, 649–656 (2019).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Biomedical Imaging Center, Center for Biotechnology and Interdisciplinary Studies, Department of Biomedical Engineering, School of Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

    Ge Wang

  2. Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

    Jong Chul Ye

  3. Radiation Imaging Laboratory, Biology and Applied Physics, GE Research, Niskayuna, NY, USA

    Bruno De Man

Authors
  1. Ge Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jong Chul Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruno De Man
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ge Wang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, G., Ye, J.C. & De Man, B. Deep learning for tomographic image reconstruction. Nat Mach Intell 2, 737–748 (2020). https://doi.org/10.1038/s42256-020-00273-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-020-00273-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics