Skip to main content

Advertisement

SpringerLink
Log in

The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Recently, researchers have built new deep learning (DL) models using a single image modality to diagnose age-related macular degeneration (AMD). Retinal fundus and optical coherence tomography (OCT) images in clinical settings are the most important modalities investigating AMD. Whether concomitant use of fundus and OCT data in DL technique is beneficial has not been so clearly identified. This experimental analysis used OCT and fundus image data of postmortems from the Project Macula. The DL based on OCT, fundus, and combination of OCT and fundus were invented to diagnose AMD. These models consisted of pre-trained VGG-19 and transfer learning using random forest. Following the data augmentation and training process, the DL using OCT alone showed diagnostic efficiency with area under the curve (AUC) of 0.906 (95% confidence interval, 0.891–0.921) and 82.6% (81.0–84.3%) accuracy rate. The DL using fundus alone exhibited AUC of 0.914 (0.900–0.928) and 83.5% (81.8–85.0%) accuracy rate. Combined usage of the fundus with OCT increased the diagnostic power with AUC of 0.969 (0.956–0.979) and 90.5% (89.2–91.8%) accuracy rate. The Delong test showed that the DL using both OCT and fundus data outperformed the DL using OCT alone (P value < 0.001) and fundus image alone (P value < 0.001). This multimodal random forest model showed even better performance than a restricted Boltzmann machine (P value = 0.002) and deep belief network algorithms (P value = 0.042). According to Duncan’s multiple range test, the multimodal methods significantly improved the performance obtained by the single-modal methods. In this preliminary study, a multimodal DL algorithm based on the combination of OCT and fundus image raised the diagnostic accuracy compared to this data alone. Future diagnostic DL needs to adopt the multimodal process to combine various types of imaging for a more precise AMD diagnosis.

The basic architectural structure of the tested multimodal deep learning model based on pre-trained deep convolutional neural network and random forest using the combination of OCT and fundus image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Wong WL, Su X, Li X, Cheung CMG, Klein R, Cheng CY, Wong TY (2014) Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis. Lancet Glob Health 2:106–116. https://doi.org/10.1016/S2214-109X(13)70145-1

    Article  Google Scholar 

  2. Ferris FL, Wilkinson CP, Bird A et al (2013) Clinical classification of age-related macular degeneration. Ophthalmology 120:844–851. https://doi.org/10.1016/j.ophtha.2012.10.036

    Article  PubMed  Google Scholar 

  3. Choi JY, Yoo TK, Seo JG, Kwak J, Um TT, Rim TH (2017) Multi-categorical deep learning neural network to classify retinal images: a pilot study employing small database. PLoS One 12:e0187336. https://doi.org/10.1371/journal.pone.0187336

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  4. Lam C, Yu C, Huang L, Rubin D (2018) Retinal lesion detection with deep learning using image patches. Invest Ophthalmol Vis Sci 59:590–596. https://doi.org/10.1167/iovs.17-22721

    Article  PubMed  PubMed Central  Google Scholar 

  5. Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM (2017) Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol 135:1170–1176. https://doi.org/10.1001/jamaophthalmol.2017.3782

    Article  PubMed  PubMed Central  Google Scholar 

  6. Burlina P, Pacheco KD, Joshi N, Freund DE, Bressler NM (2017) Comparing humans and deep learning performance for grading AMD: a study in using universal deep features and transfer learning for automated AMD analysis. Comput Biol Med 82:80–86. https://doi.org/10.1016/j.compbiomed.2017.01.018

    Article  PubMed  PubMed Central  Google Scholar 

  7. Matsuba S, Tabuchi H, Ohsugi H, Enno H, Ishitobi N, Masumoto H, Kiuchi Y (2018) Accuracy of ultra-wide-field fundus ophthalmoscopy-assisted deep learning, a machine-learning technology, for detecting age-related macular degeneration. Int Ophthalmol. https://doi.org/10.1007/s10792-018-0940-0

  8. Grassmann F, Mengelkamp J, Brandl C, Harsch S, Zimmermann ME, Linkohr B, Peters A, Heid IM, Palm C, Weber BHF (2018) A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 125:1410–1420. https://doi.org/10.1016/j.ophtha.2018.02.037

    Article  PubMed  Google Scholar 

  9. Wilde C, Patel M, Lakshmanan A, Amankwah R, Dhar-Munshi S, Amoaku W, Medscape (2015) The diagnostic accuracy of spectral-domain optical coherence tomography for neovascular age-related macular degeneration: a comparison with fundus fluorescein angiography. Eye 29:602–609. https://doi.org/10.1038/eye.2015.44

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  10. Karri SPK, Chakraborty D, Chatterjee J (2017) Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration. Biomed Opt Express 8:579–592. https://doi.org/10.1364/BOE.8.000579

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  11. Lee CS, Baughman DM, Lee AY (2017) Deep learning is effective for classifying normal versus age-related macular degeneration optical coherence tomography images. Ophthalmol Retina 1:322–327. https://doi.org/10.1016/j.oret.2016.12.009

    Article  PubMed  PubMed Central  Google Scholar 

  12. Treder M, Lauermann JL, Eter N (2017) Automated detection of exudative age-related macular degeneration in spectral domain optical coherence tomography using deep learning. Graefes Arch Clin Exp Ophthalmol 256:259–265. https://doi.org/10.1007/s00417-017-3850-3

    Article  PubMed  CAS  Google Scholar 

  13. Prahs P, Radeck V, Mayer C, Cvetkov Y, Cvetkova N, Helbig H, Märker D (2017) OCT-based deep learning algorithm for the evaluation of treatment indication with anti-vascular endothelial growth factor medications. Graefes Arch Clin Exp Ophthalmol 256:91–98. https://doi.org/10.1007/s00417-017-3839-y

    Article  PubMed  Google Scholar 

  14. Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, McKeown A, Yang G, Wu X, Yan F, Dong J, Prasadha MK, Pei J, Ting MYL, Zhu J, Li C, Hewett S, Dong J, Ziyar I, Shi A, Zhang R, Zheng L, Hou R, Shi W, Fu X, Duan Y, Huu VAN, Wen C, Zhang ED, Zhang CL, Li O, Wang X, Singer MA, Sun X, Xu J, Tafreshi A, Lewis MA, Xia H, Zhang K (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172:1122–1131. https://doi.org/10.1016/j.cell.2018.02.010

    Article  PubMed  CAS  Google Scholar 

  15. Fang L, Cunefare D, Wang C, Guymer RH, Li S, Farsiu S (2017) Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search. Biomed Opt Express 8:2732–2744. https://doi.org/10.1364/BOE.8.002732

    Article  PubMed  PubMed Central  Google Scholar 

  16. Schaal KB, Freund KB, Litts KM, Zhang Y, Messinger JD, Curcio CA (2015) Outer retinal tubulation in advanced age-related macular degeneration: optical coherence tomographic findings correspond to histology. Retina 35:1339–1350. https://doi.org/10.1097/IAE.0000000000000471

    Article  PubMed  PubMed Central  Google Scholar 

  17. Tran T, Pham T, Carneiro G, et al (2017) A Bayesian data augmentation approach for learning deep models. In: Advances in Neural Information Processing Systems. pp 2794–2803

  18. Wallis TSA, Funke CM, Ecker AS, Gatys LA, Wichmann FA, Bethge M (2017) A parametric texture model based on deep convolutional features closely matches texture appearance for humans. J Vis 17:5. https://doi.org/10.1167/17.12.5

    Article  PubMed  Google Scholar 

  19. Wang Y, Zeng J (2013) Predicting drug-target interactions using restricted Boltzmann machines. Bioinformatics (Oxford England) 29:126–134. https://doi.org/10.1093/bioinformatics/btt234

    Article  CAS  Google Scholar 

  20. Ngiam J, Khosla A, Kim M, et al (2011) Multimodal deep learning. In: Proceedings of the 28th international conference on machine learning (ICML-11). pp 689–696

  21. Breiman L (2001) Random forests. Mach Learn 45:5–32

    Article  Google Scholar 

  22. Sindhwani V, Bhattacharya P, Rakshit S (2001) Information theoretic feature crediting in multiclass support vector machines. In: Proceedings of the 2001 SIAM international conference on data mining. Society for Industrial and Applied Mathematics, pp 1–18

  23. Wei JM, Yuan XJ, Hu QH, Wang SQ (2010) A novel measure for evaluating classifiers. Expert Syst Appl 37:3799–3809. https://doi.org/10.1016/j.eswa.2009.11.040

    Article  Google Scholar 

  24. Statnikov A, Aliferis CF, Tsamardinos I, Hardin D, Levy S (2005) A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. Bioinformatics (Oxford England) 21:631–643. https://doi.org/10.1093/bioinformatics/bti033

    Article  CAS  Google Scholar 

  25. Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45:171–186. https://doi.org/10.1023/A:1010920819831

    Article  Google Scholar 

  26. Schisterman EF, Faraggi D, Reiser B, Hu J (2008) Youden index and the optimal threshold for markers with mass at zero. Stat Med 27:297–315. https://doi.org/10.1002/sim.2993

    Article  PubMed  PubMed Central  Google Scholar 

  27. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

    Article  PubMed  CAS  Google Scholar 

  28. Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn 30:1145–1159. https://doi.org/10.1016/S0031-3203(96)00142-2

    Article  Google Scholar 

  29. Oh E, Yoo TK, Park EC (2013) Diabetic retinopathy risk prediction for fundus examination using sparse learning: a cross-sectional study. BMC Med Inform Decis Mak 13:106. https://doi.org/10.1186/1472-6947-13-106

    Article  PubMed  PubMed Central  Google Scholar 

  30. Smith RT, Chan JK, Nagasaki T, Ahmad UF, Barbazetto I, Sparrow J, Figueroa M, Merriam J (2005) Automated detection of macular drusen using geometric background leveling and threshold selection. Arch Ophthalmol 123:200–206. https://doi.org/10.1001/archopht.123.2.200

    Article  PubMed  PubMed Central  Google Scholar 

  31. Chen CY, Wong TY, Heriot WJ (2007) Intravitreal bevacizumab (Avastin) for neovascular age-related macular degeneration: a short-term study. Am J Ophthalmol 143:510–512. https://doi.org/10.1016/j.ajo.2006.10.004

    Article  PubMed  CAS  Google Scholar 

  32. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316:2402–2410. https://doi.org/10.1001/jama.2016.17216

    Article  PubMed  Google Scholar 

  33. Mokwa NF, Ristau T, Keane PA, Kirchhof B, Sadda SR, Liakopoulos S (2013) Grading of age-related macular degeneration: comparison between color fundus photography, fluorescein angiography, and spectral domain optical coherence tomography. J Ophthalmol 2013:385915–385916. https://doi.org/10.1155/2013/385915

    Article  PubMed  PubMed Central  Google Scholar 

  34. Castillo MM, Mowatt G, Elders A, Lois N, Fraser C, Hernández R, Amoaku W, Burr JM, Lotery A, Ramsay CR, Azuara-Blanco A (2015) Optical coherence tomography for the monitoring of neovascular age-related macular degeneration: a systematic review. Ophthalmology 122:399–406. https://doi.org/10.1016/j.ophtha.2014.07.055

    Article  PubMed  Google Scholar 

  35. Yang Q, Reisman CA, Wang Z, Fukuma Y, Hangai M, Yoshimura N, Tomidokoro A, Araie M, Raza AS, Hood DC, Chan K (2010) Automated layer segmentation of macular OCT images using dual-scale gradient information. Opt Express 18:21293–21307

    Article  PubMed  Google Scholar 

  36. Liu C, Liu A, Halabi S (2011) A min-max combination of biomarkers to improve diagnostic accuracy. Stat Med 30:2005–2014. https://doi.org/10.1002/sim.4238

    Article  PubMed  PubMed Central  Google Scholar 

  37. Yabuuchi H, Matsuo Y, Kamitani T, Setoguchi T, Okafuji T, Soeda H, Sakai S, Hatakenaka M, Nakashima T, Oda Y, Honda H (2008) Parotid gland tumors: can addition of diffusion-weighted MR imaging to dynamic contrast-enhanced MR imaging improve diagnostic accuracy in characterization? Radiology 249:909–916. https://doi.org/10.1148/radiol.2493072045

    Article  PubMed  Google Scholar 

  38. Srivastava N, Salakhutdinov RR (2012) Multimodal learning with deep boltzmann machines. In: Advances in neural information processing systems. pp 2222–2230

  39. Zhang D, Shen D, Alzheimer’s Disease Neuroimaging Initiative (2012) Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. NeuroImage 59:895–907. https://doi.org/10.1016/j.neuroimage.2011.09.069

    Article  PubMed  Google Scholar 

  40. Mitra J, Bourgeat P, Fripp J, Ghose S, Rose S, Salvado O, Connelly A, Campbell B, Palmer S, Sharma G, Christensen S, Carey L (2014) Lesion segmentation from multimodal MRI using random forest following ischemic stroke. NeuroImage 98:324–335. https://doi.org/10.1016/j.neuroimage.2014.04.056

    Article  PubMed  Google Scholar 

  41. Prashanth R, Deepak K, Meher AK (2017) High accuracy predictive modelling for customer churn prediction in telecom industry. In: International conference on machine learning and data mining in pattern recognition. Springer, pp 391–402

  42. Fellah S, Caudal D, De Paula AM et al (2013) Multimodal MR imaging (diffusion, perfusion, and spectroscopy): is it possible to distinguish oligodendroglial tumor grade and 1p/19q codeletion in the pretherapeutic diagnosis? AJNR Am J Neuroradiol 34:1326–1333. https://doi.org/10.3174/ajnr.A3352

    Article  PubMed  CAS  PubMed Central  Google Scholar 

  43. Larochelle H, Bengio Y, Louradour J, Lamblin P (2009) Exploring strategies for training deep neural networks. J Mach Learn Res 10:1–40

    Google Scholar 

  44. Yun YS, Kwon OW (1993) Postmortem change of adhesive forces between the retina and the retinal pigment epithelium. J Korean Ophthalmol Soc 34:111–116

    Google Scholar 

Download references

Author information

Author notes

  1. Tae Keun Yoo and Joon Yul Choi contributed equally to this work.

Authors and Affiliations

  1. Institute of Vision Research, Department of Ophthalmology, Yonsei University College of Medicine, 50-1 Yonsei-ro, Seodaemun-gu, Seoul, 03722, South Korea

    Tae Keun Yoo & Jeong Gi Seo

  2. Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea

    Joon Yul Choi

  3. Department of Electronics and Communication Engineering, Syed Ammal Engineering College, Ramanathapuram, Tamil Nadu, India

    Bhoopalan Ramasubramanian & Sundaramoorthy Selvaperumal

  4. Department of Medical Engineering Seoul, South Korea, Yonsei University College of Medicine, Seoul, South Korea

    Deok Won Kim

Authors
  1. Tae Keun Yoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joon Yul Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeong Gi Seo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bhoopalan Ramasubramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sundaramoorthy Selvaperumal
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Deok Won Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tae Keun Yoo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yoo, T.K., Choi, J.Y., Seo, J.G. et al. The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment. Med Biol Eng Comput 57, 677–687 (2019). https://doi.org/10.1007/s11517-018-1915-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-018-1915-z

Keywords

Search

Navigation