Elsevier

Ophthalmology

Volume 123, Issue 11, November 2016, Pages 2338-2344
Ophthalmology

Original article
Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability

https://doi.org/10.1016/j.ophtha.2016.07.026Get rights and content

Purpose

To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP).

Design

We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD.

Participants

Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units.

Methods

Expert classification of images of plus disease in ROP.

Main Outcome Measures

Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement).

Results

There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0–0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615).

Conclusions

There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

Section snippets

Methods

This study was approved by the Institutional Review Board at Oregon Health & Science University and followed the tenets of the Declaration of Helsinki. Written informed consent was obtained from parents of all infants in the i-ROP study.

Results

Table 1 displays the distribution of plus disease classification (plus, preplus, or normal) for all 8 experts (labeled 1–8) and the RSD, ranked from least severe average grade (top) to most severe average grade (bottom) for dataset A (100 images). The 8 experts are displayed in the same order for dataset B (34 images). The average percent RSD agreement was higher in dataset A (82%; range, 77%–94%) than dataset B (65%; range, 29%–91%) because of the large number of normal images with good

Discussion

This study analyzes the classification of plus disease by ROP experts, with the goal of examining the pattern of diagnostic discrepancies. Key findings from this study are: (1) even among ROP experts, there is limited agreement on diagnostic classification of plus disease; (2) diagnostic discrepancy in plus disease reflects consistent systematic biases for each expert as to the appropriate cut points for plus and preplus disease; and (3) a continuous severity score, instead of discrete

References (47)

  • J.S. Myung et al.

    Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training

    J AAPOS

    (2011)
  • R.K. Wong et al.

    Training fellows for retinopathy of prematurity care: a web-based survey

    J AAPOS

    (2012)
  • A. Sommer et al.

    Challenges of ophthalmic care in the developing world

    JAMA Ophthalmol

    (2014)
  • The International Classification of Retinopathy of Prematurity revisited

    Arch Ophthalmol

    (2005)
  • Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results

    Arch Ophthalmol

    (1988)
  • Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial

    Arch Ophthalmol

    (2003)
  • An international classification of retinopathy of prematurity

    Arch Ophthalmol

    (1984)
  • M.F. Chiang et al.

    Interexpert agreement of plus disease diagnosis in retinopathy of prematurity

    Arch Ophthalmol

    (2007)
  • J.D. Reynolds et al.

    Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies

    Arch Ophthalmol

    (2002)
  • R. Rao et al.

    Plus disease in retinopathy of prematurity: diagnostic impact of field of view

    Retina

    (2012)
  • E. Ataer-Cansizoglu et al.

    Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: image analysis features associated with expert diagnosis

    Transl Vis Sci Technol

    (2015)
  • K.M. Keck et al.

    Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk

    Retina

    (2013)
  • J.P. Campbell et al.

    Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis

    JAMA Ophthalmol

    (2016)
  • Cited by (68)

    • Eye and Vision Disorders

      2023, Avery's Diseases of the Newborn
    • Variability in Plus Disease Diagnosis using Single and Serial Images

      2022, Ophthalmology Retina
      Citation Excerpt :

      Overall, most changes in diagnosis in our study (18 of 24 [75%]) were toward more severe disease, although clinicians were inconsistent in whether serial versus single image review changed their diagnosis. It has been well known that interexpert agreement in ROP classification varies with a single image because of differences among cut points of vascular abnormality required for plus disease, differences in the field of view considered, identification of different vascular parameters by different clinicians, and differences in training and education.6–12 Serial images have been previously used as a tool in telemedicine to assess follow-up after intravitreal anti-VEGF injection.31

    • ROP screening with the Pictor Plus camera: a telemedicine solution for developing countries

      2022, Journal of AAPOS
      Citation Excerpt :

      This fact may explain the differences in sensitivity and specificity results among studies. It is important to note that there can be significant disagreements even among experts using BIO and wide-angle lens cameras for the diagnosis of plus and pre-plus disease.7,17,23 Previously published studies underscore the need for the development of technologies that improve accuracy and consistency in the diagnosis of retinal posterior pole vascular abnormalities: computer-based image analysis could be used with posterior pole images, such as the color and red-free images that are both automatically provided by the Pictor Plus digital camera, to generate quantitative and reproducible assessment scales with respect to vascular abnormalities, thus decreasing interpretive subjectivity.23

    View all citing articles on Scopus

    Financial Disclosure(s): The author(s) have made the following disclosure(s): J.D.R.: Financial support – Novartis, Basel, Switzerland

    M.F.C.: Consultant – Novartis, Basel, Switzerland; Scientific Advisory Board – Clarity Medical Systems, Pleasanton, CA

    R.V.P.C.: Consultant – Visunex Medical Systems, Fremont, CA

    Supported by the National Institutes of Health, Bethesda, Maryland (grant nos.: R01 EY19474 [J.K.C., D.E., S.O., K.E.J., R.V.P.C., M.F.C.], P30 EY010572 [J.P.C., S.O., M.F.C.], R21 EY022387 (J.K.C., D.E., M.F.C.], and T32 EY23211 (M.F.C., R.S.); the National Center for Advancing Translational Sciences at the National Institutes of Health, Bethesda, MD (Oregon Clinical and Translational Research Institute grant no.: TL1TR000129); Research to Prevent Blindness, New York, New York (J.P.C., S.N.P., J.D.R., M.X.R., S.O., K.E.J., R.V.P.C., M.F.C.); and the iNsight Foundation, New York, NY (R.V.P.C., K.E.J.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. No funding organizations had any role in the design or conduct of this research. Dr. Michael F. Chiang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Members of the Imaging and Informatics in Retinopathy of Prematurity Research Consortium: Oregon Health & Science University (Portland, OR): Michael F. Chiang, MD, Susan Ostmo, MS, Kemal Sonmez, PhD, J. Peter Campbell, MD, MPH; University of Illinois at Chicago (Chicago, IL): R. V. Paul Chan, MD, Karyn Jonas, RN; Columbia University (New York, NY): Jason Horowitz, MD, Osode Coki, RN, Cheryl-Ann Eccles, RN, Leora Sarna, RN; Bascom Palmer Eye Institute (Miami, FL): Audina Berrocal, MD, Catherin Negron, BA; William Beaumont Hospital (Royal Oak, MI): Kimberly Denser, MD, Kristi Cumming, RN, Tammy Osentoski, RN, Tammy Check, RN, Mary Zajechowski, RN; Children's Hospital Los Angeles (Los Angeles, CA): Thomas Lee, MD, Evan Kruger, BA, Kathryn McGovern, MPH; Cedars Sinai Hospital (Los Angeles, CA): Charles Simmons, MD, Raghu Murthy, MD, Sharon Galvis, NNP; LA Biomedical Research Institute (Los Angeles, CA): Jerome Rotter, MD, Ida Chen, PhD, Xiaohui Li, MD, Kent Taylor, PhD, Kaye Roll, RN; Massachusetts General Hospital (Boston, MA): Jayashree Kalpathy-Cramer, PhD; Northeastern University (Boston, MA): Deniz Erdogmus, PhD; Asociacion para Evitar la Ceguera en Mexico (APEC) (Mexico City): Maria Ana Martinez-Castellanos, MD, Samantha Salinas-Longoria, MD, Rafael Romero, MD, Andrea Arriola, MD, Francisco Olguin-Manriquez, MD, Miroslava Meraz-Gutierrez, MD, Carlos M. Dulanto-Reinoso, MD, Cristina Montero-Mendoza, MD.

    Author Contributions:

    Conception and design: Kalpathy-Cramer, Campbell, Erdogmus, Sonmez, Swan, Chan, Chiang

    Analysis and interpretation: Kalpathy-Cramer, Campbell, Erdogmus, Tian, Kedarisetti, Moleta, Reynolds, Hutcheson, Shapiro, Repka, Ferrone, Drenser, Horowitz, Sonmez, Swan, Ostmo, Jonas, Chan, Chiang

    Data collection: Kalpathy-Cramer, Campbell, Erdogmus, Tian, Kedarisetti, Moleta, Reynolds, Hutcheson, Shapiro, Repka, Ferrone, Drenser, Horowitz, Sonmez, Swan, Ostmo, Jonas, Chan, Chiang

    Obtained funding: Chiang, Erdogmus, Kalpathy-Cramer, Chan

    Overall responsibility: Kalpathy-Cramer, Campbell, Moleta, Chan, Chiang

    Both authors contributed equally as first authors.

    View full text