The diagnostic accuracy of an intelligent and automated fundus disease image assessment system with lesion quantitative function (SmartEye) in diabetic patients

This study was approved by the Ethics Committee of the Shanghai Eye Disease Prevention & Treatment Center. No personal information could be recognized or be disclosed from the imaged used in this study. This study was carried out in accordance with the Declaration of Helsinki. The object of the study was to evaluate the screening performance and accuracy of SmartEye. During the DR screening program (Shanghai Diabetic Eye Study in Diabetics (SDES), NCT03579797) in Shanghai from Apr 2016 to Aug 2017, 19,904 fundus images from 6013 patients were acquired. The screening program was organized by the Shanghai Eye Disease Prevention & Treatment Center (SEDPTC). All retinal images were collected with a non-mydriatic fundus camera (NW 400, Topcon, Japan) by community healthcare professionals who had been trained by fundus disease experts in SEDPTC. Two fundus images centered on the macula and on the optic disc were taken from each eye of each DM patient. For grading patients’ images, three ophthalmologists who are retinal specialists were invited to decide whether the images were qualified for grading and then graded the fundus images independently. The image grading standard is referred to the proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales [15]. Once the independent grading were finished, the grading ophthalmologists had a consensus meeting to discuss images without initial agreement on the image quality or DR stages until an agreement was reached.

The automatic DR screening system (SmartEye, version 3.0) identifies DR through recognizing and analyzing lesions in patient fundus images and comparing them against images of classical DR lesions.

The fundus images was firstly processed including drying and normalization. The purpose of normalization is to ensure that the color, brightness, and exposure of images are in the same gray value range to improve the feature extraction accuracy of mass images. Then based on the global gray value analysis, we performed grayscale threshold segmentation with the features of color, brightness, contrast, and combined with mathematical morphology to extract lesions of fundus, such as microaneurysm, hemorrhage, and exudation. To obtain correct information from different fundus images, the lightness, brightness, and saturation of the fundus images are normalized by referring to a standard image, and the vascular boundary is extracted according to intensity threshold separation and supervised classification based on color characteristics. Then the features of hemorrhagic points and microaneurysms in the fundus images are identified comprehensively with mathematical morphology and a support vector machine. On the basis of the feature integration theory of the human visual attention mechanism and Bayesian theory, the shape, color, and correlation of lesions are analyzed to discriminate the type of disease. We utilize the classification method of decision tree to analyze the data characteristics of different grades of DR, and generate DR classification rules based on the idea of multiple regression. Finally disease staging is performed according to the staging standards for DR. If characteristic lesions are found in the fundus images, suspicion of DR is determined. Another function of SmartEye is quantifying the lesion of fundus based on pixels. SmartEye consists of the following modules:

Before the detection of fundus anatomical structure and lesions, the region of interest (ROI) is established by adaptive thresholding and template matching. Black background is removed. Because of differences in the resolution, color, luminance, and quality of images, all fundus photographs are adjusted according to the recommended reference value before evaluation, to ensure the precision of analysis results. The vascular outline, an important structure of the retina, should be marked correctly. SmartEye identifies the vascular borderline precisely through brightness threshold segmentation and color discrimination. The optical disc is recognized according to its brightness and shape, as well as the vascular direction. “Red lesions” such as microaneurysms, hemorrhages, and neovascularization are the critical characteristic lesions. Small red lesions are recognized on the basis of mathematical morphology, and large red lesions are identified through color discrimination. The shape, structure, color, and contrast of the focus are analyzed before a determination of a “red lesion” is made. Exudation and cotton-wool spots are two kinds of bright lesions in DR. SmartEye identifies such lesions through analyzing their shape, contrast, and color. Briefly, the image preprocessing process includes the following steps:establishing region of interest (ROI), normalized processing, identifying the vascular borderline, identifying and extracting the papilla disc before extracting DR lesions, to avoid its interference with the extraction of the lesion, identifying red lesions, and identifying bright lesions. The analytic steps and demonstration figures are shown in Fig. 1 and Fig. 2. SmartEye then marks the lesion area size at the pixel level (as shown in Fig. 3). Hemorrahge is marked with green, and exudation is marked with blue. Moreover, SmartEye is sensitive to “red lesions” and can identify such lesions that are difficult for the human eye to recognize (as shown in Fig. 4). The outcomes of different modules in SmartEye were combined, and the final diagnosis was acquired for further confirmation of DR. SmartEye was also able to calculate the hemorrhage/exudation lesion number and area. The lesion area was evaluated on the basis of pixel area.

Sensitivity and specificity values were calculated for the entire group of participants and for subgroups with different stages of diabetes according to the fundus characteristics. Differences in sensitivity and specificity between the machine and clinician diagnosis were analyzed with McNemar’s test. The statistical analysis was performed in SPSS (version 19.0.0 for Mac; SPSS Inc., Chicago, IL, USA).

Sensitivity was defined as the proportion of diseased people correctly diagnosed, and specificity was defined as the proportion of non-diseased people correctly diagnosed. The rate of misdiagnosis was 1 - specificity, and the rate of missed diagnosis was 1 - sensitivity. The positive and negative predictive values are the proportions of positive and negative results in statistics and diagnostic tests that are true positive and true negative results, respectively.

A consistency check was used to determine the agreement in classification between machine and clinician diagnosis, expressed as a k value. The values for k were classified as follows: < 0.2, poor; 0.21 to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, good; and > 0.81, excellent. ROC analysis (sensitivity on the vertical axis and (1 – specificity) on the horizontal axis) was applied to evaluate the accuracy of SmartEye.

The diabetes eye classification from SmartEye was compared with the diagnoses made by clinicians (Table 1). In the sample of 19,904 images screened, 8369 (42.0%) patient images were disqualified for reading. The disqualified images usually had the features including fuzziness, large dark area, eyelash interference, overexposure to light or position deviation. Because that most of the participants were elderly people above 50 years-old, small pupil and various levels of lens opacity were commonly seen in these patients. Pupil dilation with cycloplegia was inconvenient because that only a few community healthcare centers having ophthalmology clinic in Shanghai and it is risky to use cycloplegia without ophthalmologists monitoring. 9266 (46.6%) patient fundus images appeared normal; 618 (3.1%) patients had mild nonproliferative diabetic retinopathy (NPDR); and 1190 (6.0%) patients had moderate NPDR. The number of patients with severe NPDR or above was 186 (0.9%). Among the 11,535 photos qualified for reading, the DR prevalence was 17.3%.

As shown in Table 2, the sensitivity of correctly classifying diabetic eyes stratified for no DR, mild NPDR, moderate NPDR, severe NPDR, PDR, and pan-retinal photocoagulation (PRP) were 86.19, 83.18, 88.64, 89.59, 85.02, and 84.96%, respectively. The specificity of the different DR stages was 63.07, 70.96, 64.16, 70.38, 74.79, and 74.33%, respectively. The diagnostic accuracy of severe NPDR was the highest (sensitivity, 89.59%; specificity, 70.38%). The kappa value increased with DR severity. The area under the ROC curve (AUC) of discriminating DR from normal fundus was 0.73 (p < 0.000, 95%CI: 0.694–0.765). For different stages of DR, the AUC values were PRP, 0.80 (0.79, 0.81); PDR, 0.80 (0.79, 0.81); severe NPDR, 0.80 (0.79, 0.80); moderate NPDR, 0.77 (0.76, 0.77); and mild NPDR, 0.78 (0.77, 0.79), respectively (Figs. 5 and 6). Thus, SmartEye had a high diagnostic accuracy. The AUC of discrimination of severe NPDR and above was greater than that of moderate NPDR and mild NPDR.

Several fundus images with high quality were chosen from different DR gradings, and the hemorrhage/exudation lesion number and area were calculated with SmartEye. The results are shown in Tables 3 and 4. Mild and moderate NPDR cases were included in group 1, severe NPDR cases were classified as group 2, and PDR cases were classified as group 3. There were significant differences in total hemorrhage area, hemorrhage number, maximum hemorrhage area, total exudation area, exudation number, and maximum exudation area among the three groups (Table 3). The sensitivity of pointing hemorrhage was from 88%(8%) to 96%(12%), and the sensitivity of pointing exudation was from 93%(17%) to 96%(11%) compared to the manual grading. The total hemorrhage area, maximum hemorrhage area, total exudation area, and maximum exudation area increased with the DR stage. The hemorrhage lesion number and exudation lesion number in group 2 (severe NPDR) were larger than those in group 3 (PDR). The hemorrhage lesion number and maximum hemorrhage area were different between each set of two groups. There was no difference in the total hemorrhage area in group 2 and group 3. The exudation lesion number increased significantly when DR progressed to severe NPDR. For the lesion quantification accuracy, lesion quantification was performed with SmartEye on these images for three times, and the results from three measurement were highly consistent (kappa value 1.0).