Deep learning-based postoperative visual acuity prediction in idiopathic epiretinal membrane

This study was approved by the Institutional Review Board of the Tianjin Medical University Eye Hospital (TMUEH, No. 2021KY(L)-03). In compliance with the tenets of the Declaration of Helsinki, all private information was removed in advance, and written consent was obtained from the patients.

This study retrospectively collected OCT images from iERM patients in Tianjin Medical University Eye Hospital (Tianjin, China) from July 2016 to December 2021. The patients’ gender, age, preoperative BCVA, duration of symptoms (e.g., progressive visual impairment and metamorphopsia), lens status, and 6-month postoperative BCVA were retrieved from the electronic medical records. The surgical process of VMP is shown in the online Supplementary Method 1. The inclusion criteria were as follows: (1) patients who underwent ophthalmic examination and were diagnosed with iERM, (2) underwent macular OCT scanning before surgery, (3) underwent uneventful VMP surgeries, and (4) had reliable 6-month postoperative BCVA records. The exclusion criteria included eyes with: (1) secondary ERM and concomitant diseases such as diabetic retinopathy, vein or artery occlusion, and glaucoma; (2) corneal opacity or any other ocular disease that might influence visual acuity; and (3) significant cataract prior to surgery or during the 6-month follow-up period. Spectral domain-OCT scanning was performed using Heidelberg Spectral OCT (Heidelberg Engineering, Germany) on all eyes at baseline. The details of the OCT examination and OCT parameter measurements are described in the online Supplementary Method 2.

Finally, 5,304 qualified OCT images were collected from 442 eyes (12 images from each eye). Specifically, 12 B-scans of each eye were seen as a whole, which was paired with patient data (BCVA). Subsequently, the images were randomized into a training dataset (265 eyes, 60.0% of all the eyes) for model development, a validation dataset (89 eyes, 20.1% of all the eyes) for parameter adjustment, and a testing dataset (88 eyes, 19.9% of all the eyes) for internal test and model evaluation.

Inception-Resnet-v2 network [1, 18], a DNN with 164 convolutional layers, was employed to estimate the 6-month postoperative BCVA. The parameters of the DNN were initialized against the ImageNet pretrained model. Preoperative OCT images and clinical data were used as the input data. Based on the requirements of Inception-Resnet-v2 network, the OCT images were preprocessed to normalize the input data. We removed saturated pixels from the raw images with an intensity value of 255, and then resized them into 299 × 299 pixels. For each training iteration, a Huber loss was used as the objective loss function, and an adaptive momentum (Adam) algorithm was used to update the network parameters via back-propagation. At every epoch, the performance of network was assessed using the tuning dataset. Back-propagation was repeated for all training images until the network reached satisfactory performance. The network was a regression model output according to the final fully connected layer and a regression layer in the last layer. The batch size was 30, and the learning rate was set to 0.001. The illustration of the pipeline of the work is demonstrated in Fig. 1. In our study, we employ a Convolutional Neural Network (CNN) specifically designed to process the OCT image data. Concurrently, we use a Multilayer Perceptron (MLP) to handle the structured patient data, including age, pre-operative visual acuity, gender, and duration of symptoms. The processed features from both the CNN and MLP are then seamlessly fused to create a comprehensive feature set. Utilizing this enriched feature set, our multimodal deep fusion network is capable of predicting the post-operative visual acuity with improved accuracy. This robust fusion approach capitalizes on the strengths of both image and structured data, thereby enhancing the predictive power of our model.

The 265 eyes of the training set from the DL model were subsequently used to create a stepwise multiple regression model for the prediction of 6-month postoperative BCVA. Gender, age, preoperative BCVA, symptom duration, preoperative IS/OS integrity, foveolar detachment and CMT were all considered independent variables. Specifically, IS/OS integrity was graded into two groups: intact and disrupted (the hyporeflective disruption of the hyperreflective IS/OS junction) [9, 19]. The coefficient of determination (R²) of this stepwise multiple regression model was calculated using the 88 eyes that had been included as the testing dataset in the DL model.

The metrics used to show the differences between the actual 6-month postoperative BCVA and the estimated BCVA were mean absolute error (MAE), root mean square error (RMSE), and the R², which are defined as:where N denotes the number of eyes, \({\widetilde{y}}_{i}\) denotes the actual 6-month postoperative BCVA, and \({y}_{i}\) denotes the estimated BCVA.

The prediction error was calculated by subtracting the estimated BCVA from the actual BCVA. The accuracy is defined as the percentage of BCVA prediction errors within ± 0.20 logMAR [20], namely:where N, \({\widetilde{y}}_{i}\), and \({y}_{i}\) have the same meanings as above; the function \(I\left(\cdot \right)\) returns 1 if the condition in (\(\cdot\)) is true, else returns 0.

Gradient-weighted class activation mapping (Grad-CAM), a method for creating "visual explanations", increases the transparency of judgments made by a broad class of DNN-based models. A coarse localization map highlighting the key areas in the image for concept prediction is produced by the Grad-CAM image by using the gradients of any target concept and flowing into the final convolutional layer [21]. To interpret the estimations and increase model transparency, Grad-CAM was used to highlight the discriminative regions of input images in BCVA prediction [21].

Ocular parameters in the training, validation, and testing datasets were compared using an independent t-test. The alignment of the estimated and actual BCVA was presented by scatter plots. Bland–Altman plotting was used to visualize the agreement between the estimated and actual values of BCVA. To implement and deploy the network, MATLAB (2020a) was used for training and evaluation. The model was trained on a GPU of NVIDIA RTX3090 with CUDA version 11.3 and cuDNN 8.0. The Inception- ResNet-V2 network architecture used in this work was publicly available in the deep learning tools package. To determine the relationships between the actual and estimated postoperative BCVA values from the DL and stepwise multivariate regression model (gender, age, symptom duration, preoperative BCVA, IS/OS integrity, foveolar detachment and CMT), R² was calculated. Each variable’s entry and exit criteria for the model were determined using the F test, with P values set at 0.05 and 0.1, respectively (in collinearity diagnostic tests, all variance inflation factors were < 10, showing no multicollinearity). The following equation represents a stepwise regression model: y = β0 + β1X1 + β2X2 + ⋯ + βpXp + ε, where β is the regression model’s coefficient, and ε is referred to as the error term. Differences with a P-value of 0.05 or less were considered statistically significant.