Odysight: A Mobile Medical Application Designed for Remote Monitoring—A Prospective Study Comparison with Standard Clinical Eye Tests

OdySight is a mobile medical video game available on a smartphone or tablet by prescription only. It contains a puzzle game as well as medical modules to test monocular vision (near visual acuity, contrast sensitivity, and the detection of metamorphopsia and scotoma via a digital Amsler grid). A unique, patented technology using the front camera of the smartphone ensures that the tests are performed at a standardized distance (40 cm/15.7 in. between the participant’s eye and the device) and with adequate ambient light, as measured by the device (note: these features were not used during this study). The results from the visual tests are sent via a secure server to an online dashboard which the physician can access in real time (Fig. 1). Alerts are sent to both the participant and physician when there is a significant decline in vision. A recommendation is made to the participant to schedule a physician visit.

Study participation was offered to people who were visiting the hospital for out-patient visits, as well as to their accompanying friends and family members. Each participant was noted to have one or multiple eye diseases, or none.

In order to assess the validity of the medical modules on OdySight, we conducted an open-label, single-arm, prospective, single-site study at the Quinze-Vingts National Ophthalmology Hospital, in Paris, France (ClinicalTrials.gov Identifier: NCT03457441). The research followed the tenets of the Declaration of Helsinki of 1964, as revised in 2013, and approval of the experimental protocol was obtained from an ethics committee (Comité de Protection des Personnes Ile-de-France VII, France, n° 18-005). Informed consent was obtained from each participant prior to any examination.

Participants in the study met the following key inclusion criteria: age ≥ 18 years; best corrected visual acuity (BCVA) between 0.0 and 1.0 logMAR; ability to recognize letters of the alphabet and to read French; ability to correctly distinguish body laterality; and affiliated with or beneficiary of the French healthcare system. Participants were excluded if: they presented with a pathology that was considered by the investigator to be capable of affecting the quality of the main evaluation criteria (e.g., mental illness, Parkinson disease); they were pregnant or breastfeeding; or they were not considered by the investigator or designee to correctly use the OdySight modules after the training session. To ensure a wide range of visual acuity, each participant’s eye was assigned to one of the three cohorts according to the level of near visual acuity, measured at a distance of 40 cm from the Sloan ETDRS letter chart (40-cm ETDRS): cohort 1, with near visual acuity of between + 1.0 and + 0.7 logMAR; cohort 2, with near visual acuity of between + 0.6 and + 0.3 logMAR; or cohort 3, with near visual acuity of between + 0.2 and 0.00 logMAR.

For the purpose of the trial, a scaled-down version of the OdySight app (in which the puzzle and the technical unit checking ambient light and distance to screen were removed) had been installed on two iPhone 6 smartphones. The OdySight medical tests and their algorithms had not been modified. Again for the purpose of the trial, the examiner was responsible for the navigation within the app, and the participants were asked only to perform the tests.

The sample size needed, driven by the number of eyes for the primary endpoint (to assess the agreement between a smartphone-based evaluation of near visual acuity with OdySight and a standardized method [Sloan ETDRS near vision letter chart]) was defined by performing a careful review of the relevant literature. The limits of agreement (LoA) technique was retained to analyze the results and to calibrate the sample size of this study (see Conduct of the Study, below).

The lack of agreement was estimated by the mean difference of the two measurements (d) and the standard deviation of the differences (s). Providing differences within d ± 1.96 s would not have been clinically important. Taking into account that these LoA are only estimates of the values which apply to the whole population, we viewed the sample size estimation as an attempt to obtain sufficient precision on these LoA. It can be shown that the standard error of d is root(s²/n), where n is the sample size and the standard error of d − 1.96 s and d + 1.96  s is about root(3 s²/n) [8, 9]. Bland and Altman recommended a minimum of 100 statistical units for a good sample size because this number leads to a 95% confidence interval (CI) of approximately ± 0.34 s which appears to be a reasonable and accurate estimate. To ensure that 100 studied eyes were available in the statistical reporting of agreement between the methods of measurements, and to take into account a possible lack of measurements for any reason, the study was planned to include between 60 and 120 patients as a whole.

During screening, inclusion criteria were checked, optical coherence tomography (OCT) was performed, and the participant’s BCVA was determined so that each eye was assigned to the corresponding cohort. The three medical modules were then presented to the participant. Because it is known that a learning phenomenon can occur, participants were asked to perform practice tests following instructions given by the examiner. At the end of this training session, the examiner estimated whether or not the participant was sufficiently capable of using the app. Eyes from participants who fulfilled all of the criteria were included, and a participant could be included for one or both of his/her eyes.

During the evaluation, participants performed tests to assess visual acuity, contrast sensitivity, and detection of scotoma and metamorphopsia with both standard tools and digitized tests from OdySight. Participants did not receive dilation treatment and, if relevant, were prompted to remove their contact lenses at least 1 h before any exam. All tests were performed at a distance of 40 cm, with the head of the participant leaning against an ophthalmological chin piece and the smartphone or chart fixed on a desk at an adequate height. Before each evaluation, distance was checked with a tape measure and luminosity was checked with a lux meter. All tests were monocular, the eye tested was equipped with the adequate correction, and the eye not being tested was covered with an occluder. The order of the tests (standard exams first vs. OdySight modules first) as well as the duration of resting breaks varied for each participant based on the discretion and assessment of the examiner.

All statistical outputs were generated using SAS® version 9.4 software (SAS Institute, Cary, NC, USA).

Visual acuity was assessed with two standard methods: with the ETDRS chart at a test distance of 4 m (4-m ETDRS)—and at 1 m if needed with + 0.75 D added for correction—and with the Sloan ETDRS chart at 40 cm (40-cm ETDRS). All tests were performed in a dedicated testing room with adequate illumination (≤ 161.5 lx). Visual acuity was determined by asking the participant to read each letter starting from the first line until the smallest letter readable on the chart.

During the same visit, visual acuity was also evaluated with the visual acuity module of OdySight.

Results of the visual acuity tests were recorded in terms of number of letters read.

Contrast sensitivity was assessed with the Pelli–Robson chart at 1 m, in a dedicated testing room with adequate illumination (≤ 161.5 lx). Participants removed their own glasses and instead wore a trial frame equipped with the adapted correction for distant vision increased by + 0.75 D. The contrast sensitivity level was determined by asking the participant to indicate each letter, starting from the first letter until the last readable letter. The examiner recorded each letter identified correctly by the participant, and the last triplet of letters was validated if the participant was able to read at least two of them. Results were recorded in LogCS.

During the same visit, contrast sensitivity was also evaluated with the contrast sensitivity module of OdySight.

Results of contrast sensitivity were record in logCS.

The presence of metamorphopsia and/or scotoma was assessed using an empty Amsler grid printed on a white sheet of paper (10 × 10-cm grid) and a felt tip pen. The participant was asked to draw lines (or wavy lines) where distortion was seen, or to fill an area where any type of spot was seen (e.g., black spot, white spot, area where the grid disappears) while fixing the black dot.

During the same visit, the presence of metamorphopsia and/or scotoma was also evaluated using the digital Amsler Grid module on OdySight.

After the participants performed each test, the clinical team evaluated the presence or absence of metamorphopsia and/or scotoma for each type of grid. Results were recorded as the presence or absence of metamorphopsia and/or scotoma.

For visual acuity and contrast sensitivity, statistical analysis was performed using the LoA technique provided by Bland and Altman (B&A). This method evaluates the agreement between two quantitative measurements by constructing LoA and deducing the resulting bias, which is the mean of differences between the two methods. The closer the bias is to 0, the more it indicates good agreement between the methods. LoA are an estimate of the true agreement limits for the entire population, whereby narrower limits reflect better agreement between two techniques. However, this method only defines the interval of agreements, it does not state whether those limits are acceptable or not. Acceptable limits must be defined a priori, based on clinical necessity, biological considerations, or other goals. Results are displayed on the B&A plot with the difference between the two measurements on the Y-axis, and the average of the two measurements on the X-axis [8, 9].

Among the evaluated eyes, 15.8% had no eye condition, 50.8% of eyes had AMD, 8.3% had retinitis pigmentosa, 8.3% had a Stargardt disease, and 4.2% had glaucoma (Table 2).

The OCT examination detected abnormalities in at least one eye for approximately 90.8% (n = 69/78) of participants: approximately 40% of eyes had macular atrophy, 35% had drusen, 14% had either hypo- or hyperpigmentation of retinal pigment epithelium, and 10% had choroidal neo-vascularization. OCT was not performed on four eyes (Table 3).

All 78 of the included participants underwent OdySight training for the three medical modules, and all were considered to be able to correctly use the application.

The B&A analysis showed a mean difference in units of 0.53 letters (95% CI − 0.42, 1.48) and a 95% LoA of between − 9.75 and 10.82 letters between the two methods. According to the LoA, in this study, 95% of the differences were between − 9.75 and 10.82 letters (Fig. 6).

In Fig. 6, the cloud of points for cohort 3 (good visual acuity) is more compact than those for the other two cohorts. Only cohorts 1 and 2 contain points beyond the + 1.96 s and − 1.96 s LoA, indicating that the agreement between the two methods is better for participants with good visual acuity (cohort 3). This result was confirmed by the analysis performed by cohort (Table 4).

Almost 90% of eyes (107 eyes) had a difference less than or equal to nine letters between the two methods. Of note, 94 eyes (78.3%) had a difference less than or equal to five letters.

The B&A analysis showed a mean difference in units of − 1.53 letters (95% CI − 2.78, − 0.27) and a 95% LoA of between − 15.16 and 12.11 letters between the two methods. According to the LoA, in this study, 95% of the differences were between 15.16 letters and 12.11 letters (Fig. 7).

For the 120 evaluated eyes, 99 (82.5%) showed a difference of ± 9 letters in near visual acuity between measurements by the ETDRS method and those by OdySight. Of note, a total of 78 eyes (65.0%) showed a difference of ± 5 letters between measurements by these two methods.

The B&A analysis showed a mean difference in units of − 0.16 logCS (95% CI − 0.20, − 0.13) and a 95% LoA of between − 0.54 and 0.22 between the two methods. According to the LoA, 95% of the differences were between − 0.54 and 0.22 logCS. In other words, the differences were between − 10.8 letters (3.6 lines on Pelli–Robson) and + 4.4 letters (1.46 lines on Pelli–Robson) (Fig. 8).

For the 120 evaluated eyes, 74 (61.7%) showed a difference of less than or equal to ± 0.15 logCS (three letters, one line) when contrast sensitivity as measured by the Pelli–Robson method and by OdySight was compared. A total of 105 (87.5%) eyes showed a difference of less than or equal to ± 30 logCS (six letters, two lines) between the two methods.

At least one eye with metamorphopsia was detected in 50 eyes using the paper Amsler grid and in 50 eyes using the OdySight module. At least one scotoma was detected in 50 eyes using the paper Amsler grid and in 50 eyes using the OdySight module. There was only one participant for whom the OdySight app did not detect metamorphopsia while the paper Amsler grid did (McNemar test, p value = 1), and also one participant for whom the paper Amsler grid did not detect metamorphopsia while the OdySight app did (McNemar test, p value = 1) (Fig. 9). For these cases, similar results were noted in the detection of scotoma.

OdySight received CE Marking as a class 1 medical device in the EU in May 2018 and, as of this manuscript submission, more than 40 ophthalmologists in France are prescribing it to their patients. Physicians and their clinical staff propose the application to those patients with chronic eye diseases who may benefit from remote monitoring. A starter kit composed of the basic information to install the game and perform the tests, as well as a stand to hold a smartphone, are provided to each patient. Patients then download OdySight from the iOS or Android store onto one of 3000+ compatible smartphones and tablets, play the puzzle game, and perform the eye tests on a regular basis, between clinic visits, from their home or other remote locations. At the end of each test, the data are transmitted via a secure server to a dashboard in the clinic. Significant declines in test results trigger an alert for both the physician and patient. To date, multiple anecdotal reports from physicians have demonstrated that an alert resulted in an early clinic visit, confirmation of visual decline, and an alteration in the treatment plan, such as an early injection of anti-VEGF medication for patients with AMD. Tilak Healthcare is now registered with the U.S. Food and Drug Administration, and OdySight is listed as a class I medical device in the US.