mHAT app for automated malaria rapid test result analysis and aggregation: a pilot study

The mHAT app is a mobile-friendly, web application hosted on a development server at Vanderbilt University, Nashville, USA. For the duration of the study it was accessible in any web browser by its public IP address.

mHAT was designed as a web application (hosted by a webserver and running in the browser of a mobile device) rather than a native application that is downloaded and runs locally on the device. This decision was guided by the expected use case, and was influenced by the broad device compatibility, and the ease and speed with which software updates can be deployed, of web applications. The architecture described above can work on any web-connected device (i.e., phones, tables, laptops, desktops) and interchangeably across operating systems. The mHAT app does require network connectivity, but new developments that enable progressive web apps will allow future versions of the software to have similar functionality offline as well.

Computer vision for analysis of RDTs is a substantial endeavour. These algorithms must be tolerant of lighting and background variations and different camera resolutions. A more thorough description of the technical implementation of a computer vision algorithm is detailed in Additional file 1. Here, the app use is briefly described. First, users input any information they would like to associate with this test (e.g., case ID, test ID, any other notes). From within the app, the user presses a button to take a photograph of the RDT they would like to analyse. Prior to taking the photograph, the web app asks to access the user’s GPS coordinates to assist with spatial infectious disease surveillance. Preliminary image processing is performed and determines if the photograph must be retaken, or if it is suitable for full analysis. If the preliminary photograph is suitable, a preview image is presented to the user with annotations of key test features. The user has the choice to accept this image and proceed with analysis, or they can elect to take another photograph and restart the process. If the user proceeds with analysis, the full results are presented to the user, including: a qualitative ‘Positive’/ ‘Negative’ result as well as the numerical values for test line signal, control signal and signal ratio. The result is transferred to a custom REDCap (a widely used, customizable research electronic database) [30‐32] project for storage, along with the user that uploaded the test information, GPS coordinates, and a timestamp. This information facilitates spatio-temporal surveillance, as well as allowing national healthcare systems to observe the locations of individual community healthcare workers and their teams. A detailed outline of the development of the mHAT software and image processing analysis has been included in the Additional file 1: Fig. S1).

For any new test, computer vision and image processing features must be experimentally optimized. This is a result of the different shapes of cassettes, the colours of the test and control line signals, markings and brandings on the test cassettes, different spatial differences during manufacturing of each test. However, this process is relatively straightforward, and with minimal effort can enable the use of the software to analyse a variety of tests. This optimization was performed for several malaria tests (including multiplexed tests with multiple test lines), multiple HIV test kits, schistosomiasis test kits, pregnancy and ovulation test kits. Since the algorithm utilizes feature recognition, it has the potential to automatically detect the type of test from the photograph, without additional user input. This would have obvious utility for organizations that heavily rely on point-of-care testing.

Initial image-processing optimization was performed using SD Bioline Malaria Ag P.f. tests (Standard Diagnostics, South Korea). D6 Plasmodium falciparum parasite was cultured in-house. Human whole blood (pooled, citrate phosphate dextrose anticoagulated) was purchased from BioIVT. Samples were prepared by spiking whole blood with D6 parasite culture at 5000 p/μL and diluting serially by a factor of 2.5. The parasite-positive dilution series consisted of 6 total concentrations: 5,000, 2,000, 800, 320, 128, and 51.2 parasites per μL of whole blood. A whole blood sample with no parasite culture was used as a negative control. These mock specimens were applied to the RDTs, which were run according to manufacturer’s instructions. Briefly, a 5-μL sample was added to the specimen well of the test; 4 drops of proprietary running buffer were then added to the diluent well, and the test was allowed to develop for 15 min at room temperature. After 15 min, photos were taken in the web-based mHAT app on both an iPhone 8 + (Apple, USA) and Samsung Galaxy J3 (Samsung, South Korea). All tests were analysed in method triplicate, resulting in 3 unique RDTs that were imaged for each parasite density. All mHAT results were automatically recorded and stored in a pre-configured REDCap project. Immediately after mobile phone imaging, each test was removed from its plastic casing and the nitrocellulose lateral flow test strips were analysed on a Qiagen ESEQuant lateral flow reader (QIAGEN Lake Constance GmbH, Stokach, Germany). Lateral flow readers are useful laboratory- or clinic-based tools for objective RDT analysis [33]. However, these instruments require continuous power, trained personnel, and cost thousands of dollars, making them unsuitable for field usage in remote settings. There are other optical approaches for RDT analysis [34, 35], but these have been demonstrated only in research settings and are not commercially available. Therefore, the QIAGEN reader represents one such tool that has been used previously as a benchmark for lateral flow assay analysis [36] and is commercially available.

Macha Research Trust (MRT) relies on several teams of researchers, physicians and community healthcare workers to collect surveillance data and provide treatment for the population in and surrounding Macha, using a hub-and-spoke model of satellite clinics. Aggregate surveillance data are validated quarterly by an independent team of MRT researchers, and collected weekly from each of the 14 MRT outposts in the area (Fig. 2).

RDT collection was performed in collaboration with the outpatient clinic (OPC) triage team and the field research team at MRT (Macha, Zambia), the latter as part of an existing Institutional Review Board-approved malaria epidemiology study. All patients were first screened by underarm temperature. In the OPC, SD Bioline Malaria Ag P.f. tests were performed for all patients presenting with a temperature at or above 37 °C. In the field, SD Bioline Malaria Ag P.f RDTs were performed for all patients presenting with a fever at or above 38 °C.

Upon completion of an RDT, the administering MRT clinical worker recorded the result in a physical ledger along with patient name age, village, history of travel, weight, blood pressure, pulse, and temperature. The data were not available to the mHAT app team. As the MRT clinicians recorded the necessary data for their reports, the RDT was imaged using the mHAT app and the result was recorded and stored in the REDCap database. While additional time elapsed may change the amount of blood clearance on RDTs, resulting in changes in contrast between the test line and the background of the nitrocellulose, RDTs were analysed by visual inspection immediately prior to being analysed by mHAT. The time from visual inspection of the RDT to successful mHAT analysis was under 1 min in most cases, with the exception of early tests using the first iteration of the mHAT app. This elapsed time was determined to be unlikely to introduce any significant error. The number of image re-takes prior to analysis and the binary mHAT result were additionally recorded. The image orientation and photo acceptance features were improved iteratively over the course of this pilot study.

A controlled laboratory environment is an idealized setting for a mobile application that uses computer vision to analyse global health RDTs. To mitigate potential errors in photography resulting from an uncontrolled field setting, the study incorporated: (1) an automated checkpoint where the algorithm attempts to automatically determine if a photograph is satisfactory for processing; and, (2) a manual checkpoint where a user can determine if the photograph should be retaken. In addition, it was anticipated that adjustments to the image-processing algorithms may improve field performance (Additional file 1: Fig. S2).

As one of the leading medical research sites in Southern Zambia, and an active participant in the Zambian malaria elimination effort, MRT has a well-developed system for collection and verification of malaria RDT results. Data collection is performed using a network of clinic-based healthcare workers who summarize recorded data from clinic record legers at the end of each week. Every Monday, one healthcare provider from each clinic aggregates the data from the past 7 days and enters it into an SMS message. This SMS is then sent to a member of the MRT data aggregation team who collates the data from the satellite sites into a single spreadsheet, which is then manually entered into a REDCap project.

Verification of this SMS data is performed quarterly. To verify the data, the verification team travels to each of the satellite clinics and compares reported data to the physical clinic ledger. Every physical entry describing either positive and negative RDT result is counted and the verification team notes the date and total for each week. All changes to the data are recorded on a physical copy of the SMS aggregate spreadsheet, and changes are recorded digitally upon return to the laboratory. If the tallies are divergent, the verification team resolves the disparity through a record re-count at the satellite clinic. The final corrected tally is noted in red ink on a physical copy of the collated SMS data. The verified and corrected data are then uploaded from Microsoft Excel to REDCap.

Observational analysis of this system was performed by collecting and evaluating the corrected spreadsheets from January 2017-February 2019. The number of corrections was determined by comparing the number of correct (unaltered) data points to the number of corrected (altered) data points for each clinic over time. Date and time data were collected from SMS records between November 2018 and February 2019.

Experiments to determine the limit of detection were performed in triplicate, with the average and standard deviation being calculated at each parasite density. The limit of detection was calculated using 3SD_blank divided by the slope of the regression of the linear region of the data.

The limit of detection for the mHAT app was determined to be comparable to a commercial lateral flow reader (Fig. 3A). For all devices used, signal intensity monotonically increases as the parasite density in the sample increases; at higher concentrations the rate of increase slows and the test line signal intensity begins to plateau. Both test line signal and the ratio of test line and control line signal were considered as threshold metrics. Using the test line signal, the limits of detection were found to be 15.0 ± 3 parasites/μL, 14.9 ± 4 parasites/μL and 6.12 ± 2 parasites/μL using the iPhone 8 + , Samsung Galaxy J3 and ESEQuant LFR, respectively (Fig. 3B). When considering signal ratio (test line signal over control line signal) as the read-out measure, the mHAT app limits of detection increased slightly but not significantly (Fig. 3C). The limits of detection were found to be 23.2 ± 8 parasites/μL and 20.9 ± 6 parasites/μL while the EQEQuant LFR was found to have a limit of detection of 6.15 ± 2 parasites/μL.

For the 14 satellite clinics operated by MRT, surveillance data are aggregated by mobile messaging. MRT is not immune to the variety of challenges that arise with data collection in low-resource settings, and this work sought to quantify the impact of these obstacles. Physical records of malaria RDT data are summarized by community healthcare workers (CHWs) at each clinic and reported by SMS every Monday. To assess the temporal efficacy of this system, SMS records were collected spanning the period between November 2018 and February 2019. Of the 96 weeks of data analysed, only 54.6% of the data packages were received on the Monday expected (Fig. 4A). When data were received late, typically within 1–2 days of when expected, but over 25% of the time the report was received over a week late.

When analysing physical data from January 2017 to February 2019, it was found that most weekly data packets were reported correctly and required no revision by the verification team; however, 36% of the weekly data points did require correction (Fig. 4B). Of the weeks in which data were corrected, most required at least two data points to be corrected, and almost half required three or more points to be corrected. The most common error observed was incorrect entry, where the counts were correct but correct numbers for two categories (e.g. positive RDT, negative RDT) had been exchanged. Approximately 6% of data were not verified at the time of analysis due to external causes, including environmental factors that limited the ability of the verification team to reach clinics, physical damage to clinic records, or lost records.

The most common method for RDT analysis in the field is visual inspection. It has been demonstrated that visual inspection of RDTs varies with CHW experience [37]. The healthcare workers in and around MRT have a high level of familiarity with malaria RDTs, and thus, for this study the mHAT app was compared to visual interpretation of RDTs by these experienced CHWs as a gold standard. All RDTs were identified as positive or negative by a CHW at the point of care, and validated by the researcher using the mHAT app. Iterative improvement of the image analysis software was required over the span of the trial to account for differences between photo quality in a controlled laboratory setting and the field (Additional file 1: Fig. S2).

In this analysis, both the signal at the test line and the ratio of test line and control line signal were evaluated as potential read-out values and ideal thresholds for each signal metric were determined (Fig. 5, Additional file 1: Fig. S3). Both values were assessed on iOS and Android devices.

Test line to control line signal ratio was utilized as the determinant for mHAT test results. While significant batch-to-batch variation between RDTs was not observed, the decision to use signal ratio as a metric rather than test line signal alone offers protection from this issue, as has been documented previously [38]. Sensitivity for signal ratio analysis on iOS devices was found to be 94.6% (CI 81.8, 99.3) and specificity was 88.6% (CI 73.3, 96.8). Using an Android device, sensitivity was 94.6% (CI 81.8, 99.3) and specificity was 95.6% (CI 78.1, 99.9). When both device types were analysed, the combined sensitivity was then observed to be 94.6% (CI 81.1, 99.3) and specificity was 88.6% (CI 73.3, 96.8). Positive predictive value (PPV) and negative predictive value (NPV) for the mHAT app were observed to be 0.923 and 0.952, respectively.