Comparative performance study of three Ebola rapid diagnostic tests in Guinea

All three tests were evaluated using frozen serum samples stored at the Viral Hemorrhagic Fever Laboratory (VHFL) in Conakry, Guinea, which was established in 2014 by the Institut Pasteur [15]. Serum samples had been collected from symptomatic patients during the 2014–15 outbreak, and had been tested at the VHFL using the Trombley RT-PCR assay [16], soon after collection. After this initial characterization as Ebola RNA (+) or (−), leftover sera were stored at − 80 °C in the VHFL.

All tests evaluated in the present study use the Dual Path Platform (DPP; Chembio, Medford, USA) technology, which consists of an immunochromatographic test cartridge and a small, battery-operated reflectance reader. The tests are designed for use with EDTA whole blood (venous or capillary), plasma, or serum. The DPP Fever Panel Antigen System (the “Fever Panel Test”) is a multiplex assay detecting Plasmodium falciparum (HRP2), pan-Plasmodium (pLDH), and protein antigens specific for Lassa, Pan-Ebola (Zaire, Sudan, Bundibugyo), Marburg, Dengue, Chikungunya, and Zika Viruses. The DPP Ebola Antigen System (the “Ebola Test”) tests for Ebola Zaire antigen, and the DPP Ebola-Malaria Antigen duplex system (the “Ebola-Malaria Test”) detects Ebola Zaire antigen, Plasmodium falciparum (HRP2), and pan-Plasmodium (pLDH). All tests are qualitative and detect the VP40 antigen, which is specific to Ebola.

The DPP format uses a sample strip perpendicular to the test strip (in contrast to the classic lateral flow assay format), which delivers sample directly to the test line. Sample (50uL) and buffer are added to a well on the cassette, and migrate to the test site, where Ebola VP40 antigen is captured by the VP40 antibodies on the test line. The buffer dilutes blue and green dye at the test site, thereby indicating that the sample and buffer have migrated properly. Running buffer is then added to the conjugate pad after a 5-min incubation period, solubilizing gold nanoparticles conjugated to mouse VP40 antibodies. The buffer front sends any antigen left on the membrane towards the test line for additional capture, and the gold conjugate binds and accumulates on the test line. Additional buffer follows and washes the membrane for increased contrast and sensitivity.

After 10–15 min, a reflectance reader is used to read the test results. This “Micro Reader” analyzes the reflectance of test and control lines to detect VP40 antigen. It interprets the results using assay-specific cut-off values, and reports a reactive Ebola Virus Disease (EVD+), non-reactive (EVD-), or invalid (INV) result as text scrolling through a liquid crystal display (LCD) on the top of the instrument [17]. The DPP tests and Micro Reader are illustrated in Fig. 1a-c. The DPP Ebola assay was previously evaluated in a collaborative study with The National Institute for Biological Standards Control (NIBSC) using samples derived from recombinant VP40 Ebola antigen [18]. All samples of low, medium, or high levels of VP40 tested positive using the Ebola DPP assay. A separate limit of detection study using gamma irradiated serum samples estimated the limit of detection to be 150 ng/ml, corresponding to 7.5 ng/test [17]. Chembio provided technical training and guidance on operating the tests, but was not involved in the funding or execution of this study. As of September 2019, none of the DPP Ebola devices are yet available commercially.

Ebola-positive and Ebola-negative samples were identified from a database at the VHFL. Samples with sufficient volume of sera were then selected by convenience from the repository based on inclusion criteria. Samples from all patients of any sex or age who were alive at the time of collection in 2014–2015 were eligible. Positive samples were selected by convenience to reflect a range of pathogen loads by molecular testing, with cycle threshold (Ct) values between 15 and 34 (a lower Ct value representing high pathogen RNA, and a higher Ct value indicating low pathogen RNA). No two samples were selected from the same patient.

On-site training to VHFL technicians was provided by the study team, according to manufacturer’s instructions, in January 2017, and included use of the diagnostic tests, the Micro Reader, and data management using a tablet. VHFL laboratory technicians performing the tests were blinded to the original RT-PCR result from 2014/15. All testing was completed between February and December 2017. Results were recorded in a Microsoft Excel data form. Micro Reader results were also exported to Excel. While the manufacturer had no role in the collection or analysis of the data, the data obtained from this study have been used by the manufacturer to support a regulatory submission.

The Trombley assay [16] used at the time of sample collection in 2014/15 was considered the reference standard. Sample use was restricted due to limited volume of sera, so it was not possible to retest all samples with RT-PCR prior to starting the study to confirm integrity, an important limitation to this study. Technicians performing all tests were blinded to sample status.

Sensitivity, Specificity and Accuracy were calculated with 95% confidence intervals for all three assays in reference to the Trombley PCR, including stratification by Ct value (Tables 2 and 3). Positive and negative predictive values were not included in this analysis given the potential biases associated with such a select dataset.

Since the samples had been stored for some time when our study was conducted, a portion were re-tested using a second PCR in order to confirm the integrity of the samples. The Trombley PCR was unavailable, so the RealStar Filovirus Screen RT-PCR Kit 1.0 (altona Diagnostics, Hamburg, Germany), hereinafter “altona” [19] was used instead. Since only convenience sampling was possible for this stage of testing, the results were not considered in formal analyses, but only to shed light on several limitations to data interpretation. These PCR results were further limited by the fact that, in 2017—after investigative rapid testing was completed, but before testing using the altona RT-PCR was completed,—, all samples were transported to the United States, by agreement between VHFL and the US Centers for Disease Control (CDC), and subjected to 5.10^− 6 rads of gamma irradiation before being shipped back to Guinea. This was unanticipated and not related to the study protocol; however, the gamma irradiation and double shipment may have impacted sample integrity, resulting in variability between the Trombley and altona RT-PCR results.

This study was approved as non-human subjects research by Columbia University Institutional Review Board with protocol number AAAR1524(M00Y01), as well as by the National Ethics Committee for Health Research (Comité National D’Ethique Pour La Recherche en Santé – CNERS) of Guinea. All samples were anonymized prior to analysis and no identifying information was used for this study.

Two hundred five samples—109 positive and 96 negative samples—were included in the original dataset, of which 55 (50%) had Ct values between 15 and 25, 29 (27%) had Ct values between 26 and 30, and 25 (23%) had Ct values between 31 and 34 (mean: 25.1; range: 16.5–33.8). The study population was 55% (110/205) male, with a mean age of 39 years, and an age range from 3 to 90 years. Samples were selected by convenience based on inclusion criteria and volume of sera.

Each of the 205 samples was tested using the DPP Ebola, Ebola-Malaria, and the Fever Panel assays, and the result of each was compared to the original PCR result from the time of sample collection in 2014/15. The results are presented for each assay in Tables 1a-c, in the form of 2 × 2 tables.

When comparing the rapid test result to the original PCR, 20 samples (9.76%) were discordant when using the Fever Panel test, 33 samples (16.1%) were discordant when using the Ebola test, and 28 samples (13.7%) were discordant when using the Ebola-Malaria test. This resulted in the highest sensitivity being 89.91% (95% CI 82.3–94.6%) when using the Fever Panel test, and the lowest being 77.06% sensitivity with the Ebola test. Specificity was highest with the Ebola-Malaria test (95.83%), and lowest with the Fever Panel test (90.63%), see Table 2.

While representing a unique evaluation of Ebola RDTs, we note several limitations to this study. Overall, the samples had unclear storage history, which was the reason behind using two different RT-PCR reference standards, selected based on availability in 2014 and 2017 [16, 19]. The quality of the samples available in 2017 was uncertain, and samples may have been exposed to freeze-thaw cycles prior to the study, which could have resulted in antigen degradation. This would mean that a positive sample in 2014/15 could have been negative on repeat testing in 2017. However, varying sensitivities of the two RT-PCR methods themselves make these results difficult to interpret, and since the altona PCR could only be applied to a subset of samples, these test results were not considered when calculating measures of accuracy. In addition, all Ebola study samples were sent to the United States for gamma irradiation before being returned to Guinea. The double shipment and gamma irradiation of the samples took place before confirmatory RT-PCR testing, but after testing with the DPP RDTs as part of this research study, and may have additionally contributed to sample degradation. While gamma rays are often used as a safe decontamination method in external quality assurance studies [26], it has also been shown to impact the sensitivity of RT-PCR, possibly by causing nicks in the viral genome [27, 28], and therefore could result in false-negative results from altona. The extent of this impact is unknown.

While multiple factors resulted in important limitations to the interpretation of these results, the hurdles (both expected and unexpected) faced during study execution would likely apply to many, if not most other attempts to evaluate an Ebola rapid diagnostic using banked samples. The results of the altona PCR analysis indicated that stored samples had likely degraded over time, where originally low positives became negative, or that the irradiation had negatively impacted sample quality. Scarce data on the impact of gamma irradiation on viral RNA means that a PCR may be negative, while an antigen-based test could still be positive after irradiation. Given the unknown impact of storage conditions and irradiation on these samples, it is unlikely that another study using these samples could result in more a definitive evaluation of a new diagnostic. The recent withdrawal of the ReEBOV RDT from the market underscores this challenge, and the dearth of Ebola RDTs on the market is perhaps a reflection of the dilemmas associated with safe and high-quality field evaluation for Ebola tests. It is clear that in order to support development and evaluation of new, high quality diagnostics, careful sample processing, characterization, and storage should be prioritized for research purposes whenever possible.