Evaluation of serological lateral flow assays for severe acute respiratory syndrome coronavirus-2

We procured 56 SARS-CoV-2-positive, 46 pre-pandemic SARS-CoV-2-negative, and 10 SARS-CoV-2-negative HIV-positive EDTA plasma samples. SARS-CoV-2 positive EDTA plasma samples were obtained from clinical discards banked within 24–72 h of collection at the Crimson Core of the MGB Biobank, which was composed of hospitalized symptomatic patients. All samples had positive SARS-CoV-2 PCR results using an EUA approved test at Brigham and Women’s Hospital (Panther Fusion SARS-CoV-2 assay, Hologic, Inc., San Diego, CA or Xpert Xpress SARS-CoV-2, Cepheid, Inc., Sunnyvale, CA) or the Clinical Research Sequencing Platform at the Broad Institute of MIT and Harvard (in house Laboratory Developed Test) 10–40 days prior to sample collection. The participants’ charts were reviewed by study staff to identify samples collected ≥10 days after onset of symptoms and to exclude immunosuppressed participants, after which samples were anonymized and stripped of protected health information. Pre-pandemic negative control samples were randomly selected from healthy participants with a Charlson Age-Comorbidity Index [28] score ≤ 2 and EDTA plasma banked in the MGB Biobank between Jan 1-Dec 1, 2019 from inpatients. HIV-positive control samples were obtained from EDTA plasma samples banked prior to January 2020 in a study on neuropathic pain in HIV. All HIV-positive participants were on antiretroviral therapy. For 8 out of the 10 HIV-positive samples, viral load quantification was available and showed 256 copies/ml or less, and 5 showed loads either undetectable or under 20 copies/ml. All samples were collected from consented individuals. Sample and data collection conformed to Good Clinical Practice guidelines and Declaration of Helsinki. The study was approved under the Massachusetts General Brigham (MGB) Institutional Review Board (protocol no. 2020P001204).

Twenty commercial IgG/IgM lateral flow assays (LFAs) from 18 manufacturers were evaluated (Supplementary Table 1). LFAs were analyzed by blinded operators according to manufacturer instructions for use (IFU), with the exception of using micropipettes instead of manufacturer-provided droppers to minimize technical variability. Samples were thawed on ice, randomized, and brought to room temperature. Kit components were also brought to room temperature. The IFU-specified volumes of sample and buffer were added to the cassette. Specified sample volumes varied for different LFAs but were typically in the 5-20 μL range. The cassettes were run at room temperature on a flat surface and results read immediately after the time interval defined in the IFU (typically ranging from 10 to 15 min). Each cassette was independently scored by two blinded raters as either “positive,” “negative,” or “invalid”. Ratings were designated according to the interpretation guidelines outlined in each individual IFU. Each cassette was photographed under four standardized illumination conditions and viewpoints for future analysis.

For inter-operator reproducibility analysis, separate pools of EDTA plasma were obtained from > 30 pre-pandemic healthy individuals (negative pool) and > 30 convalescent participants collected after symptom resolution at the Massachusetts General Hospital (MGH) respiratory illness clinic (positive pool). Convalescent samples for the positive pool were confirmed to be positive using the COBAS SARS-CoV-2 PCR test (Roche Diagnostics, Indianapolis, IN) at MGH. A total of 20 replicates per pool were run by two independent pairs of blinded operators, alternating between positive and negative pools (10 replicates per pair). Reproducibility was calculated according to agreement between operator ratings as well as concordance of readout with sample pool COVID status.

Our cohort of 112 EDTA plasma samples was used across all 20 LFAs to evaluate performance: sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Samples were sub-aliquoted throughout the analysis to minimize freeze-thaw cycles. Binary presence/absence classifications were used, and discordant calls were resolved by a third operator inspecting photographs taken of the relevant LFA.

In addition to initial screening [27], each LFA kit was assessed for consumer usability based on complexity of kit materials, sample requirements, and IFU clarity. Supplied kit components were documented for completeness and examined for ease-of-use. IFU protocols were rated on a scale from 0 to 14 according to a predefined rubric (Supplementary Table 2) by three independent raters. Usability evaluations are shown in Supplementary Table 3. Sample input requirements for each LFA are in Supplementary Table 1.

Plasma samples were diluted 4000-fold, and the total IgG and IgM levels against the SARS-CoV-2 spike protein were measured using a custom Single Molecule Array (Simoa) assay as described [29] on an automated HD-X Analyzer (Quanterix, Billerica, MA, USA), to provide a quantitative reference for anti-spike antibody titers in the plasma samples. Normalized mean Average Enzymes per Bead (AEB) levels were calculated using a standard set of calibrators produced by serially diluting a large volume of plasma from seroconverted individuals. Antibody concentrations were estimated using a calibration curve of recombinant anti-SARS-CoV-2 antibodies [30].

We obtained plasma samples from 56 pre-pandemic patients, including 10 HIV+, and 56 symptomatic inpatients in March and April 2020. HIV-negative samples were matched for sex and age. The overall study population included 25.9% Blacks, 4.5% Latinx, 9.8% Asians, and 43.8% Non-Hispanic Whites (Table 1). COVID+ samples were from individuals between 10 and 40 days post symptom onset, with 60.7% samples taken between 2 and 4 weeks. Of all the COVID+ participants, 7 (12.5%) were symptomatic outpatients and 49 (87.5%) were hospitalized. Among those hospitalized, 25 (51%) required intensive care unit (ICU) treatment and 4 (8.2%) were deceased at the time of chart review (Supplementary Table 4). Additional mortalities were possible after chart review since some participants were in critical condition in the ICU.

Four independent operators working in teams of two on separate days applied each pool 10 times to each LFA, with processing as dictated by the instructions for use (IFU). API version1 LFAs were not assessed for reproducibility due to limited cassette availability and the high volume required. Of the remaining 19 LFAs, three (BTNX, Camtech, and Carehealth) had 100% consistent correct outcomes across both isotypes, with an additional three (BioHit, Zhuhai Livzon, and Phamatech) having no incorrect or inconsistent outcomes with one or two invalid tests (Fig. 1). IgG was the more reproducible isotype. The majority of incorrect consistent calls came from operators calling a COVID+ sample IgM negative (Fig. 1).

We assessed the LFAs according to this rubric (Supplementary Table 2) by three independent raters and assigned a composite score on a scale of 0–14 (Supplementary Table 3). Five LFAs (BioHit, InTec, Lumiquick, Phamatech, and U2U) received full scores for IFU clarity. LFAs frequently lost points for imprecise instructions regarding time between adding sample and having sufficient sample flow to read the results. Important kit usability criteria, such as whether the included pipette droppers show clear volume markings, were also recorded (Supplementary Table 3). The IFU ambiguity led to administration of too much or little sample for a valid test. While these IFUs may not yet be intended for the general public, it will be important to clarify the instructions moving forward and include droppers to minimize potential for sample volume errors.

To focus on LFA specificity, given the likely use of these tests in low-prevalence settings, disagreement between operators was interpreted as a negative call. Across all but three of the LFAs (Biohit, BTNX, and Vivacheck), sensitivity was higher for IgG than IgM. Sensitivity for IgG ranged from 98.2% (Lumiquick) down to 72.7% (Oranoxis), and for IgM from 96.4% (BioHit) to 23.2% (Oxo and U2U) (Table 2). LFA specificity was much higher for both isotypes, with 11 LFAs having a specificity of 100% for IgG (API, API v2, BTNX, Camtech, Genobio, Oranoxis, Phamatech, Ray Biotech, Ray Biotech v2, U2U, and Zhuhai Livzon), and 8 LFAs having a specificity of 100% for IgM (API, CareHealth, Cellex, Lumiquick, Oranoxis, Ray Biotech v2, U2U, and Zhuhai Livzon) (Table 2). Under the assumption that the likelihood of two randomly occurring false positives for any one individual is low, an IgG/IgM composite score (averaged operator scores, see Supplementary Methods) was produced to maximize test specificity. Using this composite score, all LFAs except BioHit, Cellex, Edinburgh, InTec, and Vivachek achieved a specificity of 100%. This result underscores the potential of considering the outcome in both isotypes to minimize false positives, although it is more likely that a single isotype will be used in clinical testing.

We created heatmaps to visualize individual sample outcomes across all LFAs to assess whether we systematically detected the same miscalls across multiple LFAs (Fig. 2). False negatives (blue squares in the COVID+ panel) amongst COVID+ patients were somewhat reproducible, with three COVID+ samples called negative in both isotypes by all, or all but one LFA. These miscalls were not clearly explained by known demographics (age, sex) or clinical variables (disease severity, weeks post symptom onset) (Fig. 2, bottom panel). To investigate whether these miscalls were related to low titers of anti-SARS-CoV-2 antibodies from participants with a suppressed immune response, all 112 samples were analyzed for anti-spike IgG and IgM antibodies using a custom quantitative Simoa assay [29]. The three samples that were called negative across almost all LFAs, which were also 2–3 weeks post-symptoms, had the lowest levels of anti-spike antibodies in COVID+ samples for both IgG and IgM, suggesting these participants had a slower or suppressed immune response to SARS-CoV-2 infection.

Unlike false negatives, most false positives (Fig. 2, red squares in COVID- panel) amongst COVID- individuals appear largely uncorrelated between LFAs. However, two samples showed IgG false positives across multiple LFAs, which may suggest long lasting antibodies from exposure to coronaviruses other than SARS-CoV-2. Nonetheless, this observation is not reflected in the antibody levels from the Simoa analysis, which showed barely detectable anti-spike antibodies in these two samples.

The custom Simoa anti-spike IgG and IgM antibody assays use a standard curve to determine standardized antibody concentration in each sample [29]. The results obtained from this assay can therefore be used to estimate a limit of detection (LOD) for each of the qualitative LFAs. The cumulative number of false negative LFA calls in COVID+ samples (Fig. 3, y-axis) were computed as a function of decreasing antibody concentrations (Fig. 3, x-axis) separately for IgG and IgM. We define the LOD for each LFA/antibody as the concentration at which ≥95% of the COVID+ samples are called unambiguously positive (Supplementary Table 5). Using this definition, for IgG all LFAs (except Genobio, Oranoxis, OZO, Ray Biotech, and U2U) have an LOD within the linear range of the SIMOA assay (1–10,000 μg/mL). Lumiquick has the lowest LOD at 0.1 μg/mL, which was extrapolated by dilution to be within the linear range of the SIMOA standards, and CareHealth, Cellex, KHB, and Vivachek all have an LOD of 1.5 μg/mL (Supplementary Table 5). Considering the generally lower sensitivity observed with the IgM assays (Fig. 2), IgM assays consistently have higher LODs, with 9 exceeding 1000μg/mL. BioHit has the lowest IgM LOD at 0.6 μg/mL, and API v2, BTNX, CareHealth, and Vivachek all have LODs under 10 μg/mL.

Low prevalence places a high burden on specificity [25]. Given the high proportion of true negative individuals in the population being studied, prevalence increases the ratio of false positive to true positive test outcomes [31]. Positive predictive values (PPV) correspond to the likelihood that a positive test result reflect true positivity as measured by a gold standard PCR result. We computed PPVs as a function of the fraction of the population infected with (and assumed to have produced antibodies to) SARS-CoV-2 (Fig. 4). Here we see that even with the conservative interpretation of LFA outcome (requiring the majority of operators to see a band to call a sample positive for either IgG or IgM), when the population seroprevalence is ~ 5% the PPV for these assays spans a large range (from ~ 30 to 100%). As the population prevalence increases, the burden on specificity is decreased, and at 50% prevalence the PPV of all LFAs is above 87.5%. Posterior PPV can be improved for most LFAs by requiring both IgG and IgM to be read as positive in order to count an individual as positive (Fig. 4, right panels).

To visualize the effect of changing population prevalence on the PPV, we created an interactive webapp (https://​covid.​omics.​kitchen; Fig. 5), which allows the user to extrapolate the likelihood that they do in fact have SARS-CoV-2 antibodies if they have a positive LFA result, given the infection prevalence and test accuracy. The app includes benchmark performance of all 20 LFAs evaluated here, as well as those reported in Whitman et al. [10] (filtered to remove samples taken under 10 days post symptom-onset). In an effort to further generalize the utility of this tool, we allow the user to explore the effects of assay performance under difference prevalence scenarios, and to input the reported prevalence data from specific geographic locations within the US [1], based on national, state, and county records.