Comparison of SARS-CoV-2 indirect and direct RT-qPCR detection methods

Archived nasopharyngeal swab samples were obtained from the MSH/UHN clinical diagnostics lab with approvals from the Research Ethics Boards (REB #20-0078-E) of Mount Sinai Hospital in Toronto, Canada. Samples were stored at − 80 °C, and had undergone ≥ 2 freeze–thaw cycles at the time of our analysis, with the exception of the COVID-negative sample, L013F, which was a fresh sample from the same patient as L013. Original clinical diagnostic data was obtained using the Seegene STARMag RNA extraction kit (Microlab STAR Liquid Handling System, Hamilton Company) and Allplex 2019-nCoV RT-qPCR assay analyzed using the Bio-Rad CFX96 IVD real-time qPCR detection system.

Qiagen RNeasy, Invitrogen Purelink, Norgen Biotek Total RNA Purification Kit and the BGI Magnetic Bead Viral RNA/DNA extraction kit were used as per manufacturer’s protocols. For each extraction, 100 µl of sample was used and eluted in 32 µl.

The 2019-nCoV TaqMan RT-PCR Kit from Norgen Biotek and 2019-nCoV: Real-Time Fluorescent RT-PCR kit from BGI were used essentially as per manufacturer’s instructions. Ct value cut-offs used to determine positive versus negative samples were as per the manufacturer’s protocol (Table 1). For comparison of different plate formats (Additional file 1: Fig. S1a), 10 and 20 µl reaction volumes were used with either 2.5 or 5 µl synthetic RNA standard (Twist Biosci.), respectively, using the Norgen system. These were analyzed in parallel on separate BioRad CFX96 (20 μl reactions) or CFX384 (10 μl reactions) real-time PCR systems. All other experiments used 10 µl reaction volumes (384-well plates) with 2.5 µl of sample (synthetic standard, extracted RNA or direct UTM) and were analyzed using a Bio-Rad CFX384 detection system. For testing alternative primers/probes with the Norgen system, primers/probes were purchased from Integrated DNA Technologies (IDT) and primers were used at 500 nM with probes at 250 nM. Probes were FAM-labelled, E Sarbeco and HKU Orf1 sequences are published [12, 16], and newly designed N gene primers/probe (N Pearson) sequences are Fwd: CCAGAATGGAGAACGCAGT, Rev: TGAGAGCGGTGAACCAAGA, probe: GCGATCAAAACAACGTCGGCCCC). RT-qPCR cycling protocols were as per manufacturers recommendations, except for the testing of alternative annealing/elongation temperatures (Additional file 1: Fig. S1f) with the Norgen system where the indicated temperatures were used.

Primer pairs were designed using PrimerQuest software, and purchased from IDT. Primers selected for testing had ΔG values for self-dimers and heterodimers greater than − 9.0 kcal/mole. Newly designed primers were specific for SARS-CoV-2 with no cross-reactivity to other coronaviruses based on published sequences (SH N1 Fwd: AATTGCACAATTTGCCCCCA, Rev: ACCTGTGTAGGTCAACCACG; SH S1 Fwd: TCAGACAAATCGCTCCAGGG, Rev: TCCAAGCTATAACGCAGCCT). The published S gene primers used in this study were S1 Fwd: CCTACTAAATTAAATGATCTCTGCTTTACT, Rev: CAAGCTATAACGCAGCCTGTA [26]. Primers were used at 400 nM. RT-qPCR was performed on a LightCycler 480 (Roche) with a 384 well plate using the NEB Luna Universal One-Step RT-qPCR kit (NEB #E3005L, New England Biolabs Inc) and a reaction volume of 10 μl with 2.5 μl of sample. Cycling conditions were as follows: 55 °C for 10 min (RT), 95 °C for 1 min (denaturation), 45 cycles: 95 °C for 10 s, 60 °C for 30 s (amplification), melt curve. Standard curves were generated for each primer set with serial dilutions of viral RNA from 0.8 to 800,000 copies/μl; SARS-CoV-2 RNA (strain USA_WA1/2020) was provided by the World Reference Centre for Emerging Viruses and Arboviruses (Galveston, TX) (WRCEVA).

For direct detection, 2.5 μl of patient sample in UTM (Copan) were added to the RT-qPCR reaction mix. For comparison to extracted RNA, an equivalent input of extracted RNA was used (i.e. extracted RNA eluted in 32 μl was diluted 1:2 with RNase-free water). To optimize direct detection, RNaseOUT (ThermoFisher) was added to UTM samples (2 U/μl). Samples were then left untreated, heated at 95 °C for 15 min, mixed 1:1 with Lucigen QuickExtract DNA extraction solution with heating at 95 °C for 5 min or treated with MyPOLS Bio VolcanoCell2G lysis buffer, 1% Triton X-100, 1% Tween-20 or 1% Saponin and incubated on ice 15 min. Samples were then directed added to the RT-qPCR reaction mixture and compared to UTM samples that had been left untreated.

Receiver Operator Characteristic (ROC) curve analysis was performed using MedCalc software according to methodology by DeLong et al. [27]. Liddell’s test and confidence intervals for sensitivity and specificity calculations were determined using StatsDirect v3 software. Paired, 2-tailed t-tests used to compare Ct values were calculated using GraphPad Prism software.

Many diagnostic protocols utilize 20 µl reactions in 96-well plates. Using the Norgen RT-qPCR COVID-19 detection kit (which utilizes CDC-approved N1 and N2 primers), we observed similar Ct values in a comparison of 20 versus 10 µl reactions in 96- or 384-well plates, respectively (Additional file 1: Fig. S1a). Thus, to reduce cost and increase throughput we focused on 384-well plates.

We assessed four extraction methods. First, we compared the widely used Qiagen RNeasy RNA extraction kit to another column-based kit from Norgen Biotek. None of the SARS-CoV-2-negative samples generated any signal, and we detected no significant difference in Ct values across four clinically-diagnosed positive patient samples (Fig. 1a and Additional file 1: S1b), suggesting similar extraction efficiencies of these two systems. We next compared efficiency of the Norgen (column-based), Invitrogen Purelink (column-based) and BGI (magnetic bead-based) RNA isolation systems. We tested each of these three extraction methods with three recently developed Taqman detection systems, including Norgen N1 and N2 primers, plus BGI primers targeting the Orf1ab gene. Ct values for two new positive patient samples were similar with both the Norgen and BGI extraction systems, but higher with the Invitrogen kit, independent of detection method (Fig. 1b). For all three extraction methods, the BGI RT-qPCR system demonstrated Ct values ~ 1–3 cycles better than either of the Norgen primers (Fig. 1b), and similar results were obtained with seven additional samples all isolated using the Norgen RNA extraction kit (Fig. 1c). Pairwise analysis confirmed a statistically significant improvement with the BGI primers (Additional file 1: Fig. S1c). Original clinical Ct values were available for 8/9 of these samples, which were obtained using the Seegene STARMag RNA extraction kit and Allplex 2019-nCoV RT-qPCR assay targeting the SARS-CoV-2N and RdRp genes. We observed a strong correlation between the clinical data and Ct values obtained using either the BGI or Norgen RT-qPCR detection modules (Fig. 1d and Additional file 1: Fig. S1d). Notably, there was no significant difference between the BGI and the clinical RdRP or N gene Ct values (Additional file 1: Fig. S1d). While the Ct values obtained with the N1 and N2 Norgen primers were not significantly different from the clinical RdRP Ct values, both were significantly higher than the clinical N gene values (Additional file 1: Fig S1d). In summary, these initial analyses suggested that the BGI system exhibits better performance than the Norgen system.

To validate these data, we utilized a larger patient cohort of 59 clinically-diagnosed samples (29 positive and 30 negative). Both the Qiagen RNeasy and BGI extraction methods (using BGI RT-qPCR detection) demonstrated 100% specificity on this larger cohort, and while sensitivity was slightly higher with the BGI extraction system (93.1% vs. 82.7%, Additional file 1: Fig. S1e), this difference was not significant. We then compared the BGI and Norgen RT-qPCR detection modules using the BGI-extracted samples. While both provided 100% specificity, the 93.1% sensitivity with BGI detection system outperformed the 69.0% or 75.9% sensitivities with Norgen N1 or N2 primers, respectively, although only the difference between BGI and Norgen N1 primers was significant (Additional file 1: Fig. S1e). It is important to note that for a positive diagnosis, the Norgen system requires detection with both the N1 and N2 primers/probes, so sensitivity will be dictated by the primers with poorer performance (in this case for the N1 gene). The human control gene was detected in all samples tested (not shown). Similar to these findings, area under the curve (AUC) data from Receiver Operator Characteristic (ROC) curve analyses confirmed that the BGI RT-qPCR system significantly outperformed the Norgen N1 and N2 primers (p < 0.05 for both, Fig. 1e). As with the pilot cohort (Fig. 1d), Ct values with the BGI, or Norgen N1 and N2 primers correlated strongly to those obtained for RdRp and N gene at original clinical diagnosis (Fig. 1f). As before (Fig. 1d), BGI and clinical Ct values were comparable, whereas Norgen Ct values were higher (Fig. 1f). The Norgen detection system performed well on samples with clinical values of Ct < 34 (20/21 positives detected, with 1/21 inconclusive as only N2 primers were positive), but not on those with Ct > 34 (0/8 positives detected with 1/8 inconclusive in which only N2 primers were positive) (Fig. 1f). In contrast, the BGI system performed well across the entire range of clinical Ct values, and of the 29 clinical positives tested, the two “false negatives” were actually marginal/ambiguous clinical positives with very high Ct values (> 38) for the N gene and negative for both the RdRp gene (Fig. 1f) and the pan-Sarbecovirus E gene (not shown). Whether this result was affected by RNA degradation due to freeze–thaw of the samples is unknown, but remains a possibility.

Next, we compared the limit of detection (LOD) of the BGI and Norgen RT-qPCR systems. For this, we ran 20 replicates each with various concentrations of synthetic SARS-CoV-2 standards (TWIST Bioscience), with LOD defined as the concentration exhibiting ≥ 95% (19/20) sensitivity. The LOD with BGI primers was 2.5 copies/μl, compared to 10 copies/μl with the Norgen primers (Fig. 1g). To determine if the latter could be enhanced, we tested different annealing/elongation temperatures in the qPCR reaction along with two other published SARS-CoV-2 primers/probes shown to have high sensitivity (E Sarbeco and HKU Orf1) [12, 16, 28, 29], and new primers/probes we designed to target the viral N gene. The recommended annealing/elongation temperature for the Norgen system is 55 °C whereas the BGI system utilizes 60 °C, but increasing the temperature did not affect Ct values for either the N1 or N2 primers (Additional file 1: Fig. S1f). Using the Norgen RT-qPCR mix, we observed poor performance of the HKU Orf1 primer set, and the newly designed N gene primers provided higher Ct values compared to the CDC N1 and N2 primers, but the E gene primers/probes demonstrated lower Ct values compared to the N1/N2 primers, particularly at an annealing/elongation temperature of 59 °C (Additional file 1: Fig. S1f). This improvement, however, did not translate to a better LOD (Fig. 1g). Thus, while both BGI and Norgen detection systems reliably detect purified SARS-CoV-2 RNA from patients with clinical values Ct < 34, the BGI detection system provides a lower LOD and higher sensitivity.

Next, we compared the BGI detection system to a SYBR green-based method. We tested various published primers, some designed for SYBR green and some from TaqMan assays [12, 16, 26, 30], and designed our own. One published set for the viral S gene [26] and two new N or S gene primer sets gave little/no signal in no-template control (NTC) samples and generated a linear response across 8—800,000 viral copies/μl (unpublished observation). These were thus selected for future analyses. We then compared SARS-CoV-2 standards using the SYBR green primers and the BGI detection kit and observed comparable Ct values between the two systems across 20 to 20,000 genome copies/μl (Fig. 2a). Identical Ct values were obtained using SARS-CoV-2 RNA from WRCEVA (unpublished observation). The BGI system provided a slightly better LOD than the SYBR green systems (compare Fig. 1g and Additional file 1: S2a).

We next analyzed a pilot cohort of 7 positive and 2 negative patient samples comparing the SYBR green primers to previous data obtained with the BGI kit (Fig. 2b and Additional file 1: Fig. S2b). One of the primers (SH S1) did not perform well on patient samples and was excluded from these experiments. The other SYBR green primers reliably identified all 7 positive patient samples, with SH-N1 primers generating slightly lower Ct values (0.3 to 1.1 Ct values, p = 0.02) and S1 primers providing slightly higher Ct values compared to the BGI system (− 0.2 to 1.6 Ct values, p < 0.01). Quantification of viral gene copy numbers generated similar results for SYBR green and BGI, and ranged from 24 copies to > 120,000 copies/µl (Additional file 1: Fig. S2c). Melt curve analysis revealed non-specific amplification in the small number of negative and low virus copy positive samples in this cohort (Additional file 1: Fig. S2d). Therefore, we analyzed 2 additional higher-level positive patient samples (P7 and P35), 2 low-level positive samples (P6 and P33) and, to rigorously assess specificity, 30 negative clinical samples. While higher-level positives were easily identified, we observed amplification in all 30 clinical negative samples, which gave similar Ct values as low-level positives (Fig. 2c). Melt curve analysis demonstrated that high-level positive samples (P7, P35) gave specific melt peaks comparable to synthetic SARS-CoV-2 standards, while low-level samples (P6, P33) gave multiple melt peaks, one of which overlapped with standards (Fig. 2d,e). Among the 30 negatives, the melting temperatures of 24 were distinct from the synthetic SARS-CoV-2 standards, but 6 matched those of positive samples (Fig. 2d,e), indicating a specificity of only 80%. Thus, with the primers and conditions tested, SYBR green detection generated several false-positives, even with melt curve analysis.

To reduce the number of steps required for viral detection, we tested direct, extraction-free RT-qPCR on patient samples in UTM. For this, we added 2.5 μl of sample directly to the RT-qPCR mix and compared this to an equivalent input of extracted RNA. UTM blocked SYBR-green detection of SARS-CoV-2 RNA standards (unpublished observation), but both the BGI and Norgen TaqMan detection systems identified positive patient samples (Fig. 3a). Ct values were lower for BGI vs Norgen, consistent with data from purified RNA (c.f. Fig. 1c and Additional file 1: S1c to Fig. 3a). Furthermore, the Norgen system did not reliably identify some positive samples with lower levels of virus (Fig. 3a). Relative to extracted RNA, virus detection by direct RT-qPCR with the BGI detection kit was 2–26 fold lower as assessed by quantification of viral copy number (except sample L021, which was ~ 600-fold reduced, see below for an explanation), whereas with the Norgen kit it was 20–1000 fold lower with direct detection (L033 with the N2 primers was an exception at 4.4-fold). Despite the higher Ct values, there was a strong correlation between BGI and original clinical Ct values (Fig. 3b).

Others reported that heat or different lysis buffers/detergents may improve direct detection [19‐21, 25]. Thus, using a pilot series of four patient samples, we assessed the effect of heating at 95 °C for 15 min, five different lysis buffers/detergents (Lucigen QuickExtract DNA extraction solution, MyPOLS Bio VolcanoCell2G lysis buffer, 1% Triton X-100, 1% Tween-20 or 1% Saponin), and treatment with the RNase inhibitor, RNaseOUT. Notably, simply adding the RNase inhibitor was sufficient to dramatically increase virus detection > 100 fold (as assessed by copy number) using the Norgen system, and generated Ct values comparable to those obtained with the BGI RT-qPCR system. Most importantly, this permitted detection of previous “false-negative” samples L021 and L032 (Fig. 3c). Furthermore, addition of the RNase inhibitor brought direct RT-qPCR results with the Norgen detection kit to within 3 Ct values (~ 10 fold) of those obtained with extracted RNA (compare Fig. 3a, c). Treatment with heat, lysis buffers or detergents did not appreciably increase virus detection, and in some cases reduced virus detection (higher Ct values). For the BGI detection system, none of the treatments dramatically improved detection, with the exception of sample L021 (Fig. 3c), which previously showed the largest difference between extracted RNA and direct UTM analysis (Fig. 3a). We presume, therefore, that L021 had higher RNase levels that were not fully inhibited by the (proprietary) RNase inhibitor already present in the BGI mix. Thus, addition of an RNase inhibitor is sufficient to improve direct detection, and under these conditions BGI and Norgen kits perform similarly.

Following these pilot assays, we assessed direct detection on 60 patient samples, including 30 clinical positives and 30 negatives, focusing on the more sensitive BGI detection system. We observed a strong correlation with clinical Ct values, and, with the exception of two invalid samples in the direct RT-qPCR (no human actin detected), accurately identified all samples with original clinical Ct values < 33 (Fig. 3d). Akin to our observation with most samples in the pilot study, adding RNaseOUT did not further improve direct detection (unpublished observation). Combining the 31 negatives and 37 positives from the pilot and expanded datasets (Fig. 3a, d), the BGI direct detection strategy generated an AUC of 0.892 in ROC curve analysis, and this test exhibited a sensitivity of 78.4% and specificity of 100% (Fig. 3e).