Comparison of the performance of two real-time fluorescent quantitative PCR kits for the detection of SARS-CoV-2 nucleic acid: a study based on large real clinical samples

Since December 2019, coronavirus disease 2019 (COVID-19) which is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, has spread rapidly around the world and become a pandemic that has posed a huge threat to human health [1]. As of May 3rd, 2022, more than 511 million people worldwide have been infected with COVID-19 and more than 6.23 million have died from this disease [2]. Thus, a rapid and accurate diagnosis is highly demanded to efficiently control the spread of COVID-19. Currently, analytical methods in the laboratory for COVID-19 pathogen diagnosis include nucleic acid testing (NAT), antigen testing, and serological testing [3]. Among them, several studies have shown that antigen and serological testing have limitations such as low sensitivity and lack of commercial reagents. In contrast, NAT is the preferred method for diagnosing COVID-19 [4]. NAT approach is mainly performed by testing human nasopharyngeal and oropharyngeal swabs to observe the presence of viral nucleic acids in vivo using reverse transcription-polymerase chain reaction (RT-PCR), reverse transcription loop-mediated isothermal amplification, and genome sequencing [5]. The National Health Commission of the People’s Republic of China has updated the COVID-19 treatment protocol several times, and RT-PCR detection of viral nucleic acid has been always used as a gold-standard indicator for diagnosis and treatment monitoring [6]. The NAT method has good sensitivity and specificity, and it requires specialized personnel, analytical instruments, and kits [7].

As the number of people infected with COVID-19 is surging worldwide, manufacturers in various countries are rapidly developing and producing kits due to the huge demand for NAT [8]. Under conventional procedures, the approval of newly developed kits is subject to a series of rigorous clinical validation and evaluation. Due to the severe situation of COVID-19 caused by the rapid transmission of the virus, the Food and Drug Administration approved the kits in an emergency. However, the kits which may not have undergone sufficient performance validation and evaluation can increase risk during clinical testing [9]. Currently, the approved NAT kits are mainly targeting three conserved gene sequences in the SARS-CoV-2 viral genome, including open reading frame 1ab (ORF1ab), nucleocapsid protein (N), and envelope protein (E) genes [10]. Recently, studies have shown that different brands of NAT kits have varied diagnostic efficacies [11]. Thus, laboratories should pay attention to the evaluation of the kits from different vendors and select appropriate testing protocols so that NAT kits can be better utilized in clinical settings.

The “Guidelines for the implementation of total 2019-nCoV nucleic acid detection (2nd edition)” issued by the National Health Commission of the People’s Republic of China stated that: The laboratory should choose at least two kits, one as the primary screening kit for routine testing, with a minimum detection limit of less than or equal to 500 copies/mL; the other should be more sensitive than the primary screening kit, with a minimum detection limit of 100 to 300 copies/mL, for re-testing confirmation when the primary screening is positive [12]. In view of the above, the Novel Coronavirus 2019-nCoV Nucleic Acid Detection Kits (RT-PCR method) produced by Maccura Biotechnology Co., Ltd (Sichuan, China) and Sansure Biotechnology Co., Ltd (Hunan, China) were selected for this study (abbreviation: Maccura kit, Sansure kit). The Maccura kit is a triple fluorescent PCR NAT kit for ORF1ab, N, and E gene targets. It uses RNA pseudovirus as an exogenous internal standard, and the main components of its enzyme mixture are Taq and UDG enzymes, and it has a minimum detection limit of 450 copies/mL. The Sansure kit targets ORF1ab and N genes. Rnase P is used as the endogenous internal standard, and the main components of its enzyme mixture are RT and Taq enzymes, with a minimum detection limit of 200 copies/mL. Both kits were authorized for clinical use in China and have received emergency use authorizations issued by the Food and Drug Administration [13]. Currently, most performance comparison studies on SARS-CoV-2 NAT kits have used characteristic reference substances and lacked analysis using a large number of clinical samples [14, 15]. To this end, this study analyzed the results of both kits on the same batch of clinical samples to evaluate the performance of the different kits in actual clinical work and to optimize the kits already in use. The purpose of this study was to provide a reference for future NAT and its application in the clinical laboratory.

NAT method exhibits the advantages of promising sensitivity and specificity for viral detection. However, the accuracy of the test also depends on the sample quality, experimental factors, and kit performance [20]. In this study, we compared two commercial kits (Maccura and Sansure) for RT-PCR detection of SARS-CoV-2 nucleic acids. The Ct values obtained from the two kits were highly correlated, with a linear relationship and good agreement. Meanwhile, it was found that the difference between the two Ct-value datasets was also statistically significant, except for the Ct values of the N gene amplification at high viral loads.

Currently, most studies have focused on false-negative results of NAT, since it may affect the outcomes of early diagnosis and treatment [21]. Despite the low number, the false-positive results cannot be neglected [22‐24]. It will mislead the self-isolation of patients, and interfere with the routine of the normal population and the treatment plan of the sick population, leading to an unnecessary burden on the health system [25]. In this study, the Sansure kit was used as a reference standard to evaluate the detecting performance of the Maccura kit. The Sansure kit was selected as the reference standard for two reasons. First, it has higher sensitivity with a minimum detection limit of 200 copies/mL, compared to the Maccura kit. Second, it has been proved as the RT-PCR kit with the best combination of sensitivity and specificity compared to the other kits [26]. In this study, the Maccura kit was found to have false-positive results with low specificity (57.3%). However, the false-positive results for the Maccura kit were not the first to occur. Tembo et al. conducted a diagnostic validation study of multiple SARS-CoV-2 kits. They evaluated the detection of the Maccura kit using the RealStar SARS-CoV-2 RT-PCR kit as the gold standard and showed that there were 5 false-positive samples among the 18 samples diagnosed as SARS-CoV-2 positive by the Maccura kit, with a false-positive rate of 27% [27]. A higher false-positive rate than ours was observed in the above study. The possible reasons for the false-positive results of the Maccura kit in this study were: 1) The lower limit of detection differed slightly between kits. 2) There were differences in the primers and probes used in different kits [28]. 3) The Maccura kit’s more comprehensive assay design allowed it to detect a broader range of SARS-CoV-2 variants. We checked the announcement of SARS-CoV-2 variant detection status released by two companies (Maccura Biotechnology and Sansure Biotechnology) and found that the Maccura kit could detect more SARS-CoV-2 variant strains (B.1.526 Iota, B.1.429/427 Epsilon, P.2 zeta, etc.) [29, 30]. 4) The Maccura kit targets three different genes (ORF1ab, N, and E), whereas the Sansure kit only targets two genes (ORF1ab and N). Compared to the ORF1ab and N genes, the E gene is less specific because it has 93.5% sequence homology with other coronaviruses (Sarbecovirus SARS-CoV) [31]. It has also been shown that E gene mutations could affect the detection efficiency of the kits [32]. 5) This study was conducted in an emergency situation (mass population screening in a high prevalence area) and did not take samples for nucleic acid integrity testing by sequencing or viral culture. It was possible that virus fragments with no infectious capacity were detected, resulting in false-positive results.

In this study, the new cut-off Ct values were used only for the ORF1ab and N genes of the Maccura kit, and the test result was considered positive based on the new criteria (ORF1ab gene: Ct ≤ 35.00; N gene: Ct ≤ 35.07). The validity and the feasibility of the new cut-off values were confirmed by the validation cohort. We did not introduce a cut-off value for the E gene because the ORF1ab and N genes in SARS-CoV-2 are longer with more variant sites [33]; The N gene has a high transcript level which will lead to higher sensitivity in the diagnosis [34]. Besides, the ORF1ab and N genes are recommended as targets of detection in the “Technical guidelines for 2019-nCoV nucleic acid screening with 10-in-1 mix samples” issued by the National Health Commission of the People’s Republic of China [35].

The COVID-19 pandemic has brought a great challenge to many resource-limited areas [36]. Each laboratory should consider its facility, specimen volume, staffing, and financial situation to reasonably procure high-quality results from the test kits. Currently, many studies have compared and analyzed the diagnostic efficacy of different commercially available kits, but most have used characteristic reference substances or the number of cases including COVID-19-positive patients is not sufficient [14, 37, 38]. Yang et al. conducted performance validation of five SARS-CoV-2 NAT kits using characteristic reference substances and found that all kits passed the performance validation except Kinghawk, which failed to detect the detection limit of 500 copies/mL [14]. Lu et al. and Wu et al. evaluated the sensitivity, specificity, positive predictive value, and negative predictive value of multiple SARS-CoV-2 NAT kits using samples from 18 and 26 COVID-19-positive patients, respectively, and their studies provided information on detection protocols in the COVID-19 pandemic [9, 37]. The advantage of this study compared to previous studies was the inclusion of a large real clinical sample (1118 COVID-19-positive patients) to evaluate the practical clinical application of the two kits. On the basis of this, we investigated the difference, correlation, and consistency in the test results between the two kits. We proposed solutions for the improvement of the kits by recalculating the new cut-off Ct values of the Maccura kit and including 163 clinical samples for large-scale validation. Ideally, laboratories should use both NAT kits to fully pursue the accuracy in practical applications. According to the compatibility of the test results with clinical practice, a suitable and cost-effective NAT kit is reasonably needed in the laboratory, or the purchased kits should be improved to minimize false positives or false negatives in practice.

It is worth noting that there are still some limitations in our study. First, the results of NAT are strongly affected by specimen quality, transportation, storage condition, laboratory environment, and operating techniques of testing personnel. It is too difficult to ensure that the testing process is completely consistent for all the samples. Second, the Sansure kit assay used in this study cannot represent the gold standard and may limit the validity of the results. Third, we did not perform additional sequencing and viral culture of false-positive samples for our own reasons (including embarrassment of time, human and material resources) and environmental impacts (emergency screening of large populations in highly endemic areas). Finally, this study has geographical limitations (data collected only in Jilin Province, China) and cannot be scaled up to make more general conclusions.

To conclude, we analyzed the difference in the testing performance between two SARS-CoV-2 NAT kits using a large number of clinical samples. Some false-positive cases were discovered from the results provided by the Maccura kit, and the cut-off Ct values of the kit were adjusted to improve the accuracy. After re-evaluating the results, the new cut-off values were considered to be applicable in actual clinical work, providing a reference for other laboratories to select and optimize the kits.