Diagnostic accuracy and feasibility of a rapid SARS-CoV-2 antigen test in general practice – a prospective multicenter validation and implementation study

This prospective multicenter validation study in general practice was conducted in 40 general practices in Würzburg, a city of nearly 130,000 inhabitants in the federal state of Bavaria, and its surrounding area from December 2020 to June 2021. During this period, the predominant virus variant was the alpha variant. From May 2021, it was largely replaced by the delta variant [22].

Inclusion criteria for patients were based on the PCR test guidelines of the Robert Koch Institute [23], the national public health authority in Germany, which, besides others, is continuously monitoring the COVID-19 situation and estimating the risk for the population. We included adult patients with any acute, recently appeared respiratory symptoms and/or symptoms suggesting a SARS-CoV-2 infection such as smell and/or taste disorders (hypo- or anosmia, hypo- or ageusia) [23]. Participating GPs should check patients for eligibility and include them consecutively in the study after written consent. Patients should complete a clinical questionnaire asking for age, sex, contact with a SARS-CoV-2 positive person, symptoms (see Tab. 1) and days since symptom onset.

The GPs were responsible for maintaining hygiene standards during their consultations. Most GPs performed the point of care test (POCT) as part of a special infectious disease consultation. GP took two nasopharyngeal swabs from each patient. The first swab for POCT and the second swab for PCR testing. According to the manufacturer’s protocol of the Roche SARS-CoV-2 Rapid Antigen Test [24], the practice staff read the result of the antigen test after 15–30 minutes. If the test showed a clear positive result earlier than 15 minutes, this positive test result was considered as valid. The second swab for RT-PCR was sent to one of six different cooperating laboratories. All laboratories were certified according to DIN EN ISO 15189:2014 or ISO 9001:2015 [25, 26]. If one of the two tests (POCT or RT-PCR) was positive, an antibody (Ab) serology was offered on a voluntary basis in order to better assess the clinical case by adding an additional test method [27]. The blood sample was taken at the respective GP office earliest 10 days after positive test result. The analysis was conducted at the Institute for Hygiene and Microbiology, Würzburg. All test results and the symptom questionnaire were linked by a study ID.

For the evaluation of the antigen test compared to PCR results, a 95% confidence interval of +/− 5% for sensitivity and specificity was considered adequate. Assuming a prevalence of 9%, a sensitivity of 78% and a specificity of 99%, the required sample size was 2930 participants.

Baseline characteristics of patients were analyzed descriptively and were expressed as percent, median with interquartile range (IQR) or mean with standard deviation (SD). To test for differences between groups, Fischer’s exact two-tailed tests were performed in case of nominal variables and the Mann-Whitney U test in case of metric and categorical variables.

To determine the accuracy of the antigen tests, we calculated specificity, sensitivity, positive and negative predictive value (PPV, NPV), with their 95% confidence intervals (95% CI). The values were calculated using the RT-PCR result as a reference. We broke down the cycle threshold (Ct-) values as a proxy for the viral load into groups and determined the sensitivity for each group. We split the PCR-positive results into two groups. In one group, PCR and POCT results matched (true positive), in the other the POCT falsely showed a negative result (false negative). The difference between the two groups was plotted as a function of the Ct value to determine the specific difference.

GPs and medical assistants (MAs) who were involved in the organization, implementation and execution of the rapid antigen test were interviewed using a standardized questionnaire about the feasibility and acceptance of the procedure at the end of the study. After eliciting some baseline characteristics, GPs and assistants could rate the organizational and logistical effort of the test procedure as well as consequences for medical treatment on six-point Likert scales, for example ranging from very good to very bad or not useful to very useful. In addition, we encouraged participants to provide free text responses in order to achieve a higher data saturation (for the detailed questionnaire, see attachment). For the conduct of the survey we used the survey software EvaSys [28].

Data analyses were performed using IBM SPSS Statistics (Version 26) software [29].

During the first 4 months of the recruitment period (until April 2021), we could enroll between 144 and 615 patients per month. In May and June, it was only possible to recruit 72 and 2 patients, respectively. Therefore, we decided to stop recruitment. The participating 40 general practices recruited 1518 patients, on average, 36.3 (SD 27; Range: 3 to 141) patients per practice (Fig. 1). After exclusion of 68 patients (0.04%), because they did not meet inclusion criteria, the final sample consisted of 1450 patients, 1386 (95.6%) of them partly or fully completed the symptom questionnaire.

Of the included patients, 730 of 1450 (53%) were female; median age was 40 years (IQR 30 to 55). Most patients (1253/1450 ≙ 86%) consulted their GP within the first 7 days after symptom onset (Table 1). On median, the initial consultation and testing took place 2 days (IQR 1.5 to 4) after symptom onset.

Of the patients, 138 of 1450 (9.5%) had a PCR-positive test. Of the patients, 138 of 1450 (9.5%) had a PCR-positive test. Of these, 108 also had a positive antigen test result yielding a sensitivity of 78.3%, 30 patients had negative antigen test results (considered as false negative result). Seven of 1312 patients with PCR negative test results had positive rapid antigen test results (considered false positive), yielding a specificity of 99.5%. In our study population with a SARS-CoV-2 prevalence of 9.5%, the rapid test achieved a PPV of 93.9% and an NPV of 97.8% (Table 2).

We investigated the sensitivity in relation to the Ct-value. The median Ct-value was 23 (IQR 20.3 to 27). The detection rate decreased significantly with increasing (Ct-value i.e. decreasing viral load). For example, the sensitivity for patients with a Ct-value of > 30 was only 25% (n = 12; 95% CI: 7.7–57.2%), whereas for a Ct value of ≤30 the sensitivity was 90.8% (n = 69; 95% CI: 81.4–95.9%) (Fig. 2). We were able show that the Ct values for true positive results were significantly lower than for false negative results (p-value: < 0.005). Figure 3 shows the dispersion of the individual test results (true positive, false negative) over the Ct-value scale.

The opportunity to determine seroconversion was offered to 145 patients who tested positive, 51 (35.2%) accepted the offer.

We defined seven cases as “false positive” (Table 2). Of the seven patients, we performed a serology in three cases. Against expectations, all of them showed a clear positive seroconversion. Independently of this study, we received reports from the GPs that two of them underwent further RT-PCR which led to positive results.

Furthermore, we defined 30 cases as “false negative” (Tab. 2). Of these, we performed serology in eleven cases, only six cases showed seroconversion with positive Ab-titers. Five cases showed a negative Ab-titer.

Moreover, we performed serology in 37 cases defined as “true positive”. Here, 35 cases showed a clearly positive seroconversion.

A total of 40 GPs and 39 MA completed the survey to evaluate the acceptance of the POCT. Almost half of the GPs were female (48.6%), 46% of them were between 45 and 54 years old and 61% worked in a group practice. The assistants were mostly female (92%), 58% of them were between 35 and 54 years old and they were predominantly (62%) active in a group practice. For more detailed information, see Table 1 in the attachment.

Professionally conducted antigen tests do not require manufacturer-independent validation to be approved for the German market [31]. When investigated in independent clinical studies, the sensitivities of the Roche SARS-CoV-2 Rapid Antigen Test in symptomatic patients ranged from less than 63% to well over 85% [15, 16, 32, 33]. The different results can probably be explained by different study characteristics (test site, inclusion criteria, staff training, etc.). Therefore, external manufacturer-independent validation in different clinical settings is needed. This allows diagnostic accuracy and scope of a test result to be interpreted correctly.

Like other studies before, our results show that the test provides reliable results especially at very low Ct-values, i.e. at a high viral load [34]. The viral load is dependent on the days since symptom onset. At the beginning of the infection, the viral load is particularly high and decreases over time [15, 35]. The Ct-value alone is not sufficient to exclude risk of transmission. Infectivity is significantly dependent on SARS-CoV-2 gene copy number as well as genome integrity [36]. However, the Ct-value correlates strongly with the viral load and for this reason, it can be a valuable tool for decision making and risk assessment [37]. Above a Ct-value of > 30, virus cultivation is difficult and infectivity seems unlikely [38]. We showed that the POCT does not always provide reliable results even at a Ct-value of less than 30 (Fig. 3), a range of diagnostic inaccuracy prevails especially at Ct-values between 20 and 30.

Our results showed a sufficiently high specificity of antigen tests in primary care practices. However, even if positive test results can be interpreted with high probability as true positives, they should be confirmed by a RT-PCR. In contrast, the sensitivity of the test was only moderate and strongly depended on the viral load. Rapid antigen tests cannot provide information about the viral load, and thresholds to rule out contagiousness are difficult to define. Furthermore, the test cannot distinguish between a beginning and a declining infection. If a patient is at the beginning of the infection, the viral load may have increased after a short time and infection of other people cannot be ruled out. A negative test result cannot exclude a transmission-relevant infection with certainty and should not lead to a false sense of security [39].

The high mutation potential of the coronavirus is also a recurring topic of discussion. The mutations mostly affect the S-protein. Rapid tests, on the other hand, usually detect the N-protein. Thus, the tests should still be able to detect infection even in the presence of mutations [40]. In the case of since recently dominating the omicron variant, however, initial data show that the sensitivity of the rapid test could be lower [41]. Regular re-evaluations seems absolutely necessary to detect diagnostic deficiencies in time.

It should be emphasized that the validity of the test strongly depends on the PPV, NPV and prevalence, respectively the pre-test probability. These values are highly dynamic due to the fluctuating character of the pandemic. Assuming that the PCR test is almost always correct, the prevalence among PCR-tested persons is approximately equal to the test positive rate. An example from reality; the test positive rate temporarily dropped to 0.82% during the summer in Germany [42]. For the rapid test, this would result in a calculated PPV of 56%. This implies, that almost half of all positive tests would be false-positive. In November 2021, the test positive rate reached up to 21.2% [43]. Here, 98% of the persons with a positive test would actually be infected. However, with a calculated NPV of 94.4%, over 5% of negative tested persons would have an active infection and could potentially pass on the virus.

In summary, the lower the prevalence, the less reliable are positive results, and the higher the prevalence, the less reliable are negative results [44]. To exclude infection, a POCT is best when the incidence of covid-19 is low, whereas in covid-19 waves, a negative antigen test should be followed by a PCR test to truly exclude covid-19 infection.

The additional test caused only minor logistical challenges for GPs and MA. Its regular implementation in everyday practice should be easily possible. GPs and MA considered a physician-performed test to be superior to lay testing because it is presumed to be more accurately administered. Vice versa, the public also seems to have the highest trust in physician-performed antigen tests. According to a survey in Germany, general practices are the preferred testing location for 33% of patients, pharmacies for 25% and testing centers for only 12% [14].

GPs considered positive test results helpful as they were perceived as very safe due to their high specificity and appropriate measures such as quarantine and specific medical care could be initiated immediately.

For the evaluation of the POCT, we used the PCR test as the gold standard. The PCR test is currently the most accurate way to detect an infection. However, this test does not always provide 100% certainty. This can lead to a slight bias in the results. Our RT-PCR samples were evaluated at different accredited laboratories. Despite of a consistently high quality standard, the Ct-values are only comparable to a limited extent. However, it can be assumed that the trends (high Ct-value = low viral load, low Ct-value = high viral load) are still correct. Furthermore, we could not consider the Ct-values of all positive patients because in some cases they were not documented by the practice or provided by the laboratories.

Two different nasopharyngeal swabs were taken consecutively from each patient. Differences in the two sample collections cannot be excluded.

MAs rated the feasibility of the tests worse than the GPs. In some cases, the additional documentation, which was needed for the study, was stated as a reason. In everyday practice without this study-specific documentation, the feasibility should achieve better ratings.

Various authors pointed out that antibody testing may serve as a complementary method for COVID-19 diagnostics [27, 45, 46]. To further verify the results of the POCT and especially PCR testing, we offered antibody testing to all positive tested participants. Unfortunately, only one third of the patients took up this voluntary offer. Another limitation of the antibody testing consisted in the fact that the study was carried out during the second major wave of Covid-19 in Germany. Even though reinfections were rare at the time, they cannot be ruled out upon positive antibody detection. However, although the performed serologies were able to confirm an infection in most of the cases, they also lead to the suggestion that the tests, including the RT-PCR test, do not always provide error-free results. Probably, a second additional PCR test could provide more reliability in future studies.

We hoped to reach the targeted sample size before summer when a strongly decreasing demand for POCT could be expected. As we realized that patient recruitment sharply decreased already in May, we decided to stop the recruitment phase, considering the comparably large group of participants, but had to accept an increase in confidence intervals, especially for sensitivity.