Skip to content
Publicly Available Published by De Gruyter June 29, 2023

Performance evaluation of SARS-CoV-2 antigen detection in the post-pandemic era: multi-laboratory assessment

  • Yuqing Chen , Lei Feng , Yanxi Han , Zihong Zhao , Zhenli Diao , Tao Huang , Yu Ma , Wanyu Feng , Jing Li , Ziqiang Li , Cong Liu , Lu Chang , Jinming Li EMAIL logo and Rui Zhang EMAIL logo

Abstract

Objectives

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antigen detection is an indispensable tool for epidemic surveillance in the post-pandemic era. Faced with irregular performance, a comprehensive external quality assessment (EQA) scheme was conducted by the National Center for Clinical Laboratories (NCCL) to evaluate the analytical performance and status of SARS-CoV-2 antigen tests.

Methods

The EQA panel included ten lyophilized samples containing serial 5-fold dilutions of inactivated SARS-CoV-2-positive supernatants of the Omicron BA.1 and BA.5 strains and negative samples, which were classified into “validating” samples and “educational” samples. Data were analyzed according to qualitative results for each sample.

Results

A total of 339 laboratories in China participated in this EQA scheme, and 378 effective results were collected. All validating samples were correctly reported by 90.56 % (307/339) of the participants and 90.21 % (341/378) of the datasets. The positive percent agreement (PPA) was >99 % for samples with concentrations of 2 × 107 copies/mL but was 92.20 % (697/756) for 4 × 106 copies/mL and 25.26 % (382/1,512) for 8 × 105 copies/mL samples. Colloidal gold was the most frequently used (84.66 %, 320/378) but showed the lowest PPAs (57.11 %, 1,462/2,560) for positive samples compared with fluorescence immunochromatography (90 %, 36/40) and latex chromatography (79.01 %, 335/424). Among 11 assays used in more than 10 clinical laboratories, ACON showed a higher sensitivity than other assays.

Conclusions

The EQA study can help to validate whether it’s necessary to update antigen detection assays for manufacturers and provide participants with information about the performance of assays to take the first step toward routine post-market surveillance.

Introduction

Coronavirus Disease 2019 (COVID-19) has taken the world by storm. In response to the pandemic, various diagnostic strategies for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), including molecular methods [1], antigen detection [2], and serological immunological methods [3], have been developed to address different application scenarios [4, 5]. At the preliminary stage, real-time reverse-transcription polymerase chain reaction (rRT-PCR) was applied for infection confirmation [6, 7]. Then gradual consensus was that asymptomatic or presymptomatic individuals posed a high risk of virus transmission [8, 9], highlighting the need for large-scale screening to identify transmission chains. This increased burden of molecular detection, substantial costs, labor-intensive efforts, and delays in results. Rapid antigen detection was initially implemented as complementary testing to block virus transmission due to its simplicity, rapidity, and cost-effectiveness [4, 10] and was stipulated to be applicable to symptomatic individuals, personnel under quarantine and community residents with self-test demands [11]. During the pandemic peak in late December 2022, the daily average of antigen tests reached a staggering 1.89 million [12], leaving people in the situation of “hard for one antigen testing”. In the post-pandemic era, as China experiences a substantial number of secondary infections [13, 14], the shortage of antigen test kits is now relieved, but there are still substantial demands for antigen testing. Antigen detection is now mainly used as a low-cost rapid detection method by individuals with suspicious symptoms or used in fever clinics for primary screening, whereas PCR tests are not as commonly employed. Antigen detection now serves as a valuable “strategic reserve” for monitoring disease trends and diagnosing those at a high risk of secondary infections.

The World Health Organization (WHO) outlined that the “acceptable” and “desirable” analytical sensitivities for rapid antigen tests are equivalent to 106 and 104 genomic viral copies/mL, respectively [15]. According to the quality evaluation requirements issued by China, approved antigen detection kits should have an analytical sensitivity of at least 5 × 105 copies/mL [16, 17]. However, studies have illustrated that some commercially available kits, particularly in terms of their sensitivities, cannot meet the required criteria [18, 19]. As of December 30th, 2022, the National Medical Products Administration (NMPA) in China has approved 50 SARS-CoV-2 antigen detection assays, including three fluorescence immunochromatography assays, nine latex chromatography assays and 38 colloidal gold assays [20]. Full-scale external quality assessment (EQA) of antigen detection assays is crucial but scarce.

In this pilot study, a nationwide EQA was carried out by the National Center for Clinical Laboratories (NCCL) of China to shed light on the characteristics and performance of the antigen detection products, offer guidance for potential product selection and encourage manufacturers to enhance their kits.

Materials and methods

Sample preparation and panel design

Inactivated cell culture supernatants of the SARS-CoV-2 Omicron BA.1 and BA.5 strains were provided by Sinovac Biotech Co., Ltd. (China) and Sinopharm Group Co., Ltd. (China), respectively. For quantification, viral RNAs were extracted with the QIAamp Viral RNA mini kit (Qiagen, Hilden, Germany) and then reverse transcribed into cDNAs with the PrimeScript RT Reagent Kit (Perfect Real Time; TaKaRa, Japan). Subsequently, the corresponding cDNAs were subjected to real-time quantitative polymerase chain reaction (RT-qPCR) on the Applied Biosystems 7,500 Fast real-time PCR system (Thermo Fisher Scientific, USA) and droplet digital PCR (ddPCR) on the QX-200 System (Bio-Rad, USA) (Figure 1A). The primers/probe set was modified from the N (nucleocapsid) gene assay recommended by China’s Centers for Disease Control (CDC) because of the mutation in the forward primer binding site (Table 1).

Figure 1: 
Flow charts of the EQA scheme. Sample preparation (A), suitability tests (B), homogeneity tests (C), stability tests (D), and geographical distribution and types of laboratories (E).
Figure 1:

Flow charts of the EQA scheme. Sample preparation (A), suitability tests (B), homogeneity tests (C), stability tests (D), and geographical distribution and types of laboratories (E).

Table 1:

Primers/probe set for viral RNA quantification.

Assays Primers/probe sets
China CDC recommended N gene reference assay Forward primer: 5′-GGGGAACTTCTCCTGCTAGAAT-3′

Reverse primer: 5′-CAGACATTTTGCTCTCAAGCTG-3′

Probe: 5′-TTGCTGCTGCTTGACAGATT-3′
Modified N gene assay Forward primer: 5′-AACGAACTTCTCCTGCTAGAAT-3′

Reverse primer: 5′- CAGACATTTTGCTCTCAAGCTG-3′

Probe: 5′-TTGCTGCTGCTTGACAGATT-3′

Detailed information on the sample panel is shown in Table 2. The test panel included ten samples containing serial 5-fold dilutions of inactivated SARS-CoV-2-positive supernatants of the Omicron BA.1 and BA.5 strains (2 × 107 copies/mL, 4 × 106 copies/mL and 8 × 105 copies/mL) and negative samples. Positive samples with 8 × 105 copies/mL and negative samples were set in duplicate to evaluate the repeatability of participants’ procedure. The samples were diluted with phosphate buffer solution (PBS, VivaCell, Shanghai, China) and immediately lyophilized after being dispensed in 0.2-mL aliquots (Supplementary Materials). Prior to distribution, suitability tests were carried out by detecting lyophilized samples with five NMPA-approved antigen tests to confirm that the samples were suitable for the EQA, including tests from BGI Biotech Co., Ltd. (fluorescence immunochromatography), Hangzhou ACON Biotech Co., Ltd. (latex chromatography), Hangzhou AllTest Biotech Co., Ltd. (latex chromatography), Shenzhen YHLO Biotech Co. Ltd. (colloidal gold), and Nanjing Vazyme Biotech Co., Ltd. (colloidal gold) (Figure 1B; Supplementary Materials Figure S1). For homogeneity tests, ten positive samples of each coded number were randomly selected and tested with the Hangzhou AllTest Biotech Co., Ltd., which was proven to have high sensitivity in a previous study (Figure 1C). For stability experiments, samples coded with 2023A03 (Omicron BA.1) and 2023A10 (Omicron BA.5) with 8 × 105 copies/mL were tested using the same kit (Figure 1D). Samples were incubated at 37 °C, room temperature, 4 °C and −20 °C for 1, 3, 5, and 7 days, respectively. The results showed that samples incubated at 37 °C degraded from the third day, while samples remained stable for at least one week at room temperature, 4 °C and −20 °C (Supplementary Materials Figure S2).

Table 2:

SARS-CoV-2 antigen testing EQA scheme – composition of panel and overall results.

Sample content Sample number Sample category Viral RNA concentration (ddPCR copies/mL)a Percent correct qualitative results, %
PPA NPA
Omicron BA.1 variant 2023A02 Validating 2 × 107 100.00
2023A04 Validating 4 × 106 91.27
2023A03 Educational 8 × 105 24.87
2023A07 Educational 8 × 105 22.75
Omicron BA.5 variant 2023A06 Validating 2 × 107 99.47
2023A08 Validating 4 × 106 93.12
2023A01 Educational 8 × 105 25.66
2023A10 Educational 8 × 105 27.78
PBS 2023A05 Validating 99.74
2023A09 Validating 99.74

Overall EQA performance

OPA, %

PPA, %

NPA, %
68.44
60.62
99.74
  1. OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. aViral RNA concentrations were quantified with ddPCR N gene modified assay.

Sample distribution and EQA testing

Lyophilized EQA samples were distributed under ice packs with paper instructions on how to process the samples and submit results. Each sample was redissolved in 200 µL of deionized water. Then, 50 µL was pipetted and added to the sample extraction solution, followed by detection with routine procedures. Upon completion of the EQA sample testing, participants were asked to report qualitative results (“positive” or “negative”) using a feedback questionnaire via a website, as well as upload the original images of results via e-mail to the NCCL. Workflow details, including detection reagents, sample input, volumes of extraction solution, time to result, and limit of detection (LOD), were also collected. We treated EQA samples as human specimens and disposed of them hygienically according to basic requirements.

Statistical analysis

For the EQA scheme, each result returned using an individual antigen test assay (workflow) was considered to be an individual dataset for statistical analysis. Qualitative results were evaluated, and test performance was displayed with the positive percent agreement (PPA)/sensitivity at different viral loads and negative percent agreement (NPA)/specificity. In addition, the difference in the detection of circulating Omicron BA.1 and BA.5 variants was also taken into account.

According to the WHO, the “acceptable” analytical sensitivity of antigen detection assays is equivalent to 106 genomic virus copies/mL [15]. Therefore, “validating” samples composed of Omicron-positive samples with viral loads of 2 × 107 copies/mL and 4 × 106 copies/mL and negative samples of PBS were used to rate participants, but “educational” samples with 8 × 105 copies/mL were also included in the EQA panel to further assess the sensitivity within a wider range without interfering with the performance evaluation for participants. Datasets that reported 100 % correct results and participants who employed ≥2 antigen assays but reported correct results by at least one assay for validating samples (2 × 107 copies/mL, 4 × 106 copies/mL and PBS) were classified as competent; otherwise, they were classified as improvable.

All analyses were performed using Excel (version 2019, Microsoft) and SPSS (version 19.0, IBM Corp.) software. The variance among different groups was compared using Pearson’s chi-square test or Fisher’s exact test, and a P value <0.05 was considered statistically significant.

Ethical approval

The conducted research is not related to either human or animal use.

Results

Participants and methods

A total of 339 laboratories in China participated in this EQA scheme, and 378 effective results were collected because 9.44 % (32/339) of participants reported results using more than one testing kit, with 0.29 % (1/339) of participants adopting four kits for detection. Figure 1E shows the geographical distribution of participants and types of laboratories in detail. Most of the laboratories were hospitals in different classes (n=230, 67.85 %), followed by third-party laboratories (n=86, 25.37 %), reagent manufacturers (n=19, 5.60 %), and local CDC and centers for clinical laboratories (n=4, 1.18 %). Of all returns, fluorescence immunochromatography, latex chromatography and colloidal gold methods accounted for 1.32 % (5/378), 14.02 % (53/378) and 84.66 % (320/378), respectively. Based on the datasets, 38 of 50 antigen testing assays approved by the NMPA were adopted by the participants, among which Zybio, Inc. (colloidal gold, n=71), Shenzhen YHLO Biotech Co., Ltd. (colloidal gold, n=35) and Hangzhou ACON Biotech Co., Ltd. (latex chromatography, n=33) were the top three most frequently used.

Percent agreements of overall results

The qualitative results were analyzed in combination with the original images submitted by the participants, and both were consistent. The color depths of the T-lines by the naked eye matched the virus loads, which became lighter as the viral loads decreased. The performances were found to be competent in 90.56 % (307/339) of the participants and 90.21 % (341/378) of the datasets. The accuracy of laboratory test results was evaluated by calculating the overall percent agreement (OPA), positive percent agreement (PPA) and negative percent agreement (NPA), which were expressed as the ratios of correct results and total results reported for all, positive and negative samples, respectively. For all ten EQA samples, the overall OPA, PPA and NPA were 68.44 % (2,587/3,780), 60.62 % (1,833/3,024) and 99.74 % (754/756), respectively (Table 2). For positive samples with various concentrations, the PPA was greater than 99 % for samples of 2 × 107 copies/mL and 92.20 % (697/756) for samples of 4 × 106 copies/mL. However, the PPA was significantly lower at 25.26 % (382/1,512) for educational samples compared to the two other samples with higher concentrations (p<0.0001) (Figure 2A). Out of the 378 datasets, 21.96 % (83/378) reported positive results for all four educational samples, while 69.05 % (261/378) failed to detect any of the educational samples. The remaining datasets were able to detect one to three out of the four educational samples.

Figure 2: 
The results of panel samples submitted by the participants. Overall results (A), results of fluorescence immunochromatography (B), latex chromatography (C) and colloidal gold (D) methods.
Figure 2:

The results of panel samples submitted by the participants. Overall results (A), results of fluorescence immunochromatography (B), latex chromatography (C) and colloidal gold (D) methods.

Percent agreements of results for positive samples

Detection methods

In this EQA scheme, one fluorescence immunochromatography, seven latex chromatography and 30 colloidal gold methods were used by the participants. Table 3 provides an overview of the datasets produced by the three methods. For all EQA samples, the OPAs were 92 % (46/50), 83.02 % (440/530) and 65.66 % (2,101/3,200) for the fluorescence immunochromatography, latex chromatography and colloidal gold methods, respectively. Among validating samples, 100 % (5/5), 92.45 % (49/53) and 89.69 % (287/320) of datasets using fluorescence immunochromatography, latex chromatography and colloidal gold methods reported correct results, which were identified as competent. All three methods demonstrated 100 % PPAs for samples of 2 × 107 copies/mL and PPAs exceeding 90 % for samples of 4 × 106 copies/mL. However, for the results of a viral load of 8 × 105 copies/mL, the PPAs varied considerably, and the performance of colloidal gold (18.52 %, 237/1,280) was far behind those of fluorescence immunochromatography (60.85 %, 129/212) and latex chromatography (80 %, 16/20) (p<0.0001). The results of the colloidal gold method had a significantly higher rate of false-negatives in comparison with the other two methods (p<0.0001) (Figure 2B–D).

Table 3:

SARS-CoV-2 antigen testing EQA scheme – performance of different antigen testing methods.

Samples Method performance, % for:
Fluorescence immunochromatography (n=5) Latex chromatography (n=53) Colloidal gold (n=320)
Positive samplesa Percent correct qualitative results (PPA, %)

2023A02

Omicron BA.1 variant

2 × 107 copies/mL
100 100 100
2023A04

Omicron BA.1 variant

4 × 106 copies/mL
100 94.34 90.63
2023A03

Omicron BA.1 variant

8 × 105 copies/mL
80 60.38 18.13
2023A07

Omicron BA.1 variant

8 × 105 copies/mL
80 56.60 16.25
2023A06

Omicron BA.5 variant

2 × 107 copies/mL
100 96.23 100
2023A08

Omicron BA.5 variant

4 × 106 copies/mL
100 98.11 92.19
2023A01

Omicron BA.5 variant

8 × 105 copies/mL
80 58.49 19.38
2023A10

Omicron BA.5 variant

8 × 105 copies/mL
80 67.92 20.31

Negative samples Percent correct qualitative results (NPA, %)

2023A05

PBS
100 98.11 100
2023A09

PBS
100 100 99.69

Overall EQA performance

OPA, % 92 83.02 65.66
PPA, % 90 79.01 57.11
NPA, % 100 99.06 99.84
  1. OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. aViral RNA concentrations quantified with ddPCR N gene modified assay.

Detection assays

For most antigen assays used in the scheme, the number of reported datasets was too limited to provide strong evidence for the detection performance of individual assays. Therefore, 11 antigen assays employed by >10 laboratories herein were analyzed (Table 4). Assays manufactured by ACON, YHLO, Kehua and Kanghua demonstrated PPAs of 92.42 % (244/264), 73.21 % (205/280), 68.75 % (77/112) and 64.47 % (98/152) for all positive samples, respectively, whereas the other seven assays showed PPAs lower than 60 % (Figure 3). For samples with 2 × 107 copies/mL, the PPAs of the 11 assays were all >95 %. However, for 4 × 106 and 8 × 105 copies/mL, the rates were variable. For the concentration of 4 × 106 copies/mL, Vazyme and Wantai had PPA values inferior to 80 %, while the remaining assays all displayed PPA values greater than 80 %. Moreover, Vazyme, Wantai, Orient Gene and Bioscience were essentially unable to detect educational samples, indicating that the LODs of these four test kits were beyond 8 × 105 copies/mL. Notably, only ACON showed a higher positive rate (>80 %) when detecting educational samples, resulting in a prominently increased OPA and PPA compared to the other ten antigen assays.

Table 4:

SARS-CoV-2 antigen testing EQA scheme – performance of antigen testing assays employed by >10 laboratories.

Items Assay performance, % for:
Zybio (n=71) YHLO (n=35) ACON (n=33) Vazyme (n=28) Wondfo (n=24) Kanghua (n=19) Wantai (n=16) Kehua (n=14) EasyDiagnosis (n=14) Orient gene (n=11) Bioscience (n=11)
LOD (TCID50/mL) 70 250 160 50 850 64 137 500 500 Not available 850
LOD (copies/mL) 7 × 105 2.5 × 106 1.6 × 106 5 × 105 8.5 × 106 6.4 × 105 1.37 × 106 5 × 106 5 × 106 Not available 8.5 × 106

Positive samples a Percent correct qualitative results (PPA, %)

2023A02

Omicron BA.1 variant

2 × 107 copies/mL
100 100 100 100 100 100 100 100 100 100 100
2023A04

Omicron BA.1 variant

4 × 106 copies/mL
98.59 97.14 100 67.86 87.5 100 50 100 100 81.82 100
2023A03

Omicron BA.1 variant

8 × 105 copies/mL
14.08 45.71 84.85 0 12.5 31.58 0 42.86 14.29 0 0
2023A07

Omicron BA.1 variant

8 × 105 copies/mL
12.68 45.71 81.82 0 12.5 26.32 0 28.57 14.29 0 0
2023A06

Omicron BA.5 variant

2 × 107 copies/mL
100 100 96.97 100 100 100 100 100 100 100 100
2023A08

Omicron BA.5 variant

4 × 106 copies/mL
97.18 97.14 100 75 91.67 100 56.25 100 100 100 100
2023A01

Omicron BA.5 variant

8 × 105 copies/mL
14.08 48.57 84.85 0 16.67 26.32 0 42.86 28.57 0 9.09
2023A10

Omicron BA.5 variant

8 × 105 copies/mL
18.31 51.43 90.91 0 12.5 31.58 0 35.71 14.29 0 0

Negative samples Percent correct qualitative results (NPA, %)

2023A05

PBS
100 100 96.97 100 100 100 100 100 100 100 100
2023A09

PBS
100 100 100 100 100 100 93.75 100 100 100 100

Overall EQA performance

OPA, % 65.49 78.57 93.64 54.29 63.33 71.58 50 75 67.14 58.18 60.91
PPA, % 56.87 73.21 92.42 42.86 54.17 64.47 38.28 68.75 58.93 47.73 51.14
NPA, % 100 100 98.48 100 100 100 96.88 100 100 100 100
  1. LOD, limit of detection; TCID50, tissue culture infectious dose required to infect 50 % of the cells; OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. Zybio, Zybio, Inc.(colloidal gold); YHLO, Shenzhen YHLO Biotech Co. Ltd. (colloidal gold); ACON, Hangzhou ACON Biotech Co., Ltd. (latex chromatography); Vazyme, Nanjing Vazyme Biotech Co., Ltd. (colloidal gold); Wondfo, Guangzhou Wondfo Biotech Co. Ltd. (colloidal gold); Kanghua, Shandong Kanghua Biotech Co., Ltd. (colloidal gold); Wantai, Beijing Wantai Biological Pharmacy Enterprise Co., Ltd. (colloidal gold); Kehua, Shanghai Kehua Bio-engineering Co., Ltd. (colloidal gold); EasyDiagnosis, Wuhan EasyDiagnosis Biomedicine Co., Ltd. (colloidal gold); Orient Gene, Zhejiang Orient Gene Biotech Co., Ltd. (colloidal gold); Bioscience, Tianjin Bioscience Bio-technology Co., Ltd. (colloidal gold). aViral RNA concentrations quantified with ddPCR N gene modified assay.

Figure 3: 
Datasets for antigen testing assays employed by >10 laboratories.
Figure 3:

Datasets for antigen testing assays employed by >10 laboratories.

BA.1 and BA.5 variants

For the Omicron BA.1 and BA.5 variants, the OPAs were 59.72 % (903/1,512) and 61.51 % (930/1,512), respectively, and no significant difference was observed (p=0.315). For 2 × 107 copies/mL and 4 × 106 copies/mL, the PPAs of BA.1 and BA.5 were all >90 %. However, the PPAs of the BA.1 and BA.5 variants were 23.81 % (180/756) and 26.72 % (202/756) for 8 × 105 copies/mL, respectively. Similar PPAs were shown for the two variants at the three concentrations.

Percent agreements of results for negative samples

The NPAs of 2023A05 and 2023A09 were both 99.74 % (377/378). Except for ACON and Wantai, 36 antigen assays used by the participants could correctly test all negative samples. The false-positive rates for ACON and Wantai were 1.52 % (1/66) and 3.13 % (1/32), respectively.

Discussion

In this study, an EQA program was implemented by the NCCL to assess the testing capabilities of laboratories using antigen detection assays that have been approved by the NMPA. Regarding the intrinsic characteristic of lower analytical sensitivity for antigen detection (105–106 copies/mL) compared to rRT-PCR [21, 22], inactivated cell culture supernatants of the SARS-CoV-2 Omicron BA.1 and BA.5 strains with high (2 × 107 copies/mL), medium (4 × 106 copies/mL), and low (8 × 105 copies/mL) concentrations were prepared as EQA-positive samples. Validating samples were used to categorize the detection performance, and educational samples were set in duplicate to enable a more comprehensive sensitivity assessment and evaluate the repeatability of the participants’ procedure.

In the EQA scheme, 378 effective results were collected from 339 laboratories in China. According to the scoring principals, 90.56 % (307/339) of the participants and 90.21 % (341/378) of the datasets were identified as competent. As the sample concentrations decreased from 2 × 107 copies/mL to 8 × 105 copies/mL, the PPAs of the datasets decreased from 99.74 % (754/756) to 25.26 % (382/1,512). This trend indicated that the LODs for most assays were beyond 8 × 105 copies/mL, making it challenging to detect weak positive samples. Next, we will delve into the discrepancies in results among three detection methods, different detection kits, and different laboratories.

The three detection methods exhibited significant disparities in terms of usage rates and PPAs. Colloidal gold was the most frequently used (84.66 %, 320/378) but showed the lowest PPA (57.11 %, 1,462/2,560). In contrast, fluorescence immunochromatography was the least popular (1.32 %, 5/378) but had the highest PPA of 90 % (36/40). Latex chromatography showed an intermediate application rate (14.02 %, 53/378) and PPA (79.01 %, 335/424). The limited adoption of fluorescence immunochromatography may be attributed to the need for a compatible reader, which adds complexity compared to visual interpretation and incurs additional costs for technical maintenance [19, 23]. Despite its lower prevalence, the integration of the fluorescence immunolabeling technique with fluorescence photometric signal detection has significantly enhanced the analytical sensitivity by 1–2 orders of magnitude, surpassing the sensitivity of the other two methods [24]. Discrepancies in results among the three detection methods were evident for educational samples, where fluorescence immunochromatography demonstrated a noteworthy 80 % (16/20) PPA, while colloidal gold only displayed a PPA of 18.52 % (237/1,280), which was consistent with the findings of our previous study [25]. However, it is important to emphasize that the conclusions drawn regarding the fluorescence immunochromatography method may be influenced by the limited dataset generated from only five laboratories, which could have introduced statistical bias.

To minimize analysis bias, we focused on 11 antigen assays performed daily in more than 10 laboratories, as most antigen assays used in this study were adopted by only a few laboratories. For all positive samples, ACON stood out as the exceptional reagent kit with a PPA of 92.42 %. Additionally, YHLO, Kehua, and Kanghua exhibited relatively strong PPAs of over 60 %. The 11 assays reported almost 100 % PPAs for samples with 2 × 107 copies/mL; however, Vazyme and Wantai showed obviously decreased PPAs for samples with 4 × 106 copies/mL. Notably, Vazyme, Wantai, Orient Gene and Bioscience were essentially unable to detect the educational samples at all, while ACON showed PPAs greater than 80 %, indicating that ACON was the most sensitive of the 11 assays. According to the study conducted by Sender R et al. [26], the claimed LODs of the 11 test kits varied between 5 × 105 and 8.5 × 106 copies/mL after conversion from TCID50 (tissue culture infectious dose required to infect 50 % of the cells). Among these 11 kits, Zybio, Vazyme, Kanghua, and Wantai produced false-negative results when the virus loads were at or above their specified LODs, while the remaining seven tests showed detection performances that matched their claimed LODs.

In the EQA scheme, discordant results were observed in datasets from laboratories using the same kits, indicating inconsistency between laboratories, which might be caused by the analytical sensitivity of assays, operational factors, as well as between-batch differences. For educational samples with 8 × 105 copies/mL, eight kits showed laboratory inconsistency to varying degrees. Among them, YHLO exhibited the highest variation, with a PPA of 47.86 % (67/140), followed by Kehua and Kanghua, with PPAs of more than 20 %. The inter-laboratory inconsistency may be due to the fact that 8 × 105 copies/mL was below their LODs, and it was expected to see a significant decrease in positive results for educational samples. For samples with 4 × 106 copies/mL, Wantai and Vazyme showed higher laboratory inconsistency with PPAs of 53.13 % (17/32) and 71.43 % (40/56), respectively. Because the LODs for the two kits were below 4 × 106 copies/mL, the inconsistency between laboratories may be caused by operational factors or between-batch differences. To minimize batch-to-batch variability, routine reagent batch validation is recommended using negative, weak positive and medium positive samples across different time points and operating conditions, utilizing different reagent lots and involving different operators.

In terms of negative samples coded with 2023A05 and 2023A09 composed of PBS, all 11 kits showed NPAs of 100 % except for ACON and Wantai. By analysis, the reason for false-positive results by ACON was that one laboratory accidentally reversed the results of samples coded as 2023A05 (PBS) and 2023A06 (Omicron BA.5 with 2 × 107 copies/mL). For Wantai, considering the relatively poor detection performance, which had an approximately 50 % PPA for samples of 4 × 106 copies/mL and could not detect educational samples at all, positive results were unexpectedly reported for negative samples. After tracing the original images and excluding the possibility of mistakes in completing the questionnaire, we speculated that the false-positive results were possibly due to poor quality of the kit batch, occasional operational errors or incorrect result interpretation.

Based on the aforementioned statements, the analytical sensitivities of the methods and reagent kits were the major cause of performance differences [18, 19]. The intrinsic properties of assays, such as the antigens targeted, the antibodies coated, and the method of coloration, are the main determinants for analytical sensitivity [27]. Research efforts are being dedicated to targeting antigens other than the N protein, such as the spike protein [28, 29] or a combination of multiple proteins [30, 31], in order to enhance detection sensitivity. The density, affinity, and specific epitopes targeted by coated antibodies also play a crucial role in determining the detection sensitivity. Additionally, fluorescent labeling combined with instrumentation significantly improves the sensitivity of the analysis compared to the less objective naked-eye interpretation of colloidal gold and latex methods [24]. Considering the considerable risk of false-negative rates, it is important to rectify the public perception that negative results definitively rule out infections.

In this study, the inclusion of educational samples was highly valuable, as performance differences were mainly displayed by educational samples with low concentrations among detection methods, detection kits, and laboratories. EQA conducted here is the initial step into post-market surveillance. Regular participation in well-designed EQAs can alert technicians to pay attention to proper sample handling and accurate result interpretation. However, relying solely on data from one EQA and two types of variants would provide limited information. Therefore, follow-up research encompassing a wider range of variants is required to gather dynamic data and provide a holistic view of antigen detection kits in the post-pandemic era.


Corresponding authors: Jinming Li, PhD and Rui Zhang, PhD, National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital/National Center of Gerontology, Beijing, P.R. China; National Center for Clinical Laboratories, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, P.R. China; Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, P.R. China; and National Center for Clinical Laboratories, Beijing Hospital, No. 1 Dahua Road, Dongdan, Beijing, 100730, P.R. China, Phone: +86-10-58115053, Fax: 86-10-65212064, E-mail: (J. Li), (R. Zhang)
Yuqing Chen and Lei Feng contributed equally to this work.

Funding source: the National Key Research and Development Program of China

Award Identifier / Grant number: 2022YFC0869900

  1. Research funding: This study was supported by the National Key Research and Development Program of China (2022YFC0869900).

  2. Author contribution: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Not applicable.

  5. Ethical approval: Not applicable.

References

1. Kang, T, Lu, J, Yu, T, Long, Y, Liu, G. Advances in nucleic acid amplification techniques (NAATs): COVID-19 point-of-care diagnostics as an example. Biosens Bioelectron 2022;206:114109. https://doi.org/10.1016/j.bios.2022.114109.Search in Google Scholar PubMed

2. Pavia, CS, Plummer, MM. The evolution of rapid antigen detection systems and their application for COVID-19 and other serious respiratory infectious diseases. J Microbiol Immunol Infect 2021;54:776–86. https://doi.org/10.1016/j.jmii.2021.06.003.Search in Google Scholar PubMed PubMed Central

3. West, R, Kobokovich, A, Connell, N, Gronvall, GK. COVID-19 antibody tests: a valuable public Health tool with limited relevance to individuals. Trends Microbiol 2021;29:214–23. https://doi.org/10.1016/j.tim.2020.11.002.Search in Google Scholar PubMed PubMed Central

4. Candel, FJ, Barreiro, P, San Roman, J, Abanades, JC, Barba, R, Barberan, J, et al.. Recommendations for use of antigenic tests in the diagnosis of acute SARS-CoV-2 infection in the second pandemic wave: attitude in different clinical settings. Rev Esp Quimioter 2020;33:466–84. https://doi.org/10.37201/req/120.2020.Search in Google Scholar PubMed PubMed Central

5. Peeling, RW, Heymann, DL, Teo, YY, Garcia, PJ. Diagnostics for COVID-19: moving from pandemic response to control. Lancet 2022;399:757–68. https://doi.org/10.1016/s0140-6736(21)02346-1.Search in Google Scholar

6. Jamil, S, Mark, N, Carlos, G, Cruz, CSD, Gross, JE, Pasnick, S. Diagnosis and management of COVID-19 disease. Am J Respir Crit Care Med 2020;201:P19–P20. https://doi.org/10.1164/rccm.2020c1.Search in Google Scholar

7. Jin, YH, Cai, L, Cheng, ZS, Cheng, H, Deng, T, Fan, YP, et al.. A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version). Mil Med Res 2020;7:4. https://doi.org/10.1186/s40779-020-0233-6.Search in Google Scholar PubMed PubMed Central

8. Buitrago-Garcia, D, Egli-Gany, D, Counotte, MJ, Hossmann, S, Imeri, H, Ipekci, AM, et al.. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: a living systematic review and meta-analysis. PLoS Med 2020;17:e1003346. https://doi.org/10.1371/journal.pmed.1003346.Search in Google Scholar PubMed PubMed Central

9. Gao, Z, Xu, Y, Sun, C, Wang, X, Guo, Y, Qiu, S, et al.. A systematic review of asymptomatic infections with COVID-19. J Microbiol Immunol Infect 2021;54:12–6. https://doi.org/10.1016/j.jmii.2020.05.001.Search in Google Scholar PubMed PubMed Central

10. Dinnes, J, Deeks, JJ, Berhane, S, Taylor, M, Adriano, A, Davenport, C, et al.. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev 2021;3:CD013705. https://doi.org/10.1002/14651858.CD013705.pub2.Search in Google Scholar PubMed PubMed Central

11. The Joint Prevention and Control Mechanism. Application scheme for COVID-19 antigen detection. https://www.gov.cn/xinwen/2022-03/11/content_5678610.htm [Accessed 7 Jun 2023].Search in Google Scholar

12. Chinese Center for Disease Control and Prevention. Epidemic situation of COVID-19 in China. https://www.chinacdc.cn/jkzt/crb/zl/szkb_11803/jszl_13141/202304/t20230429_265709.html [Accessed 7 Jun 2023].Search in Google Scholar

13. Callaway, E. COVID’s future: mini-waves rather than seasonal surges. Nature 2023;617:229–30. https://doi.org/10.1038/d41586-023-01437-8.Search in Google Scholar PubMed

14. Cai, C, Li, Y, Hu, T, Liang, R, Wang, K, Guo, C, et al.. Preplanned studies: the associated factors of SARS-CoV-2 reinfection by Omicron variant — guangdong province, China, december 2022 to january 2023. China CDC Weekly 2023;5:391–6. https://doi.org/10.46234/ccdcw2023.075.Search in Google Scholar PubMed PubMed Central

15. World Health Organization. COVID-19 target product profiles for priority diagnostics to support response to the COVID-19 pandemic v.1.0. https://www.who.int/publications/m/item/covid-19-target-product-profiles-for-priority-diagnostics-to-support-response-to-the-covid-19-pandemic-v.0.1 [Accessed 7 Jun 2023].Search in Google Scholar

16. National Institutes for Food and Drug Control. National reference panel for 2019-nCoV (SARS-CoV-2) antigen detection kit. http://aoc.nifdc.org.cn/sell/uploadft/167575391094490468911203467305.pdf [Accessed 7 Jun 2023].Search in Google Scholar

17. The State Administration for Market Regulation. Quality assessment requirements for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antigen detection kit. http://c.gb688.cn/bzgk/gb/showGb?type=online&hcno=2735D79B00E92A67221E43FA5776D00B [Accessed 7 Jun 2023].Search in Google Scholar

18. Scheiblauer, H, Filomena, A, Nitsche, A, Puyskens, A, Corman, VM, Drosten, C, et al.. Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021. Euro Surveill 2021;26:2100441. https://doi.org/10.2807/1560-7917.es.2021.26.44.2100441.Search in Google Scholar PubMed PubMed Central

19. Puyskens, A, Krause, E, Michel, J, Nubling, CM, Scheiblauer, H, Bourquain, D, et al.. Establishment of a specimen panel for the decentralised technical evaluation of the sensitivity of 31 rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021. Euro Surveill 2021;26:2011442. https://doi.org/10.2807/1560-7917.es.2021.26.44.2100442.Search in Google Scholar PubMed PubMed Central

20. National Medical Products Administration. National Medical Products Administration has approved 50 COVID-19 antigen detection assays. https://www.nmpa.gov.cn/yaowen/ypjgyw/20221230202032116.html [Accessed 7 Jun 2023].Search in Google Scholar

21. Kohmer, N, Toptan, T, Pallas, C, Karaca, O, Pfeiffer, A, Westhaus, S, et al.. The comparative clinical performance of four SARS-CoV-2 rapid antigen tests and their correlation to infectivity in vitro. J Clin Med 2021;10:328. https://doi.org/10.3390/jcm10020328.Search in Google Scholar PubMed PubMed Central

22. Toptan, T, Eckermann, L, Pfeiffer, AE, Hoehl, S, Ciesek, S, Drosten, C, et al.. Evaluation of a SARS-CoV-2 rapid antigen test: potential to help reduce community spread? J Clin Virol 2021;135:104713. https://doi.org/10.1016/j.jcv.2020.104713.Search in Google Scholar PubMed PubMed Central

23. Cubas-Atienzar, AI, Kontogianni, K, Edwards, T, Wooding, D, Buist, K, Thompson, CR, et al.. Limit of detection in different matrices of 19 commercially available rapid antigen tests for the detection of SARS-CoV-2. Sci Rep 2021;11:18313. https://doi.org/10.1038/s41598-021-97489-9.Search in Google Scholar PubMed PubMed Central

24. Zhang, C, Zhou, L, Du, K, Zhang, Y, Wang, J, Chen, L, et al.. Foundation and clinical evaluation of a new method for detecting SARS-CoV-2 antigen by fluorescent microsphere immunochromatography. Front Cell Infect Microbiol 2020;10:553837. https://doi.org/10.3389/fcimb.2020.553837.Search in Google Scholar PubMed PubMed Central

25. Chen, Y, Ma, Y, Han, Y, Diao, Z, Chang, L, Li, J, et al.. Evaluation of four strategies for SARS-CoV-2 detection: characteristics and prospects. Microbiol Spectr 2022;10:e0214322. https://doi.org/10.1128/spectrum.02143-22.Search in Google Scholar PubMed PubMed Central

26. Sender, R, Bar-On, YM, Gleizer, S, Bernshtein, B, Flamholz, A, Phillips, R, et al.. The total number and mass of SARS-CoV-2 virions. Proc Natl Acad Sci U S A 2021;118:e2024815118. https://doi.org/10.1073/pnas.2024815118.Search in Google Scholar PubMed PubMed Central

27. Donoso Mantke, O, Corman, VM, Taddei, F, McCulloch, E, Niemeyer, D, Grumiro, L, et al.. Importance of external quality assessment for SARS-CoV-2 antigen detection during the COVID-19 pandemic. J Clin Virol 2022;154:105222. https://doi.org/10.1016/j.jcv.2022.105222.Search in Google Scholar PubMed PubMed Central

28. Lee, JH, Choi, M, Jung, Y, Lee, SK, Lee, CS, Kim, J, et al.. A novel rapid detection for SARS-CoV-2 spike 1 antigens using human angiotensin converting enzyme 2 (ACE2). Biosens Bioelectron 2021;171:112715. https://doi.org/10.1016/j.bios.2020.112715.Search in Google Scholar PubMed PubMed Central

29. Huang, J, Wen, J, Zhou, M, Ni, S, Le, W, Chen, G, et al.. On-site detection of SARS-CoV-2 antigen by deep learning-based surface-enhanced Raman spectroscopy and its biochemical foundations. Anal Chem 2021;93:9174–82. https://doi.org/10.1021/acs.analchem.1c01061.Search in Google Scholar PubMed

30. Ventura, BD, Cennamo, M, Minopoli, A, Campanile, R, Censi, SB, Terracciano, D, et al.. Colorimetric test for Fast detection of SARS-CoV-2 in nasal and throat swabs. ACS Sens 2020;5:3043–8. https://doi.org/10.1021/acssensors.0c01742.Search in Google Scholar PubMed PubMed Central

31. Di Domenico, M, De Rosa, A, Di Gaudio, F, Internicola, P, Bettini, C, Salzano, N, et al.. Diagnostic accuracy of a new antigen test for SARS-CoV-2 detection. Int J Environ Res Publ Health 2021;18:6310. https://doi.org/10.3390/ijerph18126310.Search in Google Scholar PubMed PubMed Central


Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/cclm-2023-0597).


Received: 2023-04-22
Accepted: 2023-06-14
Published Online: 2023-06-29
Published in Print: 2023-11-27

© 2023 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 4.3.2024 from https://www.degruyter.com/document/doi/10.1515/cclm-2023-0597/html
Scroll to top button