Skip to content
BY 4.0 license Open Access Published by De Gruyter May 23, 2022

Results of a SARS-CoV-2 virus genome detection external quality assessment round focusing on sensitivity of assays and pooling of samples

Christoph Buchta ORCID logo, Jeremy V. Camp, Jovana Jovanovic, Elisabeth Puchhammer-Stöckl, Robert Strassl, Mathias M. Müller, Andrea Griesmacher, Stephan W. Aberle and Irene Görzer

Abstract

Objectives

Results of earlier external quality assessment (EQA) rounds suggested remarkable differences in the sensitivity of SARS-CoV PCR assays. Although the test systems are intended to detect SARS-CoV-2 in individual samples, screening is often applied to sample pools to increase efficiency and decrease costs. However, it is unknown to what extent these tests actually meet the manufacturer’s specifications for sensitivity and how they perform when testing sample pools.

Methods

The sensitivity of assays in routine use was evaluated with a panel of positive samples in a round of a SARS-CoV-2 virus genome detection EQA scheme. The panel consisted of samples at or near the lower limit of detection (“weakly positive”). Laboratories that routinely test sample pools were asked to also analyze the pooled EQA samples according to their usual pool size and dilution method.

Results

All participants could detect a highly positive patient-derived sample (>106 copies/mL). Most (96%) of the test systems could detect at least 1,000 copies/mL, meeting the minimum acceptable benchmark, and many (94%) detected the vRNA in a sample with lower concentration (500 copies/mL). The false negative ratio increased to 16 and 26% for samples with 100 and 50 copies/mL, respectively.

Conclusions

The performance of most assays met or exceeded their specification on sensitivity. If assays are to be used to analyze sample pools, the sensitivity of the assay and the number of pooled samples must be balanced.

Introduction

An assay’s sensitivity affects its clinical performance in SARS-CoV-2 testing, and if too low, it can lead to false-negative results that are not conducive to managing the pandemic [1]. Therefore, the World Health Organization (WHO) describes a limit of detection (LoD) of 1,000 copies/mL as “acceptable”, but 100 copies/mL as “desirable” [2]. Results from previous rounds of the SARS-CoV-2 virus genome detection external quality assessment (EQA) scheme of the Austrian Association for Quality Assurance and Standardization of Medical and Diagnostic Tests (ÖQUASTA) and the Center for Virology at the Medical University Vienna (the national reference laboratory for respiratory viruses) suggested remarkable differences in the sensitivities of assays in routine use [3]. Determination of the LoD is part of the validation of an assay by the manufacturer and the verification by the user [4]. If sample pools are analyzed, the sensitivity of the assay and the size of the pool must be balanced in order to reliably identify SARS-CoV-2 RNA from one positive sample in the pool [5]. After assay validation and verification procedures, the performance of diagnostics in routine use should be closely monitored. It is advisable to include data from external quality assessment schemes as they are reports from the “real life” performance apart from validation or verification settings [6]. We therefore dedicated an EQA round in early 2022 to the sensitivity of SARS-CoV-2 nucleic acid test (NAT) assays in routine use.

Materials and methods

A panel of seven samples was designed to compare the performance of SARS-CoV-2 NAT assays in routine laboratory use against the LoD recommended by WHO. Preparation and shipment of samples (including quality control assessment under storage and shipping conditions) were performed as previously described [7]. Instructions for participants to perform the analysis and to report the results, the collection of results and their independent assessment were performed by the EQA provider, and feedback was provided to the participants as previously described [7].

In addition to the data that were to be provided by participants in previous rounds, laboratories routinely screening sample pools were asked in this round to dilute the EQA samples with negative samples or buffer solution according to their standard pool size. These pools should then be analyzed and the samples should subsequently be individually tested.

One positive sample (S2) was a clinical specimen from a swab positive for SARS-CoV-2 variant omicron (BA.2), yielding Ct 23.8, corresponding to a virus load of 9.3 × 106 copies/mL, as characterized by the reference laboratory. One other sample (S3) was negative for SARS-CoV-2 RNA. Five positive samples were prepared by making dilutions of the AccuPlex™ SARS-CoV-2 Reference Material (Material Number 0505-0126, SeraCare, Milford, MA, USA) to yield the following calculated concentrations: ∼50 copies/mL (1:100 dilution, S6); ∼100 copies/mL (1:50 dilution, S1); ∼500 copies/mL (1:10 dilution, S4); and ∼1,000 copies/mL (1:5 dilution, S5 and S7). Two samples with ∼1,000 copies/mL were provided to allow discrimination of false negative results caused by random error from those caused by limited sensitivity of assays. Human cell culture material was added to all samples to provide human housekeeping genes for assays that test and require this for a valid result. The requirement to pass this EQA round was to obtain correct results for the negative sample, the highly positive clinical sample and for the positive samples with ∼1,000 and ∼500 copies/mL SARS-CoV-2.

Results

A total of 133 laboratories participated in this round and used 160 assays in total, among which were 72 different combinations of 45 extraction reagents on 39 extraction platforms, and 48 detection reagents on 35 detection platforms. A total of 52 (32.5%) assays were represented by only one to three individual participant reports in this EQA round. Taking into account that reporting of correct results for S2, S3, S4, S5, and S7 was required to pass, the success rate in this round was 149/160 (92.5%). In descending order of the virus load in the EQA samples, S2 (∼9.3E+6 copies/mL) was reported positive by every (100%) assay; S5 and S7 (∼1,000 copies/mL) were incorrectly reported negative by six (3.8%) and seven (4.4%) assays, respectively; S4 (∼500 copies/mL) was reported negative by 9 (5.6%), S1 (∼100 copies/mL) by 25 (15.6%), and S6 (∼50 copies/mL) by 41 (25.6%). Sample S3 was correctly reported negative by 159/160 (99.4%) assays. A total of 112 assays reported correct results for all seven samples (Table 1).

Table 1:

False negative/total results and approximate virus load in six positive individual (non-pooled) EQA samples by test system.

  1. TCID50, 50% tissue culture infectious dose; GC, genetic copies; GE, gene equivalents; PFU, plaque forming units. aIncluding 2019-nCoV nucleic acid test kit (Hecin), Abbott ID NOW, Abbott RT Sars-CoV-2, Alinity m SARS-Cov-2 Assay, ANDiS FAST SARS-CoV-2 RT-qPCR Detection Kit, Artus SARS-CoV-2 Prep&Amp UM Kit, SARS-CoV-2 und Influenza A/B, Convergys POC RT-PCR- SARS-CoV-2, COVID-19 Nucleic acid (RNA) Detection Kit, Easy SARS-Cov-2 Kit, FTD SARS CoV-2, genesig Real-Time PCR, GenomeCoV19 Detection kit, Luna Universal Probe One-Step, Luna Universal/One-Step-RT-qPCR (Covid), Molaccu Covid-19-Detection Kit Zybio, Multiplex RNA Virus Master Kit, MutaPlex RT-PCR Kit, NeuMoDx SARS-CoV-2 Test Strip, PerkinElmer SARS-CoV-2 Real-time RT-PCR Assay, PhoenixDX, PhoenixDx SARS-CoV-2 P681R Multiplex, QIAprep&amp Viral RNA, Real AccurateR Quadruplex SARS-CoV-2 Multiplex RT, Real Fast PCR, Respiratory Viruses 16-well (AusDiagnostics), SARS-CoV-2-N + RdRp Cubedx, SARS-CoV-RealFast Assay/ViennaLab, TaqMan Fast Virus1Step MasterMix, TaqPath Covid 19, TaqPath TM, TIB MOLBIOL Light MIX, Virella SARS-CoV-2 seqc RT-PCR Kit 2.0, Vivalytic RTI, Volcano3G® RT-PCR SARS-Cov-2.

A total of 153 (95.6%) and 152 (95.0%) assays detected at least ∼1,000 copies/mL, which was the concentration of SARS-CoV-2 in S5 and S7, respectively. Among the assays that did not detect both S5 (n=6 false negatives) and S7 (n=7) were the Abbott ID Now™ (Abbott Diagnostics Scarborough, Inc., Maine, USA) (LoD according to manufacturer information 125 copies/mL, n=1), PhoenixDx SARS-CoV-2 P681R Multiplex (Procomcure Biotech, Austria) (LoD 3 copies/μL eluate, n=2), ViroReal Kit SARS-CoV-2 & SARS (Ingenetix GmbH, Vienna, Austria) (LoD 893 copies/mL, n=2). One of the samples S5 and S7 was not detected positive by one (out of two) 2019-nCoV nucleic acid test kit (Hecin Guangdong Scientific, Inc., China) (LoD 400 copies/mL), one (out of two) Luna Universal/One-Step-RT-qPCR (Covid) (New England Biolabs) (LoD 5 copies/mL), and in total three (out of six) VitaPCR SARS-CoV-2 Assay (A. Menarini Diagnostics, Florence, Italy) (LoD 1,000 copies/mL). Eight of these nine assays and one (out of two) RealAccurate® Quadruplex SARS-CoV-2 PCR kit (Biozym Scientific GmbH, Germany) (LoD 5 copies/reaction) failed to detect S4 (∼500 copies/mL). Another 15 did not detect S1 (∼100 copies/mL), and another 19 did not detect S6 (∼50 copies/mL) as positive (data not shown).

Five laboratories routinely screened pools of patient samples, in pool sizes of five to ten patients, and reported results from individual test samples as if they were pooled. Additionally, one laboratory reported results only from pooled testing. In mock pools and in follow-up individual testing, S2 (∼9.3 × 106 copies/mL) and S7 (∼1,000 copies/mL) were reported positive by all five assays. Among the false negative test results from pooled testing, ANDiS FAST SARS-CoV-2 RT-qPCR Detection Kit (3D Biomedicine Science & Technology Co. Ltd, Shanghai, China) (LoD 5 copies/reaction) missed S1 (∼100 copies/mL) and S6 (∼50 copies/mL) in a pool of eight samples but detected both samples as positive in individual tests. The Aptima SARS-CoV-2 Assay (Hologic, Inc., California, USA) (LoD 0.001 TCID50/mL) detected all samples as positive in both individual testing and in a pool of five samples. The cobas SARS-CoV-2 assay (Roche Molecular Systems, Inc. New Jersey, USA) (LoD 46 copies/mL) detected all samples as positive in individual tests, but in a pool of 10 samples, S6 was missed. S1 and S6 were missed by Molaccu Covid-19 detection kit (Zybio, Chongqing, China) (LoD ≤500 copies/mL) in a pool of six samples as well as in individual tests. S5 (∼1,000 copies/mL), S4 (∼500 copies/mL), S1, and S6 were missed by ViroReal Kit SARS-CoV-2 & SARS (LoD 893 copies/mL) in a pool of five samples, however this assay reported S5, S4 and S6 as positive in individual tests. VitaPCR SARS-CoV-2 Assay (LoD 1,000 copies/mL) reported S2, S5, S4, and S1 as positive in a pool of 10 samples, did not obtain a valid result for S7 (e.g., internal assay control failed), and reported a false negative result for S6. Ct values for samples testing positive in both a sample pool and in individual test were consistently lower in individual tests (Table 2).

Table 2:

Results and Ct values obtained in pooled and single testing of positive EQA samples.

  1. TCID50, 50% tissue culture infectious dose.

Discussion

Data presented here are not based on a verification or evaluation of assays but are snapshots of the performance of laboratories operating under routine procedures and their test systems in an EQA round. The majority of the assays in this EQA round met the WHO recommendation to detect at least 1,000 copies/mL of SARS-CoV-2 RNA (97.3%, 467 of 480 total assays from three samples). Many also detected SARS-CoV-2 in the test samples with 500 and 100 copies/mL (94.1%, 753/800 detected in five samples with ≥100 copies/mL) and some laboratories could detect an even lower viral load (50 copies/mL). However, the overall false negative ratio increased from ∼6 to ∼16 to ∼26% when virus concentration decreased from 500 to 100 to 50 copies/mL, respectively. The robust performance in this EQA round is not surprising, as in most cases they met the LoD as specified by the manufacturers – and some even exceed it. This underscores the importance of considering the manufacturer’s specifications on sensitivity when selecting assays and verifying them for intended use.

Assay-specific performance and comparison between assays is difficult to assess in an EQA, given the many combinations between assays with sample preparation, and the infrequent use of some products across laboratories. We restricted our analysis to assays that were reported by at least four participants in our EQA round, and to assays for which LoD information is clearly provided. Eight assays matched these criteria, and among these eight, we observed no evidence that a detection limit was systematically missed. On the contrary, the LoD specified by the manufacturers were mostly exceeded and the assays detected samples with a lower viral load as positive under routine use. For other assays and test systems that were infrequently represented in this EQA round, it is difficult to determine whether false negative results were more likely to be caused by operator error or lack of sensitivity. However, it is reasonable to assume that limited sensitivity may be at least partly responsible for false negative results, as we and others have observed that error rates are higher for samples with lower virus loads.

The usage of an assay for the analysis of sample pools is a special adaptation of the assay; and the performance with pooled material is not directly evaluated by manufacturers but could be inferred from other parameters. Therefore, the diagnostic laboratory must select and verify assays that are amenable to pooled samples and still achieve the desired sensitivity. In the data from the five assays that analyzed samples both pooled and individually, we saw differences in detection rates, as expected. Specifically, some assays were used to analyze pool sizes that, combined, were inappropriate to achieve the LoD set by the WHO. In other words, the LoD for a given assay should be multiplied by a factor equal to the number of samples in the pool to calculate the “pool-size-adjusted” sensitivity of the assay relative to the dilution of the RNA to be detected; in some cases this would raise the relative (per sample) sensitivity above the recommended sensitivity with an LoD of 1,000 copies/mL.

Awareness of the LoD when preparing samples is a known caveat to verifying assays for use in diagnostic laboratories, and we have noted errors that may be attributed to this in previous rounds of SARS-CoV-2 NAT detection EQA. Specifically, laboratories may save reagents by using lower volumes of material during extraction (and/or reconstituting extracted RNA in a disproportionately larger volume of buffer) or using half-reactions when performing RT-qPCR – both of which are practices that change the relative sensitivity of the setup of the analysis. In addition, some laboratories reported following extraction-free protocols using direct sampling into lysis buffer [8], thus diluting the resulting extract and raising the relative sensitivity. When evaluating results of EQA rounds using liquid samples, the dilution of the sample virus load by the lysis buffer and the change in the chemical composition of the buffer by the carrier solution of the samples must be taken into account. Such test systems, unless they have also been validated with liquid samples, should not be included in EQA schemes targeting LoD and using liquid samples.

Ultimately, the responsibility for the selection and use of SARS-CoV-2 assays lies with the laboratory. When qualifying assays, especially when examining sample pools, it is their responsibility to consider the intended use and to ensure compliance with WHO specifications. Reports of undetectable viral RNA should consistently include the LoD of the assay used, or the relative sensitivity if applicable. A large selection of high-quality test systems is available for this purpose.


Corresponding author: Irene Görzer, Center for Virology, Medical University of Vienna, Kinderspitalgasse 15, 1090 Vienna, Austria, E-mail:
Christoph Buchta and Jeremy V. Camp contributed equally to this work.

Acknowledgments

We gratefully acknowledge all laboratories that participated in this study and made special efforts to report more data than was required to participate in this EQA round.

  1. Research funding: None declared.

  2. Author contributions: Christoph Buchta: Conceptualized, conducted and analysed this EQA study, wrote the manuscript draft. Jeremy V. Camp: Conducted and analysed this EQA study, wrote, edited and critically reviewed the manuscript. Jovana Jovanovic: Analysed data and provided technical EQA support, critically reviewed manuscript. Elisabeth Puchhammer-Stöckl: Provided scientific advice, critically reviewed the manuscript. Robert Straßl: Provided scientific advice, critically reviewed the manuscript. Mathias M. Müller: Provided scientific advice, critically reviewed the manuscript. Andrea Griesmacher: Provided scientific advice, critically reviewed the manuscript. Stephan W. Aberle: Provided sample material, conceptualized, conducted and supervised this EQA study, provided scientific advice to the study, reviewed and edited the manuscript. Irene Görzer: Conceptualized, conducted and analysed this EQA study, wrote and edited the manuscript. All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Not applicable.

  5. Ethical approval: Not applicable.

References

1. Arnaout, R, Lee, RA, Lee, GR, Callahan, C, Cheng, A, Yen, CF, et al.. The limit of detection matters: the case for benchmarking severe acute respiratory syndrome coronavirus 2 testing. Clin Infect Dis 2021;73:e3042–6. https://doi.org/10.1093/cid/ciaa1382.Search in Google Scholar

2. WHO. Technical specification for selection of essential in vitro diagnostics for SARS-CoV-2, version 14 June 2021. Available from: https://apps.who.int/iris/rest/bitstreams/1351095/retrieve.Search in Google Scholar

3. Buchta, C, Camp, JV, Jovanovic, J, Radler, U, Puchhammer-Stöckl, E, Benka, B, et al.. A look at the precision, sensitivity and specificity of SARS-CoV-2 RT-PCR assays through a dedicated external quality assessment round. Clin Chem Lab Med 2021;60:e34–7. https://doi.org/10.1515/cclm-2021-1004.Search in Google Scholar

4. Pum, J. A practical guide to validation and verification of analytical methods in the clinical laboratory. Adv Clin Chem 2019;90:215–81. https://doi.org/10.1016/bs.acc.2019.01.006.Search in Google Scholar

5. Torres, I, Albert, E, Navarro, D. Pooling of nasopharyngeal swab specimens for SARS-CoV-2 detection by RT-PCR. J Med Virol 2020;92:2306–7. https://doi.org/10.1002/jmv.25971.Search in Google Scholar

6. Buchta, C, Müller, MM, Griesmacher, A. The importance of external quality assessment data in evaluating SARS-CoV-2 virus genome detection assays. Lancet Microbe 2022:e168. https://doi.org/10.1016/s2666-5247(22)00003-9.Search in Google Scholar

7. Görzer, I, Buchta, C, Chiba, P, Benka, B, Camp, JV, Holzmann, H, et al.. First results of a national external quality assessment scheme for the detection of SARS-CoV-2 genome sequences. J Clin Virol 2020;129:104537. https://doi.org/10.1016/j.jcv.2020.104537.Search in Google Scholar PubMed PubMed Central

8. Smyrlaki, I, Ekman, M, Lentini, A, Rufino de Sousa, N, Papanicolaou, N, Vondracek, M, et al.. Massive and rapid COVID-19 testing is feasible by extraction-free SARS-CoV-2 RT-PCR. Nat Commun 2020;11:4812. https://doi.org/10.1038/s41467-020-18611-5.Search in Google Scholar PubMed PubMed Central

Received: 2022-03-19
Accepted: 2022-05-16
Published Online: 2022-05-23
Published in Print: 2022-07-26

© 2022 Christoph Buchta et al., published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.

Scroll Up Arrow