Results of a SARS-CoV-2 virus genome detection external quality assessment round focusing on sensitivity of assays and pooling of samples

Objectives: Results of earlier external quality assessment (EQA) rounds suggested remarkable differences in the sensitivity of SARS-CoV PCR assays. Although the test systems are intended to detect SARS-CoV-2 in individual samples, screening is often applied to sample pools to in-crease efficiency and decrease costs. However, it is un-known to what extent these tests actually meet the manufacturer ’ s specifications for sensitivity and how they perform when testing sample pools. Methods: The sensitivity of assays in routine use was evaluated with a panel of positive samples in a round of a SARS-CoV-2 virus genome detection EQA scheme. The panel consisted of samples at or near the lower limit of detection ( “ weakly positive ” ). Laboratories that routinely test sample pools were asked to also analyze the pooled EQA samples according to their usual pool size and dilution method. Results: All participants could detect a highly positive patient-derived sample (>10 6 copies/mL). Most (96%) of the test systems could detect at least 1,000 copies/mL, meeting the minimum acceptable benchmark, and many (94%) detected the vRNA in a sample with lower concentration (500 copies/mL). The false negative ratio increased to 16 and 26% for samples with 100 and 50 copies/mL, respectively. Conclusions: The performance of most assays met or exceeded their specification on sensitivity. If assays are to be used to analyze sample pools, the sensitivity of the assay and the number of pooled samples must be balanced.


Introduction
An assay's sensitivity affects its clinical performance in SARS-CoV-2 testing, and if too low, it can lead to false-negative results that are not conducive to managing the pandemic [1]. Therefore, the World Health Organization (WHO) describes a limit of detection (LoD) of 1,000 copies/mL as "acceptable", but 100 copies/mL as "desirable" [2]. Results from previous rounds of the SARS-CoV-2 virus genome detection external quality assessment (EQA) scheme of the Austrian Association for Quality Assurance and Standardization of Medical and Diagnostic Tests (ÖQUASTA) and the Center for Virology at the Medical University Vienna (the national reference laboratory for respiratory viruses) suggested remarkable differences in the sensitivities of assays in routine use [3]. Determination of the LoD is part of the validation of an assay by the manufacturer and the verification by the user [4]. If sample pools are analyzed, the sensitivity of the assay and the size of the pool must be balanced in order to reliably identify SARS-CoV-2 RNA from one positive sample in the pool [5]. After assay validation and verification procedures, the performance of diagnostics in routine use should be closely monitored. It is advisable to include data from external quality assessment schemes as they are reports from the "real life" performance apart from validation or verification settings [6]. We therefore dedicated an EQA round in early 2022 to the sensitivity of SARS-CoV-2 nucleic acid test (NAT) assays in routine use.

Materials and methods
A panel of seven samples was designed to compare the performance of SARS-CoV-2 NAT assays in routine laboratory use against the LoD Christoph Buchta and Jeremy V. Camp contributed equally to this work. recommended by WHO. Preparation and shipment of samples (including quality control assessment under storage and shipping conditions) were performed as previously described [7]. Instructions for participants to perform the analysis and to report the results, the collection of results and their independent assessment were performed by the EQA provider, and feedback was provided to the participants as previously described [7].
In addition to the data that were to be provided by participants in previous rounds, laboratories routinely screening sample pools were asked in this round to dilute the EQA samples with negative samples or buffer solution according to their standard pool size. These pools should then be analyzed and the samples should subsequently be individually tested.
One positive sample (S2) was a clinical specimen from a swab positive for SARS-CoV-2 variant omicron (BA.2), yielding Ct 23.8, corresponding to a virus load of 9.3 × 10 6 copies/mL, as characterized by the reference laboratory. One other sample (S3) was negative for SARS-CoV-2 RNA. Five positive samples were prepared by making dilutions of the AccuPlex™ SARS-CoV-2 Reference Material (Material Number 0505-0126, SeraCare, Milford, MA, USA) to yield the following calculated concentrations: ∼50 copies/mL (1:100 dilution, S6); ∼100 copies/mL (1:50 dilution, S1); ∼500 copies/mL (1:10 dilution, S4); and ∼1,000 copies/mL (1:5 dilution, S5 and S7). Two samples with ∼1,000 copies/mL were provided to allow discrimination of false negative results caused by random error from those caused by limited sensitivity of assays. Human cell culture material was added to all samples to provide human housekeeping genes for assays that test and require this for a valid result. The requirement to pass this EQA round was to obtain correct results for the negative sample, the highly positive clinical sample and for the positive samples with ∼1,000 and ∼500 copies/mL SARS-CoV-2.

Results
A total of 133 laboratories participated in this round and used 160 assays in total, among which were 72 different combinations of 45 extraction reagents on 39 extraction platforms, and 48 detection reagents on 35 detection platforms. A total of 52 (32.5%) assays were represented by only one to three individual participant reports in this EQA round. Taking into account that reporting of correct results for S2, S3, S4, S5, and S7 was required to pass, the success rate in this round was 149/160 (92.5%). In descending order of the virus load in the EQA samples, S2 (∼9.3E+6 copies/ mL) was reported positive by every (100%) assay; S5 and S7 (∼1,000 copies/mL) were incorrectly reported negative by six (3.8%) and seven (4.4%) assays, respectively; S4 (∼500 copies/mL) was reported negative by 9 (5.6%), S1 (∼100 copies/mL) by 25 (15.6%), and S6 (∼50 copies/mL) by 41 (25.6%). Sample S3 was correctly reported negative by 159/160 (99.4%) assays. A total of 112 assays reported correct results for all seven samples (Table 1).
Five laboratories routinely screened pools of patient samples, in pool sizes of five to ten patients, and reported results from individual test samples as if they were pooled. Additionally, one laboratory reported results only from pooled testing. In mock pools and in follow-up individual testing, S2 (∼9.3 × 10 6 copies/mL) and S7 (∼1,000 copies/mL) were reported positive by all five assays. Among the false negative test results from pooled testing, ANDiS FAST SARS-CoV-2 RT-qPCR Detection Kit (3D Biomedicine Science & Technology Co. Ltd, Shanghai, China) (LoD 5 copies/reaction) missed S1 (∼100 copies/mL) and S6 (∼50 copies/mL) in a pool of eight samples but detected both samples as positive in individual tests. The Aptima SARS-CoV-2 Assay (Hologic, Inc., California, USA) (LoD 0.001 TCID50/mL) detected all samples as positive in both individual testing and in a pool of five samples. The cobas SARS-CoV-2 assay (Roche Molecular Systems, Inc. New Jersey, USA) (LoD 46 copies/mL) detected all samples as positive in individual tests, but in a pool of 10 samples, S6 was missed. S1 and S6 were missed by Molaccu Covid-19 detection kit (Zybio, Chongqing, China) (LoD ≤500 copies/mL) in a pool of six samples as well as in individual tests. S5 (∼1,000 copies/mL), S4 (∼500 copies/mL), S1, and S6 were missed by ViroReal Kit SARS-CoV-2 & SARS (LoD 893 copies/mL) in a pool of five samples, however this assay reported S5, S4 and S6 as positive in individual tests. VitaPCR SARS-CoV-2 Assay (LoD 1,000 copies/mL) reported S2, S5, S4, and S1 as positive in a pool of 10 samples, did not obtain a valid result for S7 (e.g., internal assay control failed), and reported a false negative result for S6. Ct values for samples testing positive in both a sample pool and in individual test were consistently lower in individual tests (Table 2).

Discussion
Data presented here are not based on a verification or evaluation of assays but are snapshots of the performance of laboratories operating under routine procedures and their test systems in an EQA round. The majority of the assays in this EQA round met the WHO recommendation to detect at least 1,000 copies/mL of SARS-CoV-2 RNA (97.3%, 467 of 480 total assays from three samples). Many also detected SARS-CoV-2 in the test samples with 500 and 100 copies/mL (94.1%, 753/800 detected in five samples with ≥100 copies/mL) and some laboratories could detect an even lower viral load (50 copies/mL). However, the overall false negative ratio increased from ∼6 to ∼16 to ∼26% when virus concentration decreased from 500 to 100 to 50 copies/mL, respectively. The robust performance in this EQA round is not surprising, as in most cases they met the LoD as specified by the manufacturersand some even exceed it. This underscores the importance of considering the manufacturer's specifications on sensitivity when selecting assays and verifying them for intended use.
Assay-specific performance and comparison between assays is difficult to assess in an EQA, given the many combinations between assays with sample preparation, and the infrequent use of some products across laboratories. We restricted our analysis to assays that were reported by at least four participants in our EQA round, and to assays for which LoD information is clearly provided. Eight assays matched these criteria, and among these eight, we observed no evidence that a detection limit was systematically missed. On the contrary, the LoD specified by the manufacturers were mostly exceeded and the assays detected samples with a lower viral load as positive under routine use. For other assays and test systems that were  infrequently represented in this EQA round, it is difficult to determine whether false negative results were more likely to be caused by operator error or lack of sensitivity. However, it is reasonable to assume that limited sensitivity may be at least partly responsible for false negative results, as we and others have observed that error rates are higher for samples with lower virus loads. The usage of an assay for the analysis of sample pools is a special adaptation of the assay; and the performance with pooled material is not directly evaluated by manufacturers but could be inferred from other parameters. Therefore, the diagnostic laboratory must select and verify assays that are amenable to pooled samples and still achieve the desired sensitivity. In the data from the five assays that analyzed samples both pooled and individually, we saw differences in detection rates, as expected. Specifically, some assays were used to analyze pool sizes that, combined, were inappropriate to achieve the LoD set by the WHO. In other words, the LoD for a given assay should be multiplied by a factor equal to the number of samples in the pool to calculate the "pool-size-adjusted" sensitivity of the assay relative to the dilution of the RNA to be detected; in some cases this would raise the relative (per sample) sensitivity above the recommended sensitivity with an LoD of 1,000 copies/mL.
Awareness of the LoD when preparing samples is a known caveat to verifying assays for use in diagnostic laboratories, and we have noted errors that may be attributed to this in previous rounds of SARS-CoV-2 NAT detection EQA. Specifically, laboratories may save reagents by using lower volumes of material during extraction (and/or reconstituting extracted RNA in a disproportionately larger volume of buffer) or using half-reactions when performing RT-qPCRboth of which are practices that change the relative sensitivity of the setup of the analysis. In addition, some laboratories reported following extraction-free protocols using direct sampling into lysis buffer [8], thus diluting the resulting extract and raising the relative sensitivity. When evaluating results of EQA rounds using liquid samples, the dilution of the sample virus load by the lysis buffer and the change in the chemical composition of the buffer by the carrier solution of the samples must be taken into account. Such test systems, unless they have also been validated with liquid samples, should not be included in EQA schemes targeting LoD and using liquid samples.
Ultimately, the responsibility for the selection and use of SARS-CoV-2 assays lies with the laboratory. When qualifying assays, especially when examining sample pools, it is their responsibility to consider the intended use and to ensure compliance with WHO specifications. Reports of undetectable viral RNA should consistently include the LoD of the assay used, or the relative sensitivity if applicable. A large selection of high-quality test systems is available for this purpose. manuscript draft. Jeremy V. Camp: Conducted and analysed this EQA study, wrote, edited and critically reviewed the manuscript. Jovana Jovanovic: Analysed data and provided technical EQA support, critically reviewed manuscript. Elisabeth Puchhammer-Stöckl: Provided scientific advice, critically reviewed the manuscript. Robert Straßl: Provided scientific advice, critically reviewed the manuscript. Mathias M. Müller: Provided scientific advice, critically reviewed the manuscript. Andrea Griesmacher: Provided scientific advice, Conceptualized, conducted and analysed this EQA study, wrote and edited the manuscript. All authors have accepted responsibility for the entire content of this manuscript and approved its submission.