Skip to content
BY 4.0 license Open Access Published by De Gruyter October 20, 2021

A look at the precision, sensitivity and specificity of SARS-CoV-2 RT-PCR assays through a dedicated external quality assessment round

Christoph Buchta, Jeremy V. Camp, Jovana Jovanovic, Ulla Radler, Elisabeth Puchhammer-Stöckl, Bernhard Benka, Wolfgang Huf ORCID logo, Mathias M. Müller, Andrea Griesmacher, Stephan W. Aberle and Irene Görzer

To the Editor,

In an external quality assessment (EQA) round for SARS-CoV-2 virus genome detection in early 2021, we found first indications of the precision of individual SARS-CoV-2 RT-PCR assays in the linearity of the Ct values obtained for three samples from a dilution series (1:1, 1:10, 1:100) [1]. To further investigate the reproducibility of the Ct values and thus the precision of the participating assays, two identical samples were employed in the latest round of this EQA scheme of the Austrian Association for Quality Assurance and Standardization (ÖQUASTA) and the Center for Virology of the Medical University Vienna.

On August 16, 2021, a panel of six samples was sent to participant laboratories. Production, testing, shipping of samples and instructions for participant laboratories followed the same procedures as in the previous rounds [2]. The panel consisted of two SARS-CoV-2 negative and four positive samples, of which two were identical (S2 and S6; B.1.617.2, Delta variant, Ct∼30.6). The third positive sample (S3) was B.1.1.7, Alpha variant, Ct∼32.2, and the fourth (S4) was B.1.617.2, Delta variant, Ct∼32.8. One negative sample (S5) consisted of virus genome free NaCl, the second SARS-CoV-2 negative sample (S1) contained a recent clinical sample of human betacoronavirus OC43 (HCoV-OC43, Ct∼29.9) to assess potential cross-reactivity of assays. All samples were spiked with human cells to give positive results for housekeeping genes for internal control in RT-PCR. A total of 114 laboratories reported results before August 31, 2021. They employed a total of 139 assays, including 53 different, for detection of SARS-CoV-2 genomic RNA.

Among the two identical samples, 136/139 (98%) assays reported S2 as positive, 1/139 (1%) as negative, and 2/139 (1%) as not determinable; S6 was reported positive by 134/139 (97%) assays, negative by 2/139 (1%), and not determinable by 3/129 (2%). S3 (Ct∼32.2) was reported positive by 138/139 (99%) assays and negative by one (1%). S4 (Ct∼32.8) was reported positive by 131/139 (94%), negative by four (3%), and not determinable by four (3%) assays. Among them was one assay that reported all four positive samples as negative, the other four false negative results were reported from four different assays. No false positive results were reported for S1 and S5, however 7/139 (5%) assays found S1 and S5 not determinable (data not shown). The rates of false negative results depending on the Ct values of samples correspond to earlier rounds of this EQA scheme [1], [2], [3]. Of the total of 109 different participating assays, 70 did not show a single false negative result in any of the four rounds. The rate of false negative in all reported results ranged from 0/105 to 4/4 (Figure 1, Supplementary Material). We did not observe any difference in the rates of false negative results between the two variants of the virus used in this round. While the B.1.1.7, Alpha variant, sample (S3) was reported negative by one (1%) assay, the B.1.617.2, Delta variant, samples (S2, S4, S6) were reported negative by one (1%), four (3%), and two (1%) assays. These small differences in the rates of false negative results appear to be more due to the different Ct values of the samples, namely about 1% for samples with Ct∼32.2 (S3) and Ct∼30.6 (S2=S6), and 3% for the sample with the highest Ct∼32.8 (S4).

Figure 1: 
Percentages of false negative results increased with increasing Ct values of 15 positive samples in four rounds of an EQA scheme to detect SARS-CoV-2 virus genome.
Percentages are labeled beside plotted diamonds if they deviate from 0%. Data came from four rounds of an EQA scheme (blue, May 2020; gold, October 2020; green, February 2021; and red, August 2021).

Figure 1:

Percentages of false negative results increased with increasing Ct values of 15 positive samples in four rounds of an EQA scheme to detect SARS-CoV-2 virus genome.

Percentages are labeled beside plotted diamonds if they deviate from 0%. Data came from four rounds of an EQA scheme (blue, May 2020; gold, October 2020; green, February 2021; and red, August 2021).

For the paired samples (Ct values for both samples S2 and S6), 123 assays were reported targeting one (n=47), two (n=70), three (n=5) or four (n=1) gene targets. Mean Ct value reported for the identical samples S2 and S6 was 33.2 (range 23.6–39.4; SD 3.2); and sample-specific Ct values were 33.5 (range 22.8–40.8; SD 3.2) for S2 and 32.7 (range 23.1–43.8; SD 3.3) for S6. The differences in Ct values reported for each individual gene target were calculated per participant as [S6 Ct value − S2 Ct value] and ranged from −5.3 to +6.5 cycles. Ct value discrepancy was not associated with a specific RT-PCR target region. The mean values of the differences between both results for all gene targets reported by individual assays ranged from −4.7 to +4.7 cycles (Figure 2). Seven assays reported a Ct value for one of the target genes they used for only one of samples S2 and S6 and none for the second. The reported Ct values were 35.7 mean, in detail 32.6, 35.0, 37.0, 37.5 for S2, and 34.7, 36.6, 36.7 for S6. These single results are not included in the data but shown and highlighted in Figure 2.

Figure 2: 
Differences of Ct values reported for two identical samples of SARS-CoV-2.
The top box-and-whiskers plot shows global median and quartiles of individual Ct values for two paired samples. The bottom box-and-whiskers plot shows median and quartiles of the differences in Ct values comparing the same laboratory. The dots are colored according to three groups based on the differences in Ct values – identical results (red), similar results (green) and deviating results (dark blue) – and the means of each group are displayed as colored vertical lines on the top figure. Seven results that reported a Ct value for only one of the paired samples are shown in light blue (dots and vertical line). The assays that fall into each of the three groups are listed in the table.

Figure 2:

Differences of Ct values reported for two identical samples of SARS-CoV-2.

The top box-and-whiskers plot shows global median and quartiles of individual Ct values for two paired samples. The bottom box-and-whiskers plot shows median and quartiles of the differences in Ct values comparing the same laboratory. The dots are colored according to three groups based on the differences in Ct values – identical results (red), similar results (green) and deviating results (dark blue) – and the means of each group are displayed as colored vertical lines on the top figure. Seven results that reported a Ct value for only one of the paired samples are shown in light blue (dots and vertical line). The assays that fall into each of the three groups are listed in the table.

To better visualize the differences between the paired samples and detect assay-specific deviations, we grouped the results according to the differences in Ct values: (i) identical results (difference <1 cycle, 77/123, 63%); (ii) similar results (difference >1 and ≤3 cycles, 42/123, 34%); and (iii) deviating results (difference >3 cycles, 4/123, 3%). Means and ranges of Ct values in each group were 32.9 (22.8–40.6) for group 1, 33.8 (24.1–40.8) for group 2, and 36.8 (31.7–43.8) for group 3. Participants with identical results (group 1) reported using 37 different assays, those with similar results (group 2) reported 26 assays, and those with deviating results (group 3) reported four assays. Ten assays are represented in both groups 1 and 2, three are represented in groups 1 and 3, and one assay is represented in all three groups (Figure 2).

The results presented here indicate good reproducibility and thus high precision of the Ct values of most of the RT-PCR assays represented in this EQA round. As we previously noted, the detection rate of SARS-CoV-2 decreases with increasing Ct values [1], [2], [3]. Additionally, we observed that higher Ct values are more likely to be associated with greater differences between the two measurement results and thus with lower precision. By including paired samples and grouping the results based on the deviation in Ct values, we noted that only a few of the assays were categorized in more than one performance group (e.g., giving both identical results and deviating results). From this, we conclude that overall precision seems to rely on the performance of the individual assays and less on their operation. The fact that no false positive results were reported in this round confirms our earlier findings of low rates of false positive results in SARS-CoV-2 viral genome detection [3]. The high specificity of the test systems included in this EQA round was indicated by no false positive results obtained in the sample with the human betacoronavirus OC-43.

In summary, the performance of most SARS-CoV-2 virus genome detection assays in this EQA round is good, however some appear to tend to report comparatively high Ct values and thus less accurate and consequently false negative results. The use of such assays should be subject to risk analysis and assessment by their operators. The results presented here underline once again the great importance of EQA schemes for monitoring the performance of laboratories as well as of test systems. This applies in particular, if the infectious agent to be detected is a rapidly changing virus, the newly emerging variants of which have different clinical significance and a manifold of different test systems are in use.


Corresponding author: Irene Görzer, Center for Virology, Medical University of Vienna, Kinderspitalgasse 15, 1090 Vienna, Austria, E-mail:
Christoph Buchta and Jeremy V. Camp contributed equally to this work.

Acknowledgments

We gratefully acknowledge all laboratories that participated in this study and made special efforts to report more data than was required to participate in this EQA round.

  1. Research funding: None declared.

  2. Author contributions: Christoph Buchta: Conceptualized, conducted and analysed this EQA study, wrote and edited the manuscript draft. Jeremy V. Camp: Conducted and analysed this EQA study, wrote, edited and critically reviewed the manuscript. Jovana Jovanovic: Analysed data and provided technical EQA support, critically reviewed manuscript. Ulla Radler: Analysed data and provided technical EQA support, critically reviewed the manuscript. Elisabeth Puchhammer-Stöckl: Provided scientific advice, critically reviewed the manuscript. Bernhard Benka: Provided public health advice, critically reviewed and edited the manuscript. Wolfgang Huf: Performed statistical analysis and visualisation of data, critically reviewed the manuscript. Mathias M. Müller: Provided scientific advice, critically reviewed the manuscript. Andrea Griesmacher: Provided scientific advice, critically reviewed the manuscript. Stephan W. Aberle: Provided sample material, conceptualized, conducted and supervised this EQA study, provided scientific advice to the study, reviewed and edited the manuscript. Irene Görzer: Conceptualized, conducted and analysed this EQA study, wrote and edited the manuscript. All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Not applicable.

  5. Ethical approval: Not applicable.

References

1. Buchta, C, Camp, JV, Jovanovic, J, Chiba, P, Puchhammer-Stöckl, E, Mayerhofer, M, et al.. The versatility of external quality assessment for the surveillance of laboratory and in vitro diagnostic performance: SARS-CoV-2 viral genome detection in Austria. Clin Chem Lab Med 2021;59:1735–44. https://doi.org/10.1515/cclm-2021-0604.Search in Google Scholar

2. Buchta, C, Görzer, I, Chiba, P, Camp, JV, Holzmann, H, Puchhammer-Stöckl, E, et al.. Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR. Clin Chem Lab Med 2020;59:987–94. https://doi.org/10.1515/cclm-2020-1602.Search in Google Scholar

3. Görzer, I, Buchta, C, Chiba, P, Benka, B, Camp, JV, Holzmann, H, et al.. First results of a national external quality assessment scheme for the detection of SARS-CoV-2 genome sequences. J Clin Virol 2020;129:104537. https://doi.org/10.1016/j.jcv.2020.104537.Search in Google Scholar


Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2021-1004).


Received: 2021-09-12
Accepted: 2021-10-10
Published Online: 2021-10-20
Published in Print: 2022-01-27

© 2021 Christoph Buchta et al., published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.

Scroll Up Arrow