Evaluation of three fully-automated SARS-CoV-2 antibody assays

Objectives Serological assays for detection of SARS-CoV-2 antibodies are increasingly used during the COVID-19 pandemic caused by the SARS-Coronavirus-2. Here we evaluated the analytical and clinical performance of three commercially available SARS-CoV-2 antibody assays. Methods A total of 186 samples from 58 patients with PCR-confirmed COVID-19 infection were measured using SARS-CoV-2 antibody assays by Siemens Healthineers, Roche Diagnostics and Euroimmun. Additionally, 123 control samples, including samples collected before December 2019 and samples with potential cross-reactive antibodies were analyzed. Diagnostic specificity, sensitivity, agreement between assays and ROC curve-derived optimized thresholds were determined. Furthermore, intra- and inter-assay precision and the potential impact of interfering substances were investigated. Results SARS-CoV-2 antibody assays by Siemens and Roche showed 100% specificity. The Euroimmun assay had 98 and 100% specificity, when borderline results are considered as positive or negative, respectively. Diagnostic sensitivity for samples collected ≥14 days after PCR-positivity was 97.0, 89.4 and 95.5% using the Siemens, Roche and Euroimmun assay, respectively. Sensitivity of the Roche assay can be increased using an optimized cut-off index (0.095). However, a simultaneous decrease in specificity (98.4%) was observed. Siemens showed 95.8 and 95.5% overall agreement with results of Euroimmun and Roche assay, respectively. Euroimmun and Roche assay exhibited 92.6% overall agreement. Discordant results were observed in three COVID-19 patients and in one COVID-19 patient none of the investigated assays detected antibodies. Conclusions The investigated assays were highly specific and sensitive in detecting SARS-CoV-2 antibodies in samples obtained ≥14 days after PCR-confirmed infection. Discordant results need to be investigated in further studies.


Introduction
A new disease, known as coronavirus disease 2019 (COVID-19) caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), emerged in the region of Wuhan (China) in December 2019 [1,2]. The virus spread subsequently all over the world and the WHO officially declared the novel SARS-CoV-2 infection a pandemic. An accurate and prompt detection of SARS-CoV-2 is essential not only for the diagnosis of infected patients, but also for the establishment of infection control measures and prevention of virus spread [3,4]. Real-time reverse transcription polymerase chain reaction (RT-PCR) using respiratory samples obtained with naso-or oropharyngeal swabs are considered as the reference method for screening and diagnosis of acute SARS-CoV-2 infection [3]. However, the sensitivity of this method may vary depending on time of sampling, the quality and origin of the sample and the viral load [5,6]. In contrast, serologic testing of anti-SARS-CoV-2 antibodies (IgG, IgM and IgA) can be easily performed by using venous blood samples. SARS-CoV-2 antibody assays are proposed to play an important role in diagnosis of COVID-19 convalescents and in patients with clinical symptoms suggestive for SARS-CoV-2 infection but with negative or not available RT-PCR results [4]. Furthermore, antibody assays may help to understand the epidemiology of SARS-CoV-2 infections, support the selection of convalescent plasma for therapeutic treatments and potentially to evaluate the efficiency of future vaccines [7,8].
Numerous SARS-CoV-2 antibody assays have been approved including lateral flow tests, enzyme-linked immunosorbent assays (ELISA) and fully-automated electrochemiluminescent (ECLIA) or chemiluminescent immunoassays (CLIA). Due to the increasing numbers of these serological tests, the lack of comparative data in combination with the high demand on serological assays it is essential for laboratory professionals to carefully evaluate SARS-CoV-2 antibody assays. Therefore, the aim of this study was to compare the clinical and analytical performance of three commercially available, fully-automated ELISA-, CLIA-and ECLIA-based SARS-CoV-2 antibody assays. To date, this is the first evaluation of the Siemens SARS-CoV-2 antibody assay.

Study
The study was conducted as part of the diagnostic evaluation of SARS-CoV-2 serologic testing for routine use at the Institute for Clinical Chemistry and Pathobiochemistry at the University Hospital Tübingen according to the declaration of Helsinki of 1964 and its later amendments in accordance with the local Ethics Committee of the University of Tübingen. Routine blood samples (n=186) of hospitalized COVID-19 patients (n=58) were used for serial antibody measurements. COVID-19 diagnosis was based on detection of SARS-CoV-2-RNA in oro-and/or nasopharyngeal swab by RT-PCR. Median time between positive PCR result and blood sample collection was 19 days (interquartile range: 12-29 days). COVID-19 negative control samples were all obtained before the beginning of the pandemic and comprises intensive care patients (n=88). In addition, potential cross-reactive antibodies (n=35) were investigated using samples from patients with laboratory confirmed acute infections with influenza A virus (n=5), human respiratory syncytial virus (n=1) and common cold coronaviruses (NL63: n=1; HKU-1 + NL63: n=1; NL63 + 229E: n=1). Furthermore, samples with IgM antibodies against human cytomegalovirus (n=5) and varicella zoster virus (n=2), samples from patients with respiratory symptoms not suspicious of COVID-19 disease (n=11), samples containing antibodies (n=6) against chlamydia pneumoniae (IgG, IgA and/or IgM) or candida albicans (IgG and/or IgA) and samples positive for rheumatoid factor (n=2) were included in the study.

Assessment of analytical performance
Intra-and inter-assay precision was determined using quality control samples, if provided by the manufacturer, or antibody positive and negative patient samples. Intra-and inter-assay precision was determined from serial measurements (n=10) and measurements of quintuplicates during five consecutive days according to the Clinical and Laboratory Standards Institute (CLSI) EP15-A3 protocol [9]. For interference testing hemolytic (n=6), icteric (n=6) and lipemic (n=6) samples were spiked with plasma from highly SARS-CoV-2 antibody positive samples. Concentrations of plasma hemoglobin, triglycerides and total bilirubin were determined using photometric and enzymatic methods on a Dimension Vista and ADVIA Clinical chemistry XPT, respectively. Furthermore, potential interference of biotin was assessed using SARS-CoV-2 antibody positive and negative samples that were spiked with biotin (final concentrations of biotin: 1,000, 1,500 and 2,000 ng/mL).

Laboratory measurements
SARS-CoV-2 specific antibodies were detected according to the manufacturers' instructions on three different automated platforms.

Statistical analyses
Results of antibody measurements were evaluated according to the manufacturers' cut-off indices or ratios as positive or negative for the Roche and Siemens assays and as positive, borderline or negative for the Euroimmun assay. Diagnostic sensitivity and specificity were calculated under the following assumption: all samples obtained prior to the onset of the pandemic were considered as true negative. Positive antibody results from patients with a PCR-confirmed SARS-CoV-2 infection in their history were considered as true positive. Concordance analyses (Cohen's Kappa) and percent agreement (overall, positive/negative) were performed to compare results of each antibody assay. Receiver operating characteristic (ROC) curve analysis and Youden index were used to identify optimized thresholds (cut-off indices). Analyses were performed using JMP 14 software (SAS Institute, Cary, United States) and Analyse-it software for Microsoft Excel 5.40 (Analyse-it Software, Ltd., Leeds, UK).

Results
Specificity, sensitivity and agreement of the three serological assays To evaluate the diagnostic specificity of SARS-CoV-2 antibody assays we used control samples of patients collected before December 2019, which were considered true negative. The specificity of the Siemens and Roche assays was 100% (Table 2). Moreover, analysis of samples from patients with other acute viral and bacterial or fungal infections showed no cross-reactivity using these two assays. The Euroimmun assay showed 98% specificity using the same samples, when borderline results were considered positive. However, when borderline results were considered negative, the Euroimmun assay exhibited 100% specificity as well. False positive results were not detected by the Euroimmun assay. Borderline results were observed for samples from two patients with acute influenza A virus infection and from one patient with infection of coronaviruses HKU-1 and NL63. Diagnostic sensitivity was determined using samples from patients with RT-PCR confirmed COVID-19 disease at different time points. Sensitivity of the Siemens assay was 56.5% in the first six days after PCR positive SARS-CoV-2 result, 90.3% at days 7-13 and 97.0% at day 14 or later ( Table 3). The Roche assay revealed a diagnostic sensitivity of 65.2% in the first six days after PCR positive SARS-CoV-2 result, 90.3% at days 7-13 and 89.4% at day 14 or later. Diagnostic sensitivity of the Euroimmun assay was 47.8% in the first six days, 83.9% at days 7-13 and 95.5% at day 14 or later, if borderline results were considered negative. Considering borderline results as positive, diagnostic sensitivity in the first six days changed to 56.5%. Sensitivity for the other sampling time points remained unchanged. No further borderline results were detected by the Euroimmun assay.

Siemens Roche
Euroimmun IgG    Figure 1). Based on ROC curves optimized cut-off indices were derived. We found optimal decision thresholds for cut-off indices of >0.75, >0.095 and >0.958 for the Siemens, Roche and Euroimmun assay, respectively (Table 5). Applying these thresholds the sensitivity at day ≥14 after PCR-positivity of the Roche assay improved (97.0 vs. 89.4%). However, at the same time, the specificity decreased to 98.4%, because two samples collected before December 2019 were identified as false positive (COI 0.530 and 0.500). The Siemens assay showed slightly improved sensitivity in the first 6 days after PCR-positivity (60.9 vs. 56.5%). Sensitivity for samples collected >6 days and specificity of the Siemens assay were unaffected. Using the single cut-off index for the Euroimmun assay sensitivities remained unchanged but specificity improved to 100%.

Intra-and inter-assay precision and assessment of analytical interferences
Determination of intra-and inter-assay precision revealed coefficient of variations between 2.6-11.3 and 4.9-17.1% for the investigated assays using quality control samples provided by the manufacturers or antibody positive and negative patient samples (Supplemental Table 1 and 2). Potential analytical interferences by increased plasma hemoglobin (hemolysis), total bilirubin (icterus) or triglyceride (lipemia) concentrations were negligible (Supplemental Table 3). Biotin spiking (final concentrations ≤2000 ng/mL) of plasma samples of SARS-CoV-2 antibody negative and positive samples showed no significant impact on antibody results using all three serological assays.

Discussion
The present study evaluated and compared three SARS-CoV-2 antibody assays by Siemens Healthineers, Roche Diagnostics and Euroimmun which were run on fully-automated platforms. To date, this is the first evaluation of the Siemens SARS-CoV-2 antibody assay. All three assays demonstrated high diagnostic specificity. Only the Euroimmun assay detected three false borderline results. The Siemens assay was found to be slightly more sensitive than the Roche and Euroimmun assays but none of these assays was sufficiently sensitive to safely detect antibodies <14 days after PCR-confirmed SARS-CoV-2 infection.   High diagnostic specificity is crucial to reduce the number of false positive results in serological testing. The Roche and Siemens assays demonstrated 100% specificity in samples collected before December 2019 when the virus was not present in Germany and in samples with potential cross-reactive antibodies from acute viral and bacterial or fungal infections. Only in the case of the Euroimmun assay three false borderline results were observed, which were found in two samples with influenza A virus and in one sample with other coronaviruses (HKU-1 and NL63). The specificities of the Roche and Euroimmun assays are similar to those given by the manufacturers and other recent studies [10][11][12][13][14]. Some reports showed slightly reduced specificities (>96%) for the Euroimmun assay [15,16]. Specificity of the Roche assay was demonstrated as >98-100% in three other reports [10][11][12]. However, specificities between studies are difficult to compare due to different numbers and origins of samples used for the evaluations. Although serological assays exhibit high diagnostic specificity and sensitivity (>99%), the positive predictive value of a single test can be markedly reduced in populations with low prevalence of SARS-CoV-2 [7]. This should be taken into account when evaluating antibody screening studies.
Regarding the diagnostic sensitivity, results of all three assays were similar in the early phase of the SARS-CoV-2 infection (<7 days after PCR-positivity). Sensitivities varied between 47.8 and 65.2% and were not sufficient for detection of acute SARS-CoV-2 infections. This is in accordance with studies using comparative commercially available SARS-CoV-2 antibody assays showing insufficient diagnostic sensitivity in the early phase (<7 days) after symptom onset [10][11][12][13][14][16][17][18]. With progression of disease the sensitivities of all three assays markedly increased. Considering samples collected >14 days after PCR-positivity, the Siemens assay showed higher sensitivity (97.0%) in comparison to the Euroimmun (95.5%) and the Roche (89.4%) assay. These results contradict the information provided by the manufacturers claiming 100% sensitivity for samples obtained ≥14 days after PCR-positivity. Other studies reported sensitivities between 89 and 100% using the Euroimmun and Roche assay for samples collected >14 days after symptoms onset [10-12, 15, 16]. Euroimmun additionally offers an Anti-SARS-CoV-2-ELISA for the detection of IgA antibodies which may increase the sensitivity in the early phase according to the manufacturer and to recent publication [19]. However, the performance analysis of this assay was not in the scope of the study.
Although the majority of patients showed comparable and plausible results in the investigated assays, exceptional cases were observed. One PCR positive patient was antibody negative (confirmed in follow-up measurements) when tested with all three assays and remained negative even when applying the optimized cut-off indices presented in this study. Moreover, in two patients with positive PCR the Roche or Euroimmun assay failed to detect antibodies. Using the optimized cut-off indices, results by the Roche assay become positive whereas results by the Euroimmun assay remain negative. The use of optimized thresholds was recently highlighted by some other studies [12,15,18]. In accordance with our results, a lower cut-off index was proposed for the Roche assay and also for other SARS-CoV-2 antibody assays optimized cut-off indices have been proposed. In our study, the optimized cut-off indices for the Siemens and Euroimmun assay only slightly improved the performance of these assays. Larger studies are needed to reevaluate modified thresholds.
The main explanation for discordant results may be the use of different antigen targets and detection of different antibody classes by the assays used in our study. The Roche assay targets the nucleocapsid of the SARS-CoV-2 virus and detects total antibodies (including IgG), the Euroimmun assay targets the S1 domain of the spike protein and detects IgG antibodies, whereas the Siemens assay detects total antibodies (IgG and IgM) against the receptor binding domain (RBD) of the S1 protein. Therefore, discordant results are likely to be expected. Recent data suggest that the RBD domain, besides other parts of the S protein, is a potent target for neutralizing antibodies. Thus, these may be promising target structures for the design of therapeutic antibodies and future vaccines [20][21][22]. Although neutralizing antibodies have been described in COVID-19 convalescents, there are no definitive data available regarding antibody results by commercially available antibody assays and protective immunity [3,23]. Moreover it is interesting to note, that in one further COVID-19 patient the antibody ratio decreased over time using the Roche assay and became negative in samples >28 days using the manufacturer's cut-off index whereas the Siemens and Euroimmun antibody ratio did not decrease. Decreased antibody ratio with time was recently demonstrated in a patient with chronic lymphatic leukemia and also in another evaluation decreased antibody ratio with time was shown in several patients using the Roche assay [11,24]. However, no pre-existing conditions were known for our patient. Since Roche declares that mainly highaffinity antibodies are detected in their assay, it is speculated that this may explain the decrease in antibody ratio with time [11]. Important to note, results of the Roche assay remain positive when applying the optimized cut-off index presented in the study. Finally, to elucidate whether the decrease in antibody ratio is an analytical issue or whether special parts of antibodies may disappear after time needs to be elucidated in long-term follow-up antibody studies.
A limitation of the study includes that sensitivity of the investigated assays was based on time from PCR-positivity and therefore may be overestimated for the earlier time points due to potentially delayed PCR diagnostic. Most of the patients were critically ill and treated on the intensive care unit, therefore precise data regarding the time of symptoms onset was not available in the majority of these patients. However, results were compared to manufacturers' claims reporting diagnostic sensitivities based on time from PCR-positivity as well (Table 1). Furthermore, in a study comparing sensitivities of two serological SARS-CoV-2 antibody assays between time from symptom onset and time from PCR-positivity similar results were found for samples obtained at 14 days or later [25].
In conclusion, the investigated assays by Siemens Healthineers, Roche Diagnostics and Euroimmun are all highly specific and sensitive serologic assays for the detection of antibodies against SARS-CoV-2. Highest sensitivity of antibody detection was observed in samples >14 days after PCR-positivity. In addition, in a low percentage of all samples discordant results were observed which are likely due to the different antigen targets of the investigated assays.