Comparison of Sigma metrics computed by three bias estimation approaches for 33 chemistry and 26 immunoassay analytes

Abstract Objectives Sigma metric can be calculated using a simple equation. However, there are multiple sources for the elements in the equation that may produce different Sigma values. This study aimed to investigate the importance of different bias estimation approaches for Sigma metric calculation. Methods Sigma metrics were computed for 33 chemistry and 26 immunoassay analytes on the Roche Cobas 6000 analyzer. Bias was estimated by three approaches: (1) averaging the monthly bias values obtained from the external quality assurance (EQA) studies; (2) calculating the bias values from the regression equation derived from the EQA data; and (3) averaging the monthly bias values from the internal quality control (IQC) events. Sigma metrics were separately calculated for the two levels of the IQC samples using three bias estimation approaches. The resulting Sigma values were classified into five categories considering Westgard Sigma Rules as ≥6, <6 and ≥5, <5 and ≥4, <4 and ≥3, and <3. Results When classifying Sigma metrics estimated by three bias estimation approaches for each assay, 16 chemistry assays at the IQC level 1 and 2 were observed to fall into different Sigma categories under at least one bias estimation approach. Similarly, for 12 immunoassays at the IQC level 1 and 2, Sigma category was different depending on bias estimation approach. Conclusions Sigma metrics may differ depending on bias estimation approaches. This should be considered when using Six Sigma for assessing analytical performance or scheduling the IQC events.


Introduction
A vast majority of medical diagnosis and treatments, estimated 60-70 %, are based on data from clinical laboratories [1].Therefore, the activities of total testing process, which are carried out in three phase namely pre-analytical, analytical, and post-analytical, should be closely observed in order to ensure patient safety.Internal and external quality assurance procedures are routinely performed to monitor the accuracy and precision of the analytical testing process.The analytical performance of a new or existing system can be also evaluated by additional approaches e.g., six Sigma [2].
Six Sigma methodology was introduced into laboratory medicine in 2001, and its implementations extended beyond the acceptability of method [3].With Sigma metrics, laboratorians can define the internal quality control (IQC) rules, and the number of control measurements required per run, and now apply a risk-based IQC plan by determining control frequency [2].
Six Sigma methodology combines three elements, namely, imprecision, bias, and total allowable error (TEa) into a single value [2].Multiple sources for the TEa targets exist worldwide.In consensus statement from the 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM), three models have been recommended to set analytical performance specifications (APS): the first model based on clinical outcomes, the second on biological variation, and the third on state-of-the-art [4].The TEa values from different sources for any measurand may be different, which produces a challenge for laboratories in choosing the quality specifications for the Sigma metric calculation [5].
Similar to TEa targets, there is heterogeneity in estimating imprecision for Sigma metric calculation.To obtain the imprecision data, some studies have performed the replication study following a standardized protocol in a relatively brief period [6][7][8][9][10].In other studies, the imprecision has been computed using data coming from long-term IQC studies performed routinely [11][12][13][14][15].
Another challenge for Sigma metric calculation is bias estimation.Bias can be determined from different approaches.In an ideal manner, the preferred approach for bias estimation is to compare the results from a reference material or patient specimens being assayed using a field method with those of a reference method [16].In previous studies regarding Six Sigma in the clinical laboratory, the bias of a given test has usually been obtained through comparison with the group mean of the external quality assurance (EQA) program.In this approach, multiple bias values obtained from long-term EQA studies were converted into one bias value through averaging [11][12][13]15] or regression analysis [6,17] for Sigma calculation.Another preferred approach to determine bias is to calculate the difference between the observed control results and the IQC target (or mean) value [5,10,14,15].Furthermore, some studies have estimated bias through the regression analysis of the results from method comparison studies [7][8][9].
The previous studies reported that the Sigma values directly depend on TEa goals from different sources [5,15,18].However, there is limited data on whether Sigma values vary to the preferred approach for estimating bias.
This study aimed to compare Sigma values of 33 chemistry and 26 immunoassay analytes determined using three bias estimation approaches, including averaging the monthly bias values obtained from the EQA studies during a 1-year period, calculating the bias values from regression equation derived from the EQA data, and averaging the monthly bias values from the IQC studies according to the target control values.

Materials and methods
This observational study was conducted based on the EQA and IQC performance of 59 analytes, 33 for chemistry and 26 for immunoassays, on the Cobas 6000 analyzer (Roche Diagnostics, Manheim, Germany).The analytes evaluated are listed in Table 1.

Total allowable error (TEa)
There are multiple sources for the TEa, but none of them comprises all analytes evaluated in the current study.
The hierarchy model proposed in Stockholm in 1999 and Milan in 2014 was followed to choose APS [4].In this model, the primary choice for the selection of APS is the clinical outcome, but this is unfortunately available for a few analytes.TEa targets were selected for triglyceride [19], total cholesterol [20], high-density lipoprotein cholesterol [20], lowdensity lipoprotein cholesterol [21], and glycated hemoglobin (HbA 1c ) [22] based on clinical outcome.
The secondary choice was the desirable TEa based on biological variation obtained from EFLM Biological Variation Database [23].If there was no data in this database, then the biological variation database created by The Spanish Society of Laboratory Medicine Analytical Quality Commission, was updated every two years until 2014, was consulted [24].If the biological variation targets were not practical (too  [25], The Royal College of Pathologists of Australasia (RCPA) [26], and Guideline of the German Medical Association on Quality Assurance in Medical Laboratory Examinations -Rili-BAEK goals [27] were referenced next.Finally, for procalcitonin and anti-streptolysin O (ASO), no TEa data were available from the above sources.The TEa for procalcitonin was obtained from a paper based on the biological variation [28], and a TEa of 10 % was arbitrarily preferred for ASO.
The source of TEa preferred for each assay is listed on Table 1.

Bias
To calculate bias, the EQA data were collected from RIQAS EQA schemes (Monthly clinical chemistry, specific proteins, glycated hemoglobin, lipid, monthly immunoassay, liquid cardiac, and immunoassay specialty) throughout 1-year from 1 March 2019 to 29 February 2020.The bias of each analyte was estimated by three different approaches.In the first approach, the difference between the laboratory result and the peer group mean value was first calculated for each of the 12 EQA surveys and then the average bias was calculated by the root mean square of the individual bias values.
In the second approach, Passing-Bablok regression analysis [29] was performed using the laboratory results and the peer group mean values obtained from the 12 EQA surveys.The bias was calculated from the regression equation constructed as y=b+ax, where a was an intercept, b was a slope, and x was a value having the concentration at the mean of the IQC sample.In this approach, the bias of each analyte was computed separately for the IQC level 1 and 2.
When using the bias estimation approaches based on the EQA results, if the standard deviation index for an assay was above 3 or below −3 at an EQA event, the related bias value was not used for Sigma metric estimation.
In the final approach, the difference between the laboratory result and the target value of the IQC sample obtained from the control inserts was first calculated for each month over one year, and then the average bias was estimated by the root mean square of the monthly bias values.
The IQC samples were obtained from the manufacturer of the reagent (Roche Diagnostics, Manheim, Germany).Similar to the second approach, the bias of each analyte was estimated individually for the IQC level 1 and 2.

Precision
The precision was determined through the computation of the %CV from the IQC data.The IQC data was collected during the same time frame as the EQA data.
%CV values were calculated monthly, and then the average precision was estimated.The precision values were estimated separately for the both of IQC samples.

Categorization of Sigma metrics
Sigma metrics computed using bias values obtained from different estimation approaches were entitled following: (1) If Sigma value was estimated using bias values based on the average of multiple bias values from the EQA surveys, it was entitled "EQA Average Sigma".(2) If Sigma value was computed using bias values from regression analysis of the EQA results, it was entitled "EQA Regression Sigma".(3) When Sigma metric was calculated using bias values based on the average of the bias values from the IQC studies, it was entitled "IQC Sigma".
Data analyses were performed using MedCalc Statistical Software version 19.1 (MedCalc Software Ltd, Ostend, Belgium) and Excel Office 2019 (Microsoft, USA).

Results and discussions
Sigma values estimated using different bias determination approaches are presented in Table 2 and 2 and 3 for control levels 1 and 2, respectively.In addition,Supplementary Table 4 shows the regression equation derived from the Passing-Bablok regression analysis for each analyte, as well as the target values used to estimate the bias.
For 64 % of chemistry assays at the IQC level 1 and 73 % at level 2, the "EQA Regression Sigma" values were observed Ercan: Sigma calculations based on different methods of determining bias  Ercan: Sigma calculations based on different methods of determining bias to be higher by at least 0.5 when compared to the "EQA Average Sigma" values.Similarly, for 42 % of chemistry assays at the IQC level 1 and 45 % at level 2, the "IQC Sigma" values were found to be higher than the "EQA Average Sigma" values.In addition to this, when comparing with the "IQC Sigma" values, "EQA Regression Sigma" values were encountered to be higher in 9 chemistry assays while the "IQC Sigma" values in 5.These findings demonstrate that the bias estimation based on regression analysis of the EQA results has the potential to yield higher Sigma values for chemistry analytes.
In contrast to chemistry assays, for immunoassay analytes, Sigma values estimated using bias values based on the IQC results were higher than that computed by the other bias estimation approaches.For 65 % of immunoassays analytes at the IQC level 1 and 69 % at level 2, the "IQC Sigma" values were observed to be higher by at least 0.5 compared to the "EQA Average Sigma" values.Likewise, for 42 % of immunoassays analytes at the IQC level 1 and 62 % at level 2, the "IQC Sigma" values were found to be higher than the "EQA Regression Sigma" values.Moreover, there was a difference of at least 0.5 between the "EQA Average Sigma" values and the "EQA Regression Sigma" values in 13 assays at the IQC level 1 and 9 assays at the IQC level 2.
This difference led some assays to be classified into different Sigma categories.Twenty-three analytes (15 chemistry, 8 immunoassays) at IQC level 1 and 21 (14 chemistry, 7 immunoassays) at level 2 were classified into different Sigma categories depending on whether considering the "EQA Average Sigma" or the "EQA Regression Sigma" values.In addition, for 18 analytes (7 chemistry, 11 immunoassays) at IQC level 1 and 20 (8 chemistry, 12 immunoassays) at level 2, the Sigma categories were observed to be different depending on whether the "EQA Average Sigma" or the "IQC Sigma" values were considered.Similarly, if the bias was derived from the regression analysis of bias values from the EQA studies, instead of IQC data, 19 analytes (11 chemistry, 8 immunoassays) at IQC level 1 and 23 (12 chemistry, 11 immunoassays) at level 2 fell into different Sigma categories.For most of the assays with a Sigma value greater than 6, Sigma categories were observed to be unchanged depending on different bias estimation approaches.This is due to Sigma values equal to or greater than 6 are classified in the same category, so that the greater the Sigma value the greater the difference required to produce a change in this Sigma category.For example, although there was a difference of 2.2 units between the "EQA Average Sigma" and the "EQA Regression Sigma" for PTH at IQC level 2, the two fell into the Sigma category 5.
In the present study, Sigma values were separated into categories considering the Westgard Sigma Rules diagram [30].Therefore, the observed difference in the Sigma category depending on the bias estimation approaches for an assay can alter the QC rules to be chosen, the number of control measurements required per run, and the running frequency of control samples.Using as an example AST, the "IQC Sigma" performance at the IQC level 1 requires only a single control rule, 1 3s , with 2 control measurements in each run one on each level of control, whereas the "EQA Average Sigma" quality requires 4 rules, 1 3s /2 2s /R 4s /4 1s , with 4 control measurements in each run or 2 control measurements in each of 2 runs.
In addition to this, the observed variations in Sigma categories will have changed the judgement about the performance of an existing analytical system in the laboratory.When evaluating analytical performance of the chemistry assays by considering Sigma of <3 as unacceptable, albumin, creatinine, sodium, chloride, and ASO was classified in this category at the IQC level 1, regardless of the bias estimation approaches.In addition to these analytes, the "EQA Regression Sigma" value for HDL-cholesterol was observed to be <3.At the IQC level 2, creatinine, sodium, and ASO showed a Sigma performance of <3 considering both the "IQC Sigma" values and the "EQA Average Sigma" values.Moreover, ALP had an "IQC Sigma" value of <3.
At both the IQC level 1 and 2, FT3, FT4, folic acid, and 25-OH vitamin D showed a Sigma performance <3, irrespective of the bias estimation approaches.In addition to these analytes, the "EQA Average Sigma" and "EQA Regression Sigma" value for CA 125 was observed to be <3 at both the IQC level 1 and 2. On the other hand, at the IQC level 1, estradiol had a Sigma performance of <3 taking into consideration the "EQA Regression Sigma" or the "IQC Sigma" value, as well as NT-ProBNP had the "EQA Average Sigma" value of <3, and ferritin had the "EQA Regression Sigma" value of <3.
Bias estimation approaches evaluated here each has an advantage or disadvantage.Ideally, the EQA organizations using commutable control materials with assigned values by reference methods should be preferred to verify the accuracy of laboratory results and their impact on patient samples [31].However, several EQA providers use commutable or non-commutable control materials but without values assigned by reference methods [31].On the other hand, there are no reference materials or reference methods for several analytes commonly tested in laboratory medicine.The EQA target values based on peer-group (laboratories using the same method, analytical platform, and reagents) means were used in the present study.In that case, the uncertainty of the peer-group mean value, which depends on the imprecision of data used and the number of laboratories in the peer group, has importance on the observed bias.
Although there is a direct relationship between the uncertainty and sample size of the peer group, standardization about how many participants are required in a peer-group is not present.In a previous study [17], a peer group with at least 5 participants was considered adequate for bias estimation.In the present study, the number of laboratories in the peer group was relatively high, with at least 100 participants for immunoassays and 30 for chemistry assays.
For bias estimation from the EQA results, another limitation is that bias estimated in each EQA event is based on a single measurement result.Therefore, bias estimation in this way is susceptible to random errors, especially in case of using bias values from a few EQA studies.Bias estimation on a larger period may help to diminish the effect of possible random errors.In a recent study [15], authors recommended that laboratories calculate Sigma metrics using the data from at least 6 months.On the other hand, when using regression analysis for bias estimation by the EQA results, the sample size becomes a more complicated issue.The number of data pairs recommended for Passing-Bablok regression analysis is 30 [32].However, it is not easy to achieve this sample size using the EQA surveys.By collecting the data over one year, 12 data points could be obtained for the regression analysis in this study.
In addition to the sample size, the distribution of values is another important point in regression analysis.For example, when the regression analysis was performed using the data points including the concentrations of HDL-cholesterol ranging from 46 to 134 mg/dL, the resulting regression equation yielded a bias of 1.46 % and 11 % for the concentrations of 73.8 and 28.6 mg/dL, respectively.This was attributed to the lack of points with low concentration in the data set used in regression analysis.
Similar to the EQA target values, the IQC target values are not assigned by reference method analysis.In addition to this, although the IQC target values are recommended to be established by the laboratory [33], laboratories commonly rely on the mean values presented by the assay manufacturer in control package inserts.Therefore, the uncertainty of the target values is also an issue for bias estimation based on the IQC data.However, the approach based on the mean of bias values of the IQC results is performed using more data points than those based on the EQA results, which makes bias estimation based on the IQC data more robust to random errors.
There is heterogeneity between previous studies as to bias estimation.The studies evaluating the analytical performance of a new system by Sigma metrics have commonly calculated bias from data based on the method comparison study [7][8][9].On the other hand, bias has been computed using the IQC and EQA results when Sigma metric analysis was performed to plan the IQC studies or assess the analytical performance of an existing system [10][11][12][13][14][15].
Tran et al. [17] evaluated Sigma metrics of 20 chemistry analytes on the Beckman Coulter AU680 using the bias estimation approach based on regressing analysis of the EQA results.Similar to the findings in the present study, they computed a Sigma performance of <3 for albumin, creatinine, sodium, and chloride at the IQC level 1. Authors have also reported that calcium, total protein, glucose, and CK had a Sigma value of <3 at the IQC level 1.Moreover, for 6 of the 8 assays at the IQC level 1, the same findings have been reported for the IQC level 2. The difference in Sigma values between the two studies was attributed to the difference in TEa values considered for the assays, except glucose and CK.The reason of low Sigma values for glucose and CK was higher bias values in the cited study.The difference in bias values between the two studies may be partly explained by the difference in the analytical platforms used.Moreover, the distribution of the data used in the regression analysis may have affected the bias values.Unfortunately, the relevant data is not available in the cited study.
In another study, Nar and Emekli [14] estimated Sigma metrics of 18 immunoassay analytes on Cobas e601 analyzer, 17 of which are the same as the present study, to evaluate analytical performance.Authors computed the bias by determining the difference between the target values (mean values) in the control inserts and the observed mean values.Similar to the current study, they have reported a Sigma value of <3 for FT4.Moreover, unlike the findings from this study, Sigma value of <3 have been also reported for AFP.On the other hand, they have estimated a Sigma of >6 for folic acid which was of Sigma of <3 in the present study.A similar Sigma performance should be achieved when laboratories use the same analytical platform and reagent formulation.Therefore, in the case of poor individual-site Sigma performance, local issues that influence analytic performance should be targeted.These factors may be instrumentation malfunctions, reagent issues, sample handling, or possible operator-related errors.
Kumar and Mohan [12] assessed the Sigma performance of 16 chemistry assays on VITROS 4600 using the average bias from monthly bias values based on the EQA results.The authors reported a Sigma performance similar to that in this study for sodium and albumin, but the Sigma values for creatinine were higher.Despite similar bias and imprecision values for creatinine, the observed difference in Sigma values was due to the difference in TEa values considered in the studies.Authors have selected a TEa of 15 % based on state-ofart whereas a TEa of 7.5 % based on biological variation was preferred in the present study.They have also reported Sigma values of <3 for urea using a TEa of 9 % based on state-of-art

Table  :
The total allowable error (TEa) limits and its sources.
3 for the IQC level 1 and 2, respectively.The data used to estimate bias values are provided in the supplementary files.Bias values based on the EQA results are presented monthly in Supplementary Table 1.No bias value was excluded due to the standard deviation index.Bias values estimated by the IQC results are given monthly in Supplementary Tables

Table  :
(continued) c standardization []; a The TEa for ASO was chosen arbitrarily as data was not available.b Reference [].

Table  :
Sigma metrics calculated by three bias estimation approaches at the IQC level .

Table  :
Sigma metrics calculated by three bias estimation approaches at the IQC level .