Wrong Sigma metric causes chaos

To the Editor,

I read the recently published paper by Ozdemir et al. [1] with great interest. In the paper, the authors pointed out an important issue related to the harmonization of Sigma metric (SM) as the performance criteria in laboratory medicine. I agree with the authors that harmonization is needed for SM in laboratory medicine. However, accurate calculation of SM is essential prior to harmonization. As detailed below, SM cannot be accurately calculated using the methodology and equation given in the paper [1] and consequently harmonization of SM cannot be achieved as proposed by the authors.

As stated by Albert Einstein ‘everything should be made as simple as possible, but not simpler’. The original equation used to calculate SM is very simple and based on the normal distribution [2].

where TL is the tolerance limit (from the target to lower limit [LL] or upper limit [UL]) and SD is the standard deviation of the process. If TL is given as percentage then SD should be given in terms of coefficient of variation (CV). The number of SDs from the target to the ULs or LLs gives the SM value [2]. The normal distribution curve is symmetric around the target and the area under the curve (AUC) on the right and left sides are equal to each other (Figure 1A). In contrast to this very simple equation, the approach proposed by the authors cannot be used to calculate the performance of the process and not correct both in theory and practice as summarized below.

Figure 1:

Normal and uniform distribution.

The geometrical shape of normal distribution is bell-shaped and the distribution density is higher around the target then the tails (A). SM is the number of SDs between the target and ULs or LLs; Figure 1A illustrates a process with 3SM. The geometrical shape of the uniform distribution is rectangular and the distribution density is homogenous from the LL to the UL (B). T, target; UL, upper limit; LL, lower limit; SM, Sigma metric; SD, standard deviation.

Figure 2:

Moving the normal distribution curve from the target changes the AUC between the LLs and ULs dramatically.

If bias =0 then the AUC around the target is symmetric (A), but if bias >0 then the AUC around the target is not symmetric (B). T, target; UL, upper limit; LL, lower limit; AUC, area under the curve.

The equation used by the authors was proposed by Westgard to calculate the SM of the process in laboratory medicine.

where TEa is the total allowable error and bias is the shift from the target of the analyte. Although both equations aims to calculate the same parameter, i.e., SM, the numerator of both equations are different. If we take Eq. (1) as the reference for SM calculation, Eq. (2) will not give the same result. The difference between both equations and the approaches to calculate the SM in industry and laboratory medicine can be summarized under two main subheadings.

1. TEa cannot be used as the tolerance limit of the process

In industry, TL of a process can be given by the customer based on the quality requirement of the product. If this is not possible then the TL of the process can be calculated using Taguchi loss function [3]. In other words, TEa is not used as the TL of the process in any sector of industry [4].

The first European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Strategic Conference proposed the hierarchy of models for analytical performance specification as given below [5].

1. Model 1. Based on the effect of analytical performance on clinical outcomes;

2. Model 2. Based on the BV of the measurand;

3. Model 3. Based on the state-of-the-art.

Contrary to popular belief, the TEa as the TL of the process, has not been determined by physicians and as far as I know, never been derived from the data of model 1. Instead, it is calculated using the following equation:

With this equation it is accepted that the TEa is 5% defective. Because 1.65 coefficient corresponds to 95% probability for one-sided of the normal distribution curve. Even if all other components are perfect, the SM of a process with 5% defects corresponds to 1.65 (one-sided). Therefore using TEa as the TL and then obtaining SM >1.65 is totally nonsense. Additionally, it should be noted that in case of using Eq. (3), bias or imprecision should be minimum otherwise the result will be large which cannot be accepted as the ‘allowable’ [6].

2. Bias cannot be treated as a linear parameter in Sigma metric calculation

One of the differences between Eqs. (1) and (2) is the inclusion of bias as a linear parameter into Eq. (2). Statistically the performance of the process is directly related to the AUC of normal distribution within the TLs. In statistics the distribution of data is characterized by a mathematical function and numerous different distributions have been determined [7]. Normal distribution is not uniform (Figure 1A, B) and therefore moving the curve to the right or left, dramatically changes the AUC within TL (Figure 2A, B). In contrast to normal distribution, bias can be treated as a linear parameter only in uniform distribution. Because the geometrical shape of the uniform distribution is rectangular (Figure 1B) and the distribution density is accepted as homogenous from the LL to the UL. On the other hand, the geometrical shape of normal distribution is bell-shaped and the distribution density is higher around the target than the tails. Therefore in normal distribution, complex calculations are needed to obtain the AUC when the curve moves to the right or to the left and consequently inclusion bias as a linear parameter is not acceptable and mathematically it can be proved easily that such calculation is invalid [8].

The original SM equation (Eq. (1)) does not include bias as a linear parameter. In fact, what needs to be done is the correction of bias. However, it is not easy to monitor and calculate real-time bias. To overcome this problem the presence of 1.5 SD bias is accepted in advance. SM value can be converted to defects per million opportunities (DPMO) and vice versa. For example world class SM is 6 and it corresponds to 3.4 DPMO. It should be noted that the 3.4 DPMO corresponds to 6 sigma includes 1.5 SD shift but in reality the DPMO without bias for 6 sigma is 0.002 [9]. SM-DPMO tables with 1.5 SD shifts are known as long-term tables, while tables without 1.5 SD shifts are known as short-term tables. Since mathematically the inclusion of bias as a linear parameter in the SM equation requires complex calculations, the effect of 1.5 SD bias on the performance of the process is given in long-term DPMO tables [9] rather than using equations. In the paper the authors included bias (Eq. (2)) and used long-term DPMO Tables. In this case two biases are included in calculation: 1.5 SD and the bias calculated from the EQAS reports. Consequently the calculated performance of the process is usually underestimated than the actual level.

One of the errors in the paper is the omitting of the significance of bias. The significance of bias [10] is an important issue but unfortunately rarely is evaluated in medical laboratories. According to the International Vocabulary of Metrology, bias is defined as the difference between the mean of repeated measurements and the reference value of the analytes. It cannot be claimed that this difference is always significant. It should be noted that, if bias is included in a calculation, its significance should be evaluated using a suitable statistical test. Insignificant bias should be neglected and not included in any calculations.

In conclusion, the SM equation proposed by Westgard has caused chaos rather than solving problems. If the method used to calculate the performance of the process gives different performance values i.e., different SMs, it causes chaos rather than solving the problem.