The concept “total error” (TE) has different meanings to different authors and has also changed its definition over time , . However, the widespread use of “total analytical error” (TAE) in laboratory medicine represent a testimony to its practical value.
The Milan conference on quality specifications resulted in a “consensus statement” published in 2015 , derived from the previous Stockholm consensus for performance specifications . Recent developments contributing to the incentives for organizing the conference included the requirement of ISO 17025 and 15189 accreditation standards that laboratories routinely provide the measurement uncertainty (MU) of the results, the harmonization of the evaluation of proficiency testing procedures and recent challenges to the TE theory including the calculation of allowable total error (ATE) . It was acknowledged during the Milan conference that many issues including TAE methods remain unresolved and needed further development. An EFLM Task and Finish Group on total error (TFG-TE) was established for that purpose.
TAE methods are firmly rooted in laboratory medicine, but a transition to the MU methods has taken place in other fields of metrology. TAE methods are commonly intertwined with quality assurance, analytical performance specifications and Six Sigma methods , , , , , which are only partially included in this paper , , , .
The aim of the present paper is to fulfill the task of the TFG-TE: to present a proposal on how to use the TE concept and how to possibly combine measures of bias and imprecision in performance specifications. The theoretical and practical underpinning of TAE and uncertainty methods are presented as groundwork for future consensus on their use in practical performance specifications.
This paper presents the consensus as reached within the EFLM task group but does however not represent a general EFLM consensus.
Materials and methods
The Delphi method , widely used method for achieving convergence of opinions of experts, was used. Two Delphi iterations were used. After an initial discussion phase of 8 months in which different views were presented, literature was exchanged and monographs were written, 37 statements were selected from the submitted texts and presented together with appropriate explanations. Eleven participants were subsequently invited to state their agreements or disagreements with the statements in writing. The chair of the TFG acted as moderator (WO). The TGF met to discuss the manuscript at the Warsaw EFLM-UEMS congress, in September 2016.
It was agreed to use the terminology of Joint Committee on Guides in Metrology (JCGM) as expressed in the VIM  and GUM . The terminology of VIM is notably not neutral regarding uncertainty models and does not include a definition of TAE.
Measurement error or simply error is a property of a single measurement – “measured quantity value minus a reference quantity value” . Random measurement error is the component of measurement error that in replicate measurements varies in an unpredictable manner, whereas the systematic error remains constant – or varies in a predictable manner. The concept of error assumes that the difference between the measurement result and the “true value” or reference quantity value can be calculated.
Guide to the expression of uncertainty in measurement (GUM) and International Vocabulary of Metrology (VIM) do not use the concept of “true value”. This is among the fundamental reasons why the concepts of TAE and MU seem incompatible. In the error perspective, measurement is seen as a process aimed at discovering quantity values, which have an independent existence (the “true values”). The better they are approximated, the less will be the error in the measurement result. MU – in contrast – conceives of no preexisting references or true values; ultimately, measurement results should be characterized in terms of the belief a subject attributes to them, expressed as the uncertainty of such results .
In 1974, Westgard et al.  introduced the concept of TAE in an effort to provide a quantitative measure for the acceptability of measurement method performance especially for proficiency testing. Reference laboratories estimate imprecision and bias separately by replicate measurements. In clinical laboratories, however, patient- and quality assurance samples are routinely assayed only once. TAE in these circumstances depends on the combined effect of the random and systematic errors of the method, which is compared with a defined allowable or permissible total error (pTAE, or total error allowable). TAE defines the maximum error for patient results that a single result can show with a certain probability, e.g. 95% or 99%. TAE thus estimates the limits of an interval around the true value where measured analytical results can be found with a defined probability. The TAE model further assumes that the difference between the patients’ result and the true value can be estimated primarily from results from proficiency testing or from internal quality assurance. For many quality assurance applications, there is no need for separate performance goals for bias and imprecision.
Lately, efforts have been made to expand the TAE concept to the evaluation of results of patient samples, including all phases of the total testing process , , . However, several additional sources of errors influence patient samples compared with control samples, e.g. pre-analytical factors, patient factors (e.g. posture), matrix factors (calibrators or control samples that differ from the matrix in the patient samples), and interferences (e.g. in hemolytic, icteric or lipemic samples, drugs).
The concepts of bias and imprecision
Bias is the difference between the average of measurements made on the same sample and its reference , . References are of two types: the reference defining the hypothetical error-free “true value” and the pragmatic reference or target value with an assigned value, e.g. quality control material procedures. The primary choice for estimating analytical bias is the use of a certified reference material (CRM) . Optimal reference materials are not always available, especially when the measured quantity cannot not be unequivocally defined. Comparison with a reference method  can also be used for bias estimation. Sometimes, neither a CRM nor a reference method is available , , making a case for a reference that has a value assigned to it by agreement .
It is important to distinguish between short-term bias (e.g. within day, one shift) and long-term bias (e.g. during several weeks or months): many effects causing short-term bias, e.g. recalibrations may be seen as bias within this short time frame but may be indistinguishable from random effects when variation is observed over a longer period. When uncorrected, many short-term bias components increasingly contribute to the random error component of the MU.
In the case where samples of a particular patient are semirandomly allocated within a larger laboratory organization to different laboratories, several bias components will affect the result. This randomness may also be indistinguishable from imprecision (Figure 1).
Bias is included in the conventional TAE model as a constant without uncertainty. MU methods take this uncertainty into account. It is composed of the uncertainty of the values of the bias and of the reference standard. To improve the TAE estimation, the uncertainty of the bias was suggested to be incorporated into TAE calculation (note 3).
Precision is a concept of quality that is estimated quantitatively as its opposite – imprecision – and expressed as standard deviation or coefficient of variation (CV) . Imprecision is estimated along a gradient between two extremes depending on the measuring conditions. One extreme is the repeatability condition where it is estimated by the same operator under conditions of the same laboratory, apparatus, method, material and within a short period. The other extreme is the reproducibility condition where the same sample is measured during extended periods (weeks, months or years) in different laboratories, involving different operators and measuring systems, methods, laboratory environments, management and quality assurance policies, and even different test methods. Intermediate conditions are conditions in between the two extremes, which need to be specified (see ISO 5725-1:1994).
Westgard and Barry described performance specifications in terms of the TAE that can be tolerated in a test result without compromising its medical usefulness . Cembrowski and Carey  referred to quality goals in terms of the maximum clinically allowable error in a test result. When this paradigm is used to evaluate QC strategies, the primary performance measure of interest should be related to the probability of reporting a test result that contains an analytical error that exceeds the pTAE specification .
Recently, criteria were formulated to assign measurands to appropriate models for analytical performance specifications . The preferred performance specification model is based on clinical needs. However, this model can at present only be applied for only a few measurands (e.g. HbA1c and cholesterol). The second model is based on (components of) biological variation and can be applied for measurands that are in steady state or can be “transformed” to a steady-state situation in biological fluids . A third model is based on the state of the art and can be applied in cases where models 1 and 2 cannot be used.
Since the study of Tonks , several alternative formulas have been suggested for calculating performance specifications based on biological variation. A review of a number of models based on between- and within-subject biological variation led to the notion, “A striking feature is the fact that all of the individual approaches recommend numbers for analytical standard deviation near or equal to 0.5 times the biological standard deviation” . Expressed as analytical and biological coefficients of variation, CVA<0.5CVB. In the case of monitoring, CVI is used instead of CVB.
Some measurands are subject to tight homeostatic control (e.g. electrolytes), lending themselves to very strict performance specifications. Three quality levels are used for imprecision and bias .
These specifications do not combine specifications for imprecision and bias. The term for bias is equal to that derived from the model of Gowans et al.  (see below).
Performance specifications set limits for a test to establish whether the test is acceptable for routine use. Different specifications are needed for screening, diagnosis and monitoring. This also includes optimum goals that may be unachievable by current state-of-the-art procedures. It is vital to obtain a tool for defining the ideal specifications without the influence of the actual (or state of the art) analytical quality, as this will set a goal for manufacturers.
It might seem logical to apply the same limits for analytical performance as limits for internal quality assurance. However, there are reasons to set different limits: quality assurance applies rules to achieve a high probability of error detection and low probability of false rejection based on a singleton result. Quality assurance limits will generally be stricter than performance limits in order to maintain the performance goals and assure that – within a predefined probability – these goals are achieved.
Combining bias and imprecision and calculation of pTAE
The performance specification was proposed for proficiency testing but has been extensively used to define specifications for other purposes . The term for bias is again that of Gowans et al. , combined with the generally accepted maximum imprecision of 0.5CVI (with a coverage factor of 1.65, 95% one-sided). This expression defines a constant value for pTAE and shows a linear relationship between bias and imprecision.
In contrast with this conventional linear model is the curved model proposed by Gowans et al. . With the transferability of reference intervals as starting point, the permissible bias and imprecision are calculated based on the premise that the reference interval limits will remain valid. Because of the inclusion of the biological variation in the model, the resulting relationship between maximum permissible bias and imprecision is curved (see Figure 2) where the related model of Larsen et al.  is used.
The model of Gowans and other reference limit-based models define reference limits based on biological variation alone as a simplification. Oosterhuis and Sandberg  adapted the model of Gowans et al.  to include the influence of analytical variation on the reference interval. Even inclusion of analytical variation, however, might lead to an underestimation of the actual reference interval limits, e.g. because of preanalytical variation (note 2). As an alternative, the actual reference interval limits can be used as starting point in the model . It should be noted that in most distributions, CVG and CVI are log-Gaussian, as are most reference ranges . Performance specifications derived from biological data should ideally be based on this log-Gaussian distribution . Other models combining bias and imprecision for calculation of pTAE might also be considered (e.g. , , , ).
Considerations regarding the TE model
Two modes should be distinguished in proficiency testing and ICQ: the singleton and the multiple measurements of specimens. With a singleton IQC result, we only have the accuracy of the actual result: the difference of the measurement result and the target value (in other words the TAE). There is no way of distinguishing bias and random error components. This value has to be compared with a tolerance limit. However, during extended periods, the laboratory can in retrospect distinguish between bias and imprecision. This information can be derived from quality assurance results (both IQC and appropriately designed proficiency testing programs). In that perspective, bias and imprecision should be (and are) evaluated separately. Reduction of bias and imprecision call for different measures.
Essential statistical considerations govern the possibility to combine bias and imprecision. Bias has a sign, either positive or negative, whereas the standard deviation represents an interval of quantity values. In the CLSI-EP21, the absolute value of the bias is suggested to always be added to the imprecision . In Rili-BAEK, the squared variance and a squared bias are added before taking the square root of the sum , . A statistically correct addition of orthogonal variances using the Pythagorean theorem depends on the elimination of bias – as demanded by GUM.
The caveat with using a single statistic for the maximum permissible error is that the impact or acceptability of its different components is unknown, although bias and imprecision work out quite differently. The same TAE value can result from a low bias and a high imprecision and vice versa. Based on models for diagnosis (e.g. HbA1c in diabetes), bias and imprecision have quite different effects on the proportion of false positives and false negatives . HbA1c is a good example for which there are separate goals for bias (±2%) and imprecision (3%). The pTAE of ±6% is used for practical purposes, e.g. as by the College of American Pathologist’s for grading HbA1c proficiency testing surveys. This data compression might not serve the purpose of goal setting or clinical performance description .
- Both maxima of permissible bias and imprecision are added to obtain pTAE, a pragmatic solution first proposed for the use in proficiency testing . The theoretical basis for this is lacking. Two maximum permissible errors are added, derived under the mutually exclusive assumptions of zero bias and zero imprecision, respectively. The sum will allow an increase of the percentage of test results exceeding the predefined limits.
- The performance specification for imprecision is in general CVA<0.5CVB. In the case of diagnosis, this can be written as CVA<0.5(CVI2+CVG2)1/2. In case of monitoring, only the within-subject variation CVI is included. The maximum permissible bias was derived as 0.25CVB or 0.25(CVI2+CVG2)1/2. It should be noted, however, that in the conventional model, this bias term is applied in the case of monitoring although this expression had been derived from a reference range model and only applies to diagnosis. As an alternative, a model based on a reference change value model was developed that is only based on CVI and not on CVG .
- The conventional linear TAE model takes into account only the analytical variation and the bias. When a tolerance limit is defined at zSD – assuming a Gaussian distribution – with a fixed number of test results outside this limit (e.g. 5% at z=1.64), the combinations of bias and imprecision (CVA) fulfilling this condition show a straight line with a slope of –z (see also Figure 2, TAE conventional with z=2). At one extreme of the line, we have bias=0, with CVa=pTAE/z; at the other extreme, we have CVA=0 (a hypothetical value) and bias=pTAE. The straight line has the characteristic that the tolerance limit pTAE, a combination of bias and imprecision, is a constant equal to the bias at CVA=0. This linearity is of fundamental importance in the methods advocated by both J. Westgard and S. Westgard (www.westgard.com). This model is only valid when imprecision and bias are the only variables involved, or in other words, when the distribution of test results is Gaussian and completely defined by the analytical bias and imprecision alone. Thus – in its original form – the TE model does not include biological variation and other additional causes of variation into the model, except in the calculation of pTAE . Biological variation, although not relevant when monitoring variation in control samples, is certainly relevant when dealing with patient samples .Biological variation can be taken into account by including biological variation in the model transforming the relationship between imprecision and bias into a curve (see Figure 2 Larsen et al. ). The consequence is that the value of pTAE is not a constant in this model for any combination of bias and imprecision (Figure 2).
- It has been argued that the condition CVA<0.5CVI relates to performance specifications that cannot be maintained by internal quality assurance (e.g. leading to a sigma metric below 3, see note 1). For that reason, the 0.5 and the 0.25 coefficients for imprecision and bias might be questioned.
Uncertainty methods as endorsed by the BIPM  originated in physical measurements ,  and chemistry was included as late as in the 1980s. Laboratories of chemistry and related sciences have struggled when adapting to a long tradition established by physical metrology laboratories , .
The basic parameter of MU is the SD, and the symbol for uncertainty is u. In practice, bias correction can reduce systematic errors, and replicate measurements can diminish the effect of random errors. However, this cannot completely eliminate these errors. For that reason – according to the MU concept – the “true value” cannot be exactly known. A measurement result represents the “best estimate” of the measured quantity. The combined uncertainties of bias and imprecision provide an interval of values within which the (unknowable, hypothetical) “true value” of the measured quantity is believed to lie, with a stated coverage probability (e.g. 95%). The MU concept also assumes that if the bias of a procedure is known, then steps are to be taken to minimize it, e.g. by recalibration. However, because the bias value cannot be known exactly, an uncertainty will be associated with such a correction. Thus, in the MU concept, a measurement result can comprise two uncertainties: the uncertainty associated with bias correction and the uncertainty due to imprecision. The uncertainties that act on the measurement result are combined to one MU statistic.
Current developments within the BIPM regarding future versions of GUM and VIM
According to GUM , MU reflects the lack of exact knowledge of the value of the measurand. Developments in MU also emphasize the relationship between the measurement itself and the theoretical models and philosophies underpinning the use of the measurement results , , , , , , , , .
The 1993/1995 version of the GUM catered for both uncertainty and error approaches . Recent supplements to the GUM and the latest version of VIM (VIM3) published in 2008 went completely in the direction of the uncertainty approach, including Bayesian statistics that ultimately relate to the uncertainty of diagnosis , , . However, inconsistencies between the supplements and the main GUM text have been pointed out repeatedly , , , shedding a light on an obvious need to harmonize the approaches to error and uncertainty. Professor Luca Mari, a long-time member of the JCGM Working Group on the International Vocabulary of Metrology (VIM), has in multiple recent publications, e.g. in , , made a case that error and uncertainty methods are not only compatible but also complementary when evaluating measurement data. Our working group finds this approach highly appropriate for use in medical laboratories.
It should be noted that models for the calculation of permissible maximum bias and maximum imprecision are not restricted to either MU or TAE, but the results of these calculations are applicable in both paradigms. In IQC, the TAE and MU models approach each other and can become similar, and the goal setting developed in the TAE model combining bias and imprecision is also applicable for MU .
According to ISO 15189 , MU should be made available by the laboratory on request. Careful interpretation is needed, as MU is already taken into account also in other test characteristics such as sensitivity, specificity, predictive value, etc. Physicians might correct decision levels knowing the MU of a test, taking MU into account again. Analytical variation as well as other sources of uncertainty, including biological and preanalytical variation, are already reflected in reference interval limits and are (or should be) also taken into account in decision limits in clinical guidelines, as “grey zones”, and borderline areas. Lower MU results in higher predictive values and lower clinical uncertainty. This might even lead to a different clinical application of the test, with HbA1c assays as a typical example. The improved reliability of the results made expert organizations revise the guidelines and recommend HbA1c for diagnosis. Such an improved clinical uncertainty is also seen with high-sensitive troponin assays.
The problem of the unknown true value in relation to the definition of error is circumvented in VIM by defining the error with respect to the reference quantity value. This serves a “surrogate true value” within the system (or model), e.g. in the case of internal quality assurance or proficiency testing samples. In the case of patient test results, only an estimated value (or probability) can be assigned to the TE.
We attempt to understand the real world by modelling it. Within a model, we can measure values and make our interpretations within the assumptions of the model. Within the model, we do have “true” values. However, between the modeled values and the real world, there are many uncertainties.
Transposed to our laboratory, this translates to the following (Figure 3): our model consists of calibrators with assigned values and internal quality assurance materials with target values. These are our “true” values, and we can very well express error (trueness) as a number. Proficiency testing, reference methods and certified reference standards represent another model of another level.
However, the measurement of patient samples can be seen as an entity in the real world and knowing the “real” value of the measurands is impossible. Here we can follow the VIM approach, starting with a reference quantity value for our standards and calibrators. The reference quantity value (or reference standard) is determined by the agreed-upon reference method. Therefore, the “analytically” real value of a measurand in the patient sample is the value ultimately traceable to the reference method. The measurement procedure and other sources of variation will contribute to the combined MU of the final measurement result.
Both TAE and MU theory are related to traceability. “Comparability of results across laboratories” must be the goal of the TAE concept. The bias concept in the TAE is in fact related to “traceability”: TAE relates to the difference of the results with a true (reference) value and in this way includes the traceability of the results. What is ignored in the traditional TAE is the degree of the reliability of the transference of the assigned value through the traceability chain and also the reliability of the experiment used for the determination of bias. MU, through incorporating these sources of uncertainty, can help TAE to be an improved measure of traceability of results.
TAE methods for quantifying the quality of measuring systems and defining performance specifications are widely used in laboratory medicine. They are appropriate for analyzing data from single measurement results in proficiency testing schemes and may constitute a basis for the calculation of performance.
The ultimate aim of measuring patient samples in laboratory medicine is to improve the understanding of possible disease conditions or to monitor treatment effects. This aim is influenced by the uncertainties not only of the measuring system but also by biological, preanalytical and postanalytical uncertainty components. This is represented in the Bayesian model used by MU that takes into account all uncertainty components, including bias and imprecision, that effect the measurement result.
Although TAE is generally understood as total analytical error, the types of errors included in the TE definition are varied: preanalytical errors, biological variation and postanalytical errors. It has been argued that the TE concept only deserves the predicate “total” when all kinds of errors are included . Notably, the permissible TAE is often derived from reference intervals , ,  that are substantially influenced by other types of errors (variation) than the analytical variation.
Recent developments in the philosophy of measurement sciences, including those within the BIPM/JCGM, indicate that more than one philosophical outlook and a corresponding selection of different measurement uncertainty calculations may be endorsed. Therefore, it seems prudent to continue the use of TAE methods and when appropriate replace them with MU calculations when the latter offer proven advantages.
Among the major challenges that error methods face are the following:
- There are several variants in how the calculations of TAE are being performed and used, with flaws in pTAE calculations .
- The concept of the “true value” has been abandoned in metrology. If a true value cannot be known, TAE cannot be estimated. We must use “surrogate true values” to estimate TAE.
- TE models do not appreciate the uncertainty of bias estimation/correction.
MU methods in laboratory medicine also face several challenges:
- Error methods are well understood and widely implemented in laboratories. Procedures for implementing them in ISO accreditation schemes are well accepted by accreditation authorities. Therefore, there are no incentives for leaving error methods in favor of uncertainty methods.
- Methods to calculate “permissible MU” as well as quality assurance procedures based on MU theory are not well developed.
- Unfamiliarity with uncertainty calculations within most laboratories .
- The level of knowledge regarding other causes of variation than analytical variation including biological variation, preanalytical and postanalytical variation needed for the estimation of MU is still limited.
Uncertainty methods, according to GUM, are in the early phases of implementation in laboratory medicine in Australia , , . The majority of other countries rely on classic top-down uncertainty calculations, sometimes using ANOVA, covariance analysis and variance component analysis , , .
Imprecision can be estimated as repeatability imprecision at the one extreme, and reproducibility imprecision at the other, with intermediate imprecisions in between. Similarly, bias needs to be estimated separately in the context of the time frame and also the complexity of the measuring systems, including several measuring systems in different laboratories within a laboratory organization (Figure 1). Quality assurance in laboratory medicine needs to be more comprehensively developed to address a situation far more complicated than a single analyzer working in batch mode, for which it was originally developed. The current situation includes large laboratory conglomerates with multiple analyzers working continuously. The most important tasks for laboratory conglomerates are (1) to reduce between-analyzer bias and (2) to identify and technically correct measuring systems with excessive imprecision. Performance specifications should be related to the properties of a whole diagnostic organization including all its measuring systems for diagnosing and monitoring patients. These performance specifications are not yet sufficiently developed despite the fact that ISO 17025 and 15189 require laboratories and conglomerates of laboratories to estimate the overall uncertainty.
Bias and imprecision are different error types with different causes that obviously require different means of correction. Medical laboratories should therefore establish and maintain routines for estimating and minimizing them separately. Bias, however, remains a complicating factor. MU methods advocate correcting bias and include the uncertainty of bias correction. Short-term bias may be indistinguishable from random effects when variation is observed over extended periods and may contribute to the random error component of the MU. There are many methods for combining bias and imprecision, but each method represents some kind of concession, assumption or reduction. Further research and development in this area is needed in order to establish consensus on methods that are optimal for medical laboratories.
Analytical performance specifications should take the diagnostic uncertainty of the whole testing process into account. MU methods according to GUM have not been sufficiently developed to deal with diagnostic uncertainty in laboratory medicine. Development of analytical performance specifications for diagnostic uncertainty has the potential of creating a paradigm shift in laboratory medicine resulting in quality improvements and improved use of diagnostic methods. In anticipation of this paradigm shift, error methods remain the most used methods for quality assurance and analytical performance specifications in laboratory medicine. Currently, error and uncertainty methods are complementary when evaluating measurement results in medical laboratories.
Bossuyt P. M., Deman P., Faria A. P., Ritzen H., Surtihadi J. and Hyltoft Petersen P.
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Note 1. Calculation of sigma metric.
=(Bias+1.65(0.5CVB))/CVA with bias=0
=1.075CVB/CVA with CVA=0.5CVB
Note 2. An example is sodium, with: CVI=0.6%; CVG=0.7%, CVA=0.58%, the calculated reference interval (with CVtot= CVI2+CVG2+CVA2) is 137–143 mmol/L. The actual reference interval of this laboratory is 135–145 mmol/L, a 40% difference that will be reflected in the pTAE. The extra variation will be caused by preanalytical and other factors.
Note 3. Here is where the term uncertainty of bias (UB) needs to be addressed. The difference between the traditional TAE and MU is that MU adds UB to the SD. We can use UB in the TAE calculation to get an estimation of TE as (Uc=uncertainty of concentration):
If bias is zero (either absent from the beginning or removed via recalibration or correction), then the previous equation becomes TAE=MU. According to MU concept, bias should be corrected when known, and the uncertainty of this correction was included.
Note 4. Strictly speaking, these models are difficult to compare: The model of Larsen is aimed at monitoring and only includes CVI. The other model – although used for monitoring – includes both CVI and CVG as discussed in section, which is considered a flaw in this model.
Westgard JO, Carey RN, Wold S. Criteria for judging precision and accuracy in method development and evaluation. Clin Chem 1974;20:825–33.
Westgard JO. Useful measures and models for analytical quality management in medical laboratories. Clin Chem Lab Med 2016;54:223–33.
Sandberg S, Fraser CG, Horvath AR, Jansen R, Jones G, Oosterhuis W, et al. Defining analytical performance specifications: consensus statement from the 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med 2015;53:833–5.
Kenny D, Fraser CG, Petersen PH, Kallner A. Consensus agreement. Scand J Clin Lab Inv 1999;59:585.
Oosterhuis WP. Gross overestimation of total allowable error based on biological variation. Clin Chem 2011;57:1334–6.
Westgard JO, Westgard SA. Assessing quality on the Sigma scale from proficiency testing and external quality assessment surveys. Clin Chem Lab Med 2015;53:1531–5.
Westgard SA. Utilizing global data to estimate analytical performance on the Sigma scale: a global comparative analysis of methods, instruments, and manufacturers through external quality assurance and proficiency testing programs. Clin Biochem 2016;49:699–707.
Westgard JO, Westgard SA. Quality control review: implementing a scientifically based quality control system. Ann Clin Biochem 2016;53(Pt 1):32–50.
Panteghini M, Sandberg S. Total error vs. measurement uncertainty: the match continues. Clin Chem Lab Med 2016;54:195–6.
Oosterhuis WP, Theodorsson E. Total error vs. measurement uncertainty: revolution or evolution? Clin Chem Lab Med 2016;54:235–9.
Kallner A. Is the combination of trueness and precision in one expression meaningful? On the use of total error and uncertainty in clinical chemistry. Clin Chem Lab Med 2016;54:1291–7.
Dalkey NC, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci 1963;9:458–67.
JCGM. International vocabulary of metrology – basic and general concepts and associated terms (VIM 3): Bureau International des Poids et Mesures; 2012. [3 edition] Available at: http://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf. Accessed: 30 July 2017.
JCGM. Evaluation of measurement data – guide to the expression of uncertainty in measurement. JCGM 100:2008, GUM 1995 with minor corrections. Available at: http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf. Accessed: 30 July 2017. Joint Committee for Guides in Metrology, 2008.
Mari L. The ‘error approach’ and the ‘uncertainty approach’: are they incompatible? Leiden: Lorentz Center, 2011.
Menditto A, Patriarca M, Magnusson B. Understanding the meaning of accuracy, trueness and precision. Accred Qual Assur 2007;12:45–7.
De Bievre P. On ‘trueness control materials’, better known under the multi-purpose term of ‘Certified Reference Materials’ (CRMs). Accredit Qual Assur 2010;15:71–2.
Vesper HW, Miller WG, Myers GL. Reference materials and commutability. Clin Biochem Rev 2007;28:139–47.
Greg Miller W, Myers GL, Lou Gantzer M, Kahn SE, Schonbrunner ER, Thienpont LM, et al. Roadmap for harmonization of clinical laboratory measurement procedures. Clin Chem 2011;57: 1108–17.
Westgard JO, Seehafer JJ, Barry PL. Allowable imprecision for laboratory tests based on clinical and analytical test outcome criteria. Clin Chem 1994;40:1909–14.
Cembrowski GS, Carey RN. Considerations for the implementation of clinically derived quality control procedures. Lab Med 1989;20:400–5.
Parvin CA. Quality-control (QC) performance measures and the QC planning process. Clin Chem 1997;43:602–7.
Ceriotti F, Fernandez-Calle P, Klee GG, Nordin G, Sandberg S, Streichert T, et al. Criteria for assigning laboratory measurands to models for analytical performance specifications defined in the 1st EFLM Strategic Conference. Clin Chem Lab Med 2017;55:189–94.
Tonks DB. Quality control systems in clinical chemistry laboratories. Postgrad Med 1963;34:A58–70.
Stöckl D, Baadenhuijsen H, Fraser CG, Libeer J-C, Petersen PH, Ricós C. Desirable routine analytical goals for quantities assayed in serum. Discussion paper from the members of the external quality assessment (EQA) working group A on analytical goals in laboratory medicine. Eur J Clin Chem Clin Biochem 1995;33:157–69.
Fraser CG, Petersen PH, Libeer JC, Ricos C. Proposals for setting generally applicable quality goals solely based on biology. Ann Clin Biochem 1997;34(Pt 1):8–12.
Gowans EM, Peteresen PH, Blaabjerg O, Hörder M. Analytical goals for the acceptance of common reference intervals for laboratories throughout a geographical area. Scand J Clin Lab Invest 1988;48:757–64.
Fraser CG, Petersen PH. Quality goals in external quality assessment are best based on biology. Scand J Clin Lab Invest 1993;53:8–9.
Fraser CG, Peterson PH. Desirable standards for laboratory tests if they are to fulfill medical needs. Clin Chem 1993;39:1447–5.
Larsen ML, Fraser CG, Petersen PH. A comparison of analytical goals for haemoglobin A1c assays derived using different strategies. Ann Clin Biochem 1991;28(Pt 3):272–8.
Oosterhuis WP, Sandberg S. Proposal for the modification of the conventional model for establishing performance specifications. Clin Chem Lab Med 2015;53:925–37.
Haeckel R, Wosniok W, Postma T. Quantity quotient reporting. Comparison of various models. Clin Chem Lab Med 2015;53:1921–6.
Petersen PH, Stockl D, Blaabjerg O, Pedersen B, Birkemose E, Thienpont L, et al. Graphical interpretation of analytical data from comparison of a field method with reference method by use of difference plots. Clin Chem 1997;43:2039–46.
Haeckel R, Wosniok W, Gurr E, Peil B. Permissible limits for uncertainty of measurement in laboratory medicine. Clin Chem Lab Med 2015;53:1161–71.
Asberg A, Odsaeter IH, Carlsen SM, Mikkelsen G. Using the likelihood ratio to evaluate allowable total error – an example with glycated hemoglobin (HbA(1c)). Clin Chem Lab Med 2015;53:1459–64.
Oddoze C, Lombard E, Portugal H. Stability study of 81 analytes in human whole blood, in serum and in plasma. Clin Biochem 2012;45:464–9.
McDonald R. Quality assessment of quantitative analytical results in laboratory medicine by root mean square of measurement deviation. J Lab Med 2006;30:111–7.
White GH, Farrance I, AACB Uncertainty of Measurement Working Group. Uncertainty of measurement in quantitative medical testing: a laboratory implementation guide. Clin Biochem Rev 2004;25:S1–24.
Astles JR, Person NB, Armbruster DA, Pierson-Perry JF, Kondratovich MV, Scott MG, et al. CLSI-EP21, Evaluation of total analytical error for quantitative medical laboratory measurement procedures, 2nd ed. Wayne, NJ: Clinical and Laboratory Standards Institute, 2016.
Petersen PH, Klee GG. Influence of analytical bias and imprecision on the number of false positive results using guideline-driven medical decision limits (Reprinted from Clinica Chimica Acta vol 430C, pg 1–8, 2014). Clin Chim Acta 2014;432:127–34.
Fraser CG, Petersen PH. Desirable standards for laboratory tests if they are to fulfill medical needs. Clin Chem 1993;39:1447–53; discussion 53–5.
Williams A. What can we learn from traceability in physical measurements? Accredit Qual Assur 2000;5:414–7.
Williams A. Traceability and uncertainty – a comparison of their application in chemical and physical measurement. Accredit Qual Assur 2001;6:73–5.
Mari L, Giordani A. Modeling measurement: error and uncertainty. In: Boumans M, Hon G, Petersen A, editors. Error and uncertainty in scientific practice. London: Pickering & Chatto, 2014:79–96.
Giordani A, Mari L. Measurement, models, and uncertainty. Ieee T Instrum Meas 2012;61:2144–52.
Psillos S. Scientific realism: how science tracks truth. London: Routledge, 1999.
Giere RN. Explaining science: a cognitive approach. Chicago: University of Chicago Press, 1988.
Giere RN. Cognitive models of science. Minneapolis: University of Minnesota Press, 1992:xxviii:508.
Giere RN. Scientific perspectivism. Chicago: University of Chicago Press, 2006:151.
Tal E. Old and new problems in philosophy of measurements. Philosophy Compass 2013;8:1159–73.
Tal E. Measurement in Science. 2015. In: The Stanford Encyclopedia of Philosophy [Internet]. Available at: http://plato.stanford.edu/archives/sum2015/entries/measurement-science. Accessed: 30 July 2017.
Tal E. Measurement in science. In: Zalta EN, editor. Stanford encyclopedia of philosophy, 2015.
Kacker R, Jones AW. On use of Bayesian statistics to make the guide to the expression of uncertainty in measurement consistent. Metrologia 2003;40:235–48.
Vallverdú J. Bayesian versus frequentists. A philosophical debate on statistical reasoning. Heidelberg: Springer, 2016.
Sanogo M, Abatih E, Saegerman C. Bayesian versus frequentist methods for estimating true prevalence of disease and diagnostic test performance. Vet J 2014;202:204–7.
Weise K, Woger W. A Bayesian theory of measurement uncertainty. Meas Sci Technol 1993;4:1–11.
Lira I. The GUM revision: the Bayesian view toward the expression of measurement uncertainty. Eur J Physics 2016;37:1–16.
Kyriazis GA. Contributions to the revision of the ‘Guide to the expression of uncertainty in measurement’. J Phys Conf Ser 2015;575:1.
Bich W. How to revise the GUM? Accredit Qual Assur 2008;13:271–5.
ISO. ISO 15189:2012 Medical laboratories – requirements for quality and competence. Geneva: International Standardisation Organisation, 2012.
Haeckel R, Wosniok W, Streichert T. Optimizing the use of the “state-of-the-art” performance criteria. Clin Chem Lab Med 2015;53:887–91.
Haeckel R, Wosniok W. A new concept to derive permissible limits for analytical imprecision and bias considering diagnostic requirements and technical state-of-the-art. Clin Chem Lab Med 2011;49:623–35.
Farrance I, Frenkel R. Uncertainty of measurement: a review of the rules for calculating uncertainty components through functional relationships. Clin Biochem Rev 2012;33:49–75.
Farrance I, Frenkel R. Uncertainty in measurement: a review of monte carlo simulation using microsoft excel for the calculation of uncertainties through functional relationships, including uncertainties in empirically derived constants. Clin Biochem Rev 2014;35:37–61.
Norheim S. Computer support simplifying uncertainty estimation using patient samples. Linkoping: Linkoping University, 2008. Available at: http://liu.diva-portal.org/smash/record.jsf?pid=diva2:417298. Accessed: 30 July 2017.
Theodorsson E. Validation and verification of measurement methods in clinical chemistry. Bioanalysis 2012;4:305–20.
Theodorsson E, Magnusson B, Leito I. Bias in clinical chemistry. Bioanalysis 2014;6:2855–75.