External quality assessment schemes for glucose measurements in Germany: factors for successful participation, analytical performance and medical impact

Data

Glucose data for the years 2012–2016 were obtained from the EQA schemes RfB KS, “Clinical chemical analytes in serum – wet chemistry” (CL-RfB) and Instand 100, “Clinical Chemistry – Wet Chemistry” (CL-Instand). These schemes were designed for automated analyzers in CL where glucose measurements were part of larger panels. In addition, measurements from RfB GL, “Glucose (wet and dry chemistry)” (POCT-RfB), and Instand 800, “Dry Chemistry 01 – POCT: Glucose” (POCT-Instand), were acquired. All data were reformatted to a common structure and analyzed using the R statistical software version 3.4.0. Wherever possible, devices were matched to a common denomination across both organizations. All other devices were subsumed into the “others” device category. Full code is available at https://github.com/acnb/Glucose-EQA. Data are released under https://doi.org/10.17605/OSF.IO/F3CD5.

EQA material

Two liquid plasma samples (POCT-Instand) or two lyophilized serum samples (CL-RfB, CL-Instand, POCT-RfB) were used in each glucose EQA distribution (Supplemental Methods).

Odds ratios

We classified deviations of measurements from the target value as follows. In line with the analytical performance specifications stipulated in the German Rili-BAEK [19], deviations >15% from the target value were classified as “failed” participation of the EQA distribution (which also resulted in the denial of the EQA certificate). In line with Bukve et al. [20], participation outcomes in an EQA distribution other than “failed” were classified as “poor” if a result exceeded the target interval (target value±2 mg/dL [0.1 mmol/L]) by >10%, as “acceptable” for deviation of 10%–5% from the target interval, and as “good” for deviation ≤5%. The sample with the largest deviation from the target interval was used for the final classification unless the EQA organization retracted one sample (e.g. due to insufficient stability).

Odds ratios (OR) were calculated for the probability to reach a “good” result and for the probability to not reach a “failed” result. Independent variables were the device employed, experience from previous participations in the same EQA scheme and simultaneous participations in other glucose EQAs from the same organization. Missing experience data were substituted with plausible values using multiple imputation by chained equations [21] (Supplemental Methods).

Pathway analysis

Consequences of a year with or without failed participations were analyzed (Supplemental Methods).

Imprecision

Imprecision was calculated as coefficient of variation (CV) and as so-called characteristic function [22], [23].

For each sample, robust device-specific central location and scale were calculated using the Huber M-Estimator implemented in the R software package “robustbase” [24]. CVs were established by dividing the scale through the central location. For the overall CVs of devices, the weighted mean of the individual CVs was determined with the inverse of the squared standard error of the estimation as weight.

Additionally, imprecision was estimated using the characteristic function:

The characteristic function (Eq. 1) allows specifying an absolute imprecision α in the lower measuring range close to the limit of detection. The imprecision β for higher values depends on the concentration c and resembles the traditional CV. Parameters α and β were fitted using the Nonlinear Least Squares algorithm with the inverse of the squared standard errors as weights (Supplemental Methods).

Bias

For CL EQAs, the differences between the robust central location of the measured values of a device and the reference method value were regarded as bias. Biases were not estimated in POCT EQAs to avoid distortions by possible non-commutability of samples.

Comparison of lots

In POCT-Instand, participants could voluntarily specify the lot number of their test strip. Differences between the robust central location of the results of different lots for the same device and sample were calculated. The 95% confidence interval of these differences was determined using bootstrapping [25] (Supplemental Methods).

Bias budget

We introduce the “bias budget”, which denotes the maximum allowable bias that is still clinically acceptable for a laboratory test with a given imprecision. Clinical risk assessment was based on the surveillance error grid (SEG), a tool to assess the degree of clinical risk for diabetes patients from inaccurate blood glucose monitors [6]. For imprecision, the previously calculated CVs as well as the fitted parameters from the characteristic function were used. For each true glucose concentration up to 500 mg/dL (27.7 mmol/L), the bias had to be small enough such that for 99.7% (three standard deviations) of all measurements with the given imprecision the total error posed less than a “moderate” SEG risk. The largest bias still meeting these constraints was termed “bias budget”. Similarly, a bias budget was also calculated using insulin dosage errors during a tight glycemic control (TGC) simulation according to Karon et al. [3]. Here, 99.7% of all measurements were permitted to result in a maximum “2-category” (not “dangerous”) dosing error.

POCT measurements failed four times as often in EQAs compared to CL automated analyzer measurements

Between 2012 and 2016, overall more than 1000 individual laboratories participated in each of the four EQA schemes. For schemes run by Instand, there were six distributions per year. POCT-RfB had four and CL-RfB eight distributions per year (Table 1). Two samples were supplied in each distribution. Sixty-five and 54 unique POCT devices were employed in POCT-Instand and POCT-RfB, respectively. Across EQA organizations, nine and 11 devices were often employed in CL and POCT EQAs, respectively, and therefore were used as common device categories.

For evaluation of POCT EQAs, reference method values and device specific consensus values were both used as target values. Instand gradually increased the percentage of device subgroups evaluated according to the reference method value from 1.3% in 2012 to 40.0% in 2016 (Supplemental Figure 1). In POCT-RfB, device-specific consensus values have always been used as target values if the device subgroup was large enough.

In general, success rates were higher in the EQAs for CL automated analyzers. On average, 1%–2% of participants failed in CL schemes, compared to 9%–10% of participants in POCT schemes (Table 1). In CL schemes, the central 95% of all measurements deviated less than 10% from the assigned target values after exclusion of outliers, whereas in POCT schemes, the central 95% slightly exceeded 15% deviation from the target value (Figure 1).

Figure 1:

Relative deviation from assigned value.

The central 95% of the results are highlighted in red. EQA for automated analyzers in central laboratories are displayed with a gray background.

The device had the highest influence on performance

We investigated the influence of different factors on the percentage of “failed”, “poor”, “acceptable” and “good” participations (Supplemental Figures 2–7). Few devices accounted for the majority of all employed devices (Supplemental Figures 2–5). The “others” group subsumes devices that were rarely used or could not be coded otherwise. Especially in the POCT EQAs, this group had a high percentage of failed results. Similarly, we investigated the effect of additional participation in other EQAs (Supplemental Figure 6). A substantial number of laboratories participated in EQAs for both CL and POCT glucose measurements. These laboratories had a higher fraction of “good” participations than laboratories that took part in only one EQA. To assess the influence of a laboratory’s experience, participants were classified as “new” (first participation), “intermediate” (2–10 participations) and “experienced” (;10 participations) [20]. Most participations were submitted by “experienced” participants (Supplemental Figure 7). In all examined EQAs, the fraction of “good” participations was lowest for new participants.

Next, we calculated univariable OR for the probability to reach a “good” result (Supplemental Figures 8, 9). Although ORs for the same devices differed between EQAs, the employed device had the strongest influence in all EQAs.

To compare effect sizes and analyze interactions, multivariable ORs to reach a “good” result were calculated for CL automated analyzers (Figure 2). The choice of a device not in the “others” category increased the probability for a “good” result up to 5.4 times (for “Abbott” device). Large 95% confidence intervals for some devices (up to 1.9–4.5 “Roche Diagnostics” with “other” method) indicate a high degree of uncertainty, at least partially caused by a small sample size. “Experienced” participants had significantly higher odds (OR 1.5) to reach a “good” result than “new” participants. The effect of an additional participation in a POCT EQA was negligible (OR 1.02).

Figure 2:

Multivariable odds ratios for factors affecting the probability to reach a “good” result in CL automated analyzer and in POCT glucose EQAs.

An additional variable modeled distribution-specific variations (not shown). Error bars represent 95% confidence intervals. Methods in squared brackets: HK, hexokinase; GO, glucose oxidase. Numbers on the left indicate participations with the respective factor.

In POCT EQAs, the device had again the greatest influence on the probability to reach a “good” result (Figure 2). ORs ranged from 0.6 (“Bayer Vital Contour”) to 6.2 (“Roche Accu-Chek Inform II”). The effect of additional participation in a CL EQA scheme was greater (OR 1.2) than the reciprocal influence of POCT EQAs on CL EQAs. All ORs denoting past experience included 1 in their confidence interval.

To verify these results, we recalculated multivariable ORs for the likelihood not to fail (Supplemental Figure 10). The spread of ORs was larger but showed comparable tendencies. For 29,561 (out of 67,295) participations, the level of experience could not be determined with certainty as the first participation likely occurred before 2012. These participations were excluded from multivariable ORs calculations. To evaluate their contribution, missing data were imputed and multivariable ORs to reach a “good” result ratios recalculated (Supplemental Figure 11) [21]. The recalculated ORs slightly exceeded the 95% confidence interval of the ORs without imputed data for three and four devices in the CL and POCT EQA, respectively.

After “failed” EQA distributions participants switched devices more often and improved

The worse the result of the previous EQA participation, the larger was the fraction of failing participations (Supplemental Figure 12). After having failed the previous distribution, 11% (CL-Instand), 13% (CL-RfB), 33% (POCT-Instand) and 35% (POCT-RfB) of participants failed again.

For a better understanding of ways to improve performance after failing an EQA participation, a pathway analysis was conducted (Figure 3). After a year with failed participations, laboratories left the EQA or employed a new device more often than laboratories that did not fail in the previous year. Although small, this effect could be observed in all EQAs. Of the participants who failed but continued in the EQA, 14% (CL-Instand), 15% (CL-RfB), 41% (POCT-Instand) and 50% (POCT-RfB) failed again at least once in the following year. For participants who changed their failed POCT device (or the declaration of the same), the percentage of “good” results was higher and the percentage of “failed” results lower than for those who did not change in the following year.

Figure 3:

Development after failed participations.

(A) Pathway of participants after a year with and without failed participations. Number of participants are indicated above bar. EQA for CL automated analyzers are displayed with a gray background. (B) Comparison of the year after a failed participation with and without a new device. Numbers on top of bars depict total number of participations. Too few participants failed in CL EQAs for meaningful analysis.

Certain POCT devices approached the analytical performance of central laboratory instruments

To estimate imprecision, the weighted mean of the coefficients of variation (mCV) was determined for each device (Table 2). The mCV of CL automated analyzers ranged from 0.022 to 0.031 (median 0.026). Some POCT devices reached comparable imprecision. The mCVs of all POCT devices ranged from 0.025 to 0.084 (median 0.055).

The characteristic function was fitted to measured values of each device to determine changes of imprecision over the measuring range (Table 2, Figure 4, Supplemental Figure 13). Most CL devices exhibited small fitted values for parameter α denoting low absolute imprecision in the lower measuring range (overall median 1.15, IQR, 0.69–1.62). For POCT devices, values for α were higher but often exhibited wide confidence intervals (overall median 2.12, IQR 0.20–4.68) (Supplemental Table 1). Parameter β expresses a relative imprecision in the higher measuring range. The smaller the α, the more closely the β matched the respective mCVs for most devices. We used the characteristic function and fitted parameters to calculate CVs at 80 and 300 mg/dL (4.4 and 16.8 mmol/L). In the lower measuring range, CVs were up to 1.6 (CL) and 2.4 (POCT) times larger than in the higher range.

Figure 4:

Relative imprecision over the measuring range modeled using the characteristic functions.

Colored ribbons represent 95% confidence interval determined by bootstrapping. EQA for CL automated analyzers are displayed with a gray background.

Biases were determined for CL EQAs (Supplemental Figure 14). The same device exhibited the highest median bias in CL-Instand and in CL-RfB (Siemens Advia, 2.2% and 1.8%).

Lot-to-lot differences are a serious source of variation for POCT glucose testing

In POCT-Instand, 80 samples were measured with the same device but with at least two different test strip lots. Maximum differences between lots exceeded 5% of the assigned target value in nine samples (11%) (Figure 5). These differences were estimated with wide confidence intervals.

Figure 5:

Maximum difference between robust central locations of different lots used in the same device and for same sample.

Filled dots represent differences >5%. Error bars represent 95% confidence intervals determined by bootstrapping.

The smallest allowable bias, the bias budget, occurs in the lower measurement range for most devices

The bias budget, the maximum allowable bias that is still clinically acceptable for a laboratory test with a given imprecision, was calculated with the previously determined mCVs and parameters from the characteristic function (Figures 6–8). Bias budgets calculated using the TGC simulation as risk assessments were slightly smaller than the respective bias budgets derived from the SEG. They were symmetric for most devices implying that a positive bias carries nearly the same risk as a negative bias. Regardless of whether imprecision was formulated as mCV or as a characteristic function, glucose concentrations in the range 60–115 mg/dL (3.3–6.4 mmol/L) were likely to display an unacceptable bias and determined the bias budget for nearly all devices. The absolute bias budgets of CL automated analyzers were largely homogeneous and ranged from 20 to 27 mg/dL (1.1–1.5 mmol/L) (median 24, IQR: 23–25 mg/dL [1.3, 1.2–1.4 mmol/L]). For two POCT devices (Dytrex (Infopia) Easy Gluco; Lifescan One Touch Vita), the imprecision alone was too high to reach an acceptable risk. Overall, bias budgets for POCT devices were smaller and differences between devices greater (median: 17, IQR: 13–21 mg/dL [0.9, 0.7–1.2 mmol/L]) than for CL devices.

Figure 6:

Schematic calculation of the bias budged based on the data from EQAs and on risk assessment with the Surveillance Error Grid.

Gray overlay represents 99.7% (three standard deviations) of measurements given the imprecision. Black dotted lines represent the border between acceptable and unacceptable errors. Two blue bars represent the allowable positive or negative bias, the “bias budget”.

Figure 7:

Bias budget based on the surveillance error grid for risk assessment.

EQA for automated analyzers in central laboratories are displayed with a gray background.

Figure 8:

Bias budget based on the tight glucose control simulation for risk assessment.

EQA for automated analyzers in central laboratories are displayed with a gray background.