Are hemoglobin A_1c point-of-care analyzers fit for purpose? The story continues

Analyzers evaluated

The six POC analyzers included in this study and their key characteristics are summarized in Table 1. The choice of analyzer was both manufacturer led (Allegro, QuikReadgo and The lab 001) and investigator led (Cobas B101, InnovaStar and A1Care). The manufacturers of the last three analyzers were approached by the authors as previous evaluations had highlighted some performance issues. An initial familiarization protocol was undertaken with each instrument and the results were shared with the manufacturers to enable them to decide if they wished to continue to a full evaluation. This supports a collaborative approach to working with manufacturers with the aim to improve quality. In some cases when a product is new in development, feedback at an early stage will enable further development before a product is brought to full evaluation, saving time and resources [11]. Six analyzers were fully evaluated, however two new analyzers were not ready yet for a full evaluation.

Imprecision study (EP-5 and EP-15)

The Clinical and Laboratory Standards Institute (CLSI) EP-5 protocol was used to investigate assay imprecision. It is known that some POC methods have a bias with frozen material and as it is not known if frozen samples have an impact on the imprecision, also EP-15 was performed with two fresh patient samples (HbA_1c values of 48 and 75 mmol/mol). Both samples were analyzed five-fold for five days. Imprecision was also calculated on the basis of the duplicates of the fresh patient samples in the EP-9 protocol.

Method comparison (trueness; EP-9)

The CLSI EP-9 protocol was performed with 40 fresh patient samples and the data was used to investigate the bias between each instrument and four SRMPs (n=40, eight samples per day for five days, duplicate measurements). Values were assigned with four IFCC and NGSP Certified SRMPs [12], [13]:

1. Roche Tina-quant Gen.3 HbA_1con Cobas c513, immunoassay, IFCC and NGSP certified (Roche Diagnostics);

2. Premier Hb9210, affinity chromatography HPLC, IFCC and NGSP certified (Trinity Biotech);

3. Tosoh G8, cation-exchange HPLC, IFCC certified (Tosoh Bioscience);

4. Abbott Enzymatic method on Alinity, IFCC and NGSP certified (Abbott Diagnostics).

Imprecision (EP-5 and EP-15

Table 2 displays the CVs derived from both EP-5 and EP-15 and the duplicates from the EP-9 protocol. Only the Lab 001 and the InnovaStar achieved the performance criteria of <3% CV (SI units) (<2% in NGSP units) across all protocols and both high and low HbA_1c levels [21], [22]. The Lab 001 actually achieved <2% CV (SI units) showing very low levels of imprecision. However the Allegro failed to achieve <3% CV (SI units) in any protocol at either level, with CVs as high as 4.2%, showing unacceptable levels of imprecision. The A1Care had mixed results, performing better with fresh samples and at higher HbA_1c values. The B101 also had mixed results, with better performance at higher HbA_1c values. The QuikReadgo met the criteria in both EP-5 and EP-15 with little difference between fresh and frozen samples. However, the performance with the duplicates from EP-9 was mixed with one lot failing and one lot passing.

Table 2:

Imprecision results based on EP-5, EP-15 and on the duplicates in EP-9.

1. Red: fail performance targets (CV <3% in SI units and <2% in NGSP units). ^aBased on the duplicates in EP-9.

Method comparison (trueness; EP-9)

Table 3 details the results of the method comparison study and also NGSP pass/fail rate. From these data it is clear that all analyzers suffered some degree of bias, with some such as the Allegro showing this across multiple levels and between lot numbers. The data for the individual POC analyzers vs. the individual SRMPs in NGSP units is available in Supplemental Table 1. From this table and Table 3 it can be seen that only the Cobas B101 passed the NGSP criteria with two lot numbers compared to all four individual SRMPs and that the QuikReadgo and the Allegro failed the NGSP criteria with both lot numbers for all four individual SRMPs. Figure 1 shows the regression lines for each POC device vs. the mean of the SRMPs. All analyzers suffered some degree of bias. The Lab 001 and the Cobas B101, had the least bias across the lots and the HbA1c range and all other POC analyzers had a statistically significant difference either between the lot numbers or at different HbA_1c levels or both and showed a large dispersion around the deming regression line compared to the mean of the SRMPs.

Figure 1:

EP-9 data for each of the POC analyzers when compared to the mean of the SRMPs with two lot numbers.

(A) The A1Care, (B) the Lab 001, (C) the Cobas B101, (D) the QuikReadgo, (E) the InnovaStar, (F) the Allegro.

Linearity (EP-6)

Supplemental Table 2 details the results of the linearity study. The maximum deviation is shown between the fitted values of the best polynomial line and the regression line for the 11 samples. If the deviation exceeds allowable nonlinearity (3 mmol/mol) the data was considered nonlinear. Based on this criterion all POC analyzers were linear except for the Cobas B101 and the InnovaStar. However, the detection limit of the InnovaStar was >30 mmol/mol. Excluding the lowest sample for the calculations showed that the InnovaStar was linear. The HbA_1c result of the highest sample for the Allegro was above the detection limits (>130 mmol/mol) therefore the linearity was assessed with 10 samples instead of 11.

Hemoglobin variants AS, AC, AE, AD, elevated A2 and HbF

Table 4 shows the mean relative difference of the frozen and fresh Hb-variants samples and Supplemental Figures 1–6 show the graphs of the interference of Hb-variants for frozen and fresh Hb-variants for the different POC analyzers. All methods, except for the A1Care, had an interference with one or more of the Hb-variants with frozen or fresh samples (mean relative difference was >10%). All Hb-variants were detected and correctly identified by the Lab 001 via an S-, C-, D-, E- or F-window.

Table 4:

Mean relative difference (%) of the common Hb-variants compared to the assigned value after correction for bias in non-variant samples (number of samples for frozen and fresh samples).

1. Red: equals at or near 10% difference. Green: this variant would not be seen to affect the value if only evaluated using frozen samples. The manufacturer’s claims for are also listed. ^a% of HbF at which a significant negative bias results.

Schiff Base, icteric samples, different hemoglobin concentrations

None of the POC analyzers showed an interference for Schiff Base, icteric samples or different hemoglobin concentrations. Supplemental Tables 3–5 show the data.

Sample commutability

To investigate the impact of sample type in relation to different clinical applications, fresh (EP-15 and EP-9), frozen (EP-5 and IFCC certification program samples) and lyophilized (ERL EQA scheme) samples were compared. Figure 2A, B shows the data from EP15 and EP-9 (red circles) representing all fresh samples, IFCC certification samples (blue circles) representing all frozen samples and the ERL EQA (green circles) representing lyophilized samples. In addition, Figure 2B shows EP-5 and EP-9 data (purple circles) in order to assess the impact of using EP-5 vs. EP-15 for routine method evaluations. The EP-5 and EP-15 studies were both used to compare performance with fresh and frozen samples and for the majority of analyzers the EP-15 evaluation provided sufficient data to assess performance.

Figure 2:

Sigma metrics graphs showing the impact of different sample types on the ability to meet the International quality performance criteria.

Figure (2A) shows EP-15 and EP-9 (red) values compared to frozen (blue) and lyophilized samples (green). Figure (2B) shows EP-5 (purple) data compared to frozen and lyophilized samples showing minimal difference in performance between EP-5 and EP-15 protocols. A1Care (A), Cobas B101 (B), InnovaStar (C), The Lab 001 (D), QuikReadgo (E) and Allegro (F).

The data clearly demonstrates that lyophilized material was only commutable with The Lab 001, all other POC analyzers showed a large positive bias when analyzing lyophilized material. Frozen material was not commutable with the InnovaStar and the QuikReadgo. When using fresh patient samples all, except the Allegro, passed the IFCC criteria of having a sigma >2 at an HbA_1c concentration of 50 mmol/mol. Conversely the Allegro actually performed better when using frozen samples instead of fresh samples.

SUS scores

Table 1 shows the SUS scores of the different POCT analyzers. The usability of all the POCT analyzers was good to excellent (mean SUS score > 80) except for the QuikReadgo (mean SUS score was 60).

What progress has been made?

In the study of 2014 the InnovaStar showed an interference with fresh patient samples, which was likely due to the instrument being calibrated using frozen samples [23]. The previous publication led the manufacturer to switch to fresh patient samples, which are available from the ERL, to calibrate their cartridges resulting in lower bias in fresh samples [24].

The Lab 001 device is new to the market and the sigma graphs show excellent performance however there was a small bias at higher HbA_1c levels. Paradoxically, had the imprecision of the instrument been higher than the bias would not have been detected as the confidence intervals would be wider. This device shows that the field of POCT has moved on and quality improvements are possible.

There are still significant issues with the performance of some analyzers

The key issues we still see are: a) lot to lot variability, b) high imprecision and c) significant interference from variants. Four out of the six evaluated analyzers still do not demonstrate acceptable and/or consistent performance.

The InnovaStar had poor performance between lot numbers that was not acceptable. The new to the market QuikReadgo also showed a statistically significant difference between the lot numbers, and failed to meet the NGSP criteria with either lot number when compared to any of the four SRMPs. The Allegro also showed similarly poor performance. It should be noted that both the Allegro and A1Care were evaluated in a previous study (data not presented) in which the results were acceptable, to good. As the data for certain elements of the current study was acceptable it is likely that there is an inconsistency in the manufacturing chain that needs to be identified. The considerable variability in performance across the analyzers, albeit less than in earlier studies, shows that there is still work to be done [6].

A key issue with POC analyzers still appears to be interference from variant hemoglobins. A complicating factor when evaluating Hb-variants is the fact that some analyzers (QuikReadgo and Innovastar) are not compatible with frozen samples. This study addresses this issue with the use of fresh Hb-variant samples. However it was not possible to obtain as wide a range or number of fresh Hb-variant samples for investigation as would be desirable. Explaining the interferences seen in these analyzers, which are nearly all immunoassay, is difficult, why would frozen and fresh samples perform so differently? Why are they causing an interference at all when the epitopes that the antibodies bind to do not contain the mutations that cause the Hb variant? HbE for example is an inherited single base mutation at codon 26 of the beta-globin gene, leading to substitution of glutamic acid (46 amino acid of the beta chain) for lysine which should not, theoretically, interfere with the antibodies used in the POC analyzers. A possible explanation could be that the mutation causes folding of the hemoglobin molecule in such a way that position 46 are then very close to the first four amino-acids of the beta chain and interacts with the antibodies used in the immunoassay [25]. Alternatively this may be due to differences in the immobilization of the antibodies which lead to differences in the surface chemistry and thus the binding of the antibody.

The Lab 001 suffers from interference with fresh HbAS samples, but less so with frozen samples which are easier to obtain for method development. This does potentially pose a problem for patient samples, however this is mitigated by the fact that Lab 001 as a capillary electrophoresis method is capable of identifying the presence of a variant – unlike most other POC analyzers.

It is important to clarify that some of the manufacturers do clearly state that the presence of variants may alter the HbA_1c results however these claims and the findings of this study do not always correlate.

It is not all about analytical performance

Whilst many evaluations focus on analytical performance, it is important to consider the wider context of the use of POC analyzers. The usability/user-friendliness of each of the analyzers was assessed and found to be variable.

A crucial factor in the practical, clinical usability of an instrument is how long it takes to generate a result, with the benefit of providing real time results often touted as a key selling point for POC analyzers. The time from a ‘cold start’ (turning the power on and warming reagents if needed) to a result was assessed. The range of time needed was wide at ̴3.0 min for The Lab 001 to ̴16.5 min for the InnovaStar. This is important information for users of the analyzers as they may have little advance warning that a test may need to be undertaken.

Key messages

This complex and detailed evaluation provides a comprehensive overview of six HbA_1c POC analyzers. Whilst there are areas of excellence in performance there are still significant areas for improvement with the performance of some being unacceptable. It is possible for an analyzer to meet certification criteria for the IFCC and/or NGSP and perform well in one evaluation and then perform very poorly in subsequent evaluations. From a clinical and scientific perspective this is alarming. It is essential that performance of an analyzer is stable, especially with increased use for both monitoring and diagnosis of people with diabetes. Four of the analyzers in this study showed highly variable performance which is not acceptable.

It is unclear why such discrepant results are seen when fresh or frozen samples are used, especially as this is not commonly seen with routine laboratory analyzers. Whilst many evaluations and method development often necessitate the use of frozen samples, it is rare that a POCT would be used with anything other than fresh samples. We have shown here that there can be marked differences in performance with each sample type.

One way to identify variability in performance is through the use of EQA schemes. The authors strongly advocate the use of EQA to identify ongoing performance issues, and although POC analyzers are often exempt from the need to participate in EQA it is a valuable and powerful tool for monitoring performance. A caveat to this is that POC analyzers may not be able to utilize the frozen or lyophilized samples often used in EQA schemes. None of the manufacturers claim in their information for users that lyophilized samples can be used and this is supported by the data from the ERL-EQA samples (see Figure 2), EQA program leads need to be cognoscente of this issue and work towards providing commutable samples for POC analyzers.

As discussed earlier, the interference from Hb-variants in a number of the analyzers is perplexing. The disparity in results seen between fresh and frozen samples is of concern as many manufacturers will likely develop their methods using frozen samples but in the ‘real world’ setting where fresh samples are used, the variants pose a potential unseen problem. Not all manufacturers are accurate in their claims for Hb- variant performance.