Assessment of drug delivery devices working at microflow rates

Instruments under test

Syringe pump

Syringe pumps are motor-driven pumps that use one or more syringes to provide a steady flow rate or to administer a certain volume (bolus) of drugs to a patient, see Figure 1. These pumps are manufactured in a wide variety and are used with disposable plastic syringes or with reusable glass or metal syringes. There is a wide variation in the quality and the flow rate ranges offered by these pumps. In the EMRP JRP 18HLT08 MeDDII, a syringe pump was tested in a flow range from 0.01 to 30 mL/h with disposable plastic syringes. Two syringe sizes were used for the tests in this project, specifically a 10 and a 50 mL syringe, which are mostly used in clinical applications.

Figure 1:

Syringe pump for use with disposable syringes.

The motor in a syringe pump drives a mandrel that pushes the plunger (piston) into the syringe. Depending on the quality of the stepper motor, the mandrel, etc. a more or less uniform movement of the plunger is achieved. Enormous flow rate fluctuations can be observed solely due to imperfections of the mandrel.

Infusion device analyser

Infusion devices analyzers (IDA), Figure 2, are used to analyse the performance of a variety of infusion pumps. They measure both average and instantaneous flow rate and also check the occlusion alarm on the infusion devices by measuring the occlusion pressure. IDAs are often used by users or maintenance officers in the maintenance department of hospitals to check the performance of a drug delivery device. These devices are usually calibrated by the manufacturer before they are sold, and some of these manufacturers are not accredited to perform these calibrations. Furthermore, in many cases, subsequent calibrations are not considered and there is no documentation explaining how to perform them, so that the metrological traceability of the IDA cannot be established.

Figure 2:

Infusion device analyzer (IDA).

To maintain the traceability chain of an IDA, its calibration should be performed by a recognized, accredited laboratory or National Metrology Institute (NMI) using well-defined calibrations procedures. In order to develop and define these procedures in detail, several tests need to be performed.

Calibration methods

The following sections describe the methods used to calibrate and test the different medical infusion devices evaluated in this project. Several partners participated in experiments. Mainly, Danish Technological Institute (DTI), Portuguese Institute for Quality (IPQ), Czech – Metrology Institute (CMI), Federal Institute of Metrology (METAS), Research Institutes of Sweden (RISE), Korea Research Institute of Standards and Science (KRISS), and TUV SUD NEL.

A more detailed description can be found in the study of Graham et al. [11].

Flow error determination

According to the International Vocabulary of Metrology (VIM) [15], the absolute measurement error is the measured quantity value minus a reference quantity value.In contrast, according to the standard IEC 60601-2-24 [16] the absolute measurement error is defined as reference value minus the measured quantity value. Expressing the relative error in formulas according to the above-mentioned definitions and referring it to the case of a drug delivery device calibration, where the measurand is the flow rate of the delivered drug, the following formulas are obtained:

where:

1. A_Metro is the relative flow measurement error or systematic error as defined by VIM [15].

2. A_med is the relative flow measurement error or systematic error as defined in the standard IEC 60601-2-24 [16].

3. Q_ref is the reference flow rate determined by the reference measurement method (e.g., gravimetric method).

4. Q_set is the flow rate set or the indicated flow rate at the instrument under calibration (e.g., 1 mL/h).

These formulas are used in the later part of this paper to define the measurement error.

Tests of a syringe pump

In case of the syringe pump, the focus was on several points that are particularly relevant for testing the instrument by a manufacturer or calibration laboratory. First, it was investigated whether the accuracy of flow rate generated by the pump depends on the liquid used. Calibrations with water, saline solution and four typical drugs have been compared using the gravimetric method. The next test focused on how the flow rate generated by the pump depends on plunger position in the syringe, showing whether a test where the pump is not emptying the entire syringe could be representative for any volume of the syringe. In the next test, the performance of the pump was examined at extremely low flow rates approaching the lower limit of the pump specifications. Finally, two approaches for analysing the short-term variability of the syringe pump flow rate have been compared – one according to the standard IEC 60601-2-24 [16] and one according to the recently published technical information report (TIR) AAMI TIR101:2021 [17]. All tests have been performed using a BBraun Perfusor Space syringe pump.

Syringe pump performance for various liquids

It can be assumed that testing syringe pumps with water, as prescribed in the standard IEC 60601-2-24 [16] is representative for all liquids used in practise. The use of syringe pumps with different liquids may results in different pump performance due to mechanical (viscosity, density) or chemical properties of the liquids. For example, the liquids may change the resistance in the connecting tubing or the friction between the plunger head and the syringe wall.

In this study, six liquids with similar mechanical properties but varying chemical composition were selected, namely: distilled water, NaCl solution, dobutamine, dopamine, propofol and gelaspan. Two laboratories participated in the tests – IPQ and KRISS. The specifications of the liquids used by the two laboratories are listed in Table 1. The syringe pump with a 10 mL syringe was calibrated with a set flow rate of 1 mL/h. The average error of flow rate over a certain time has been determined and compared for various liquids. The tests for all six liquids have been performed at IPQ with averaging time of 2 h, the tests for the first four liquids have been performed at KRISS with an averaging time of 30 min. All test were performed only once.

Each liquid was tested using a new disposable syringe. When comparing calibration results obtained with different syringes of the same type, one should be aware that the variability of the syringe diameter may affect the result. As described in the standard ISO 7886-2:2020 [18], the maximum allowed tolerance for the inner diameter of a 10 mL syringe is ±1% resulting in a tolerance of ±2% for the cross-sectional area of the syringe and consequently the same tolerance for the flow rate. However, tests with several syringes have showed that an additional flow rate uncertainty of ±1% is sufficient due to syringe variability of the 10 mL syringes (Figure 2).

The results are summarised in Figure 3. The percentage flow rate deviations are calculated according to the equation (2), medical errors.

Figure 3:

Flow rate deviation from its set value for a syringe pump using various liquids.

The uncertainty bars in Figure 3 represent an expanded uncertainty with a confidence level of 95%, combined with the 1% uncertainty contribution due to the variability of the syringe diameter, which is in fact the main component of the total uncertainty value.

As can be seen, the error differences between the individual liquids in both laboratories do not exceed 1% and are thus smaller than the stated uncertainties. To verify statistically that there is conformance between the results the normalized errors (E_n value) were calculated, showing that all values were below one (<1), thus confirming conssistency. The E_n values were calculated according to the procedure described by Cox [19].

It can be concluded that there is no significant difference in the average flow rates generated by the syringe pump for the six liquids tested.

Flow rate as a function of plunger position in a syringe

A test was carried out at RISE to determine the deviation of the flow rate depending on the plunger position of the syringe. Testing a syringe pump over the entire stroke length can be very time-consuming, especially at low flow rates. If one selects a syringe with a nominal volume of 50 mL, a test at a flow rate of 1 mL/h will take (about) 50 h from full to empty syringe. Therefore, the question arises whether a shortened test using only a certain section of the syringe length provides a representative result for the entire syringe length.

A syringe with a nominal volume of 50 mL was used, which was divided into ten segments corresponding to the plunger positions of 0–5 mL, 5–10 mL, … , 45–50 mL. The corresponding segments are numbered from 1 to 10. In each segment, an average flow rate deviation has been determined for three values of flow rate set in the syringe pump, i.e., 1, 10 and 30 mL/h. The percentage deviation of flow rate from its set value is again calculated according to equation (2).

The result of this test is summarised in Figure 4. In this figure it can be seen that the difference between the maximum and minimum values of the flow rate deviations is in the order of 1.4% for the lowest set flow rate of 1 mL/h. For the higher flow rates of 10 and 30 mL/h, the range of deviations in both cases does not exceed 0.5%. It can be concluded that the selection of a syringe segment the performance test represents its entire length or syringe volume with an accuracy of 1.4% or better for the flow rates tested. It can also be seen from Figure 4 that the lower the flow rate the larger the variability and that means that in such cases a smaller syringe should be used.

Figure 4:

Deviation of flow rate as a function of plunger position. The straight horizontal lines indicate the flow rate deviation averaged over the entire tested syringe volume. The manufacturer’s specification of the flow rate accuracy limit is also included.

Extremely low flow rates

Extremely low flow rates of 0.01^[1] and 0.1 mL/h have been tested with the 10 mL syringe, which approaches the lower flow rate limit of the syringe pump according to the manufacturer’s specifications. Average flow rates have been measured by three laboratories over the time periods summarised in Table 2. The percentage deviations of the average flow rate from the set point calculated according to the equation (2) are shown in Figure 5, including the error bars representing the expanded uncertainties at the confidence level of 95% as reported by the laboratories. The uncertainties also include the 1% contribution resulting from the variability of diameters of the syringes used. Figure 5 shows that the results of all laboratories are consistent when the stated uncertainties are taken into account. This is a confirmation that different test procedures and uncertainty assessments used for the gravimetric measurements of the participating laboratories lead to consistent calibration results.

Figure 5:

Comparison of calibration results for average flow rate obtained in various labs. The bars represent the expanded uncertainty of the resulting flow rate deviation.

To verify that the results are statistically consistent the normalized errors (E _n value) were calculated, showing that all values were below 1.2. (<1.2), thus confirming consistency. The E _n values were calculated according to the procedure described by Cox [19].

In addition to the long-time average flow rate values, it is also interesting to look at the dynamic behaviour of the syringe pump at low flow rates. At RISE, the actual balance readings have been logged every second during the tests. The data was used to calculate 5 s averages of the reference flow rate and their dependence on time, which is shown in the left part of Figure 6. A pulsating character of the flow can be observed in the figure. Averaging over a long period of time (Table 2) leads to small deviations of the resulting average flow rate from the setpoint. However, reducing the averaging time would possibly result in larger deviations. To quantify this effect, the trumpet curve method of the standard IEC 60601-2-24 [16] can be used. Taking an averaging period of duration P (observation window) and shifting the start of this averaging period to various phases of the pump cycle we obtain varying values of the average flow rate and therefore also varying values of its deviation from the set flow rate value. The minimum of such deviation for a given value of P is denoted as and similarly the maximum is denoted as (we follow the notation of [16]; for details see also the following section of this paper). If the limits and are plotted as functions of the observation window duration P, the so-called trumpet curves are obtained. The trumpet curves for observation windows with a duration from 2 to 31 min are shown in the right panel of Figure 6. The average flow rate variability given by the range width E P   max − E P  min exceeds 10% for averaging times below approx. 5 min at 0.1 mL/h and below approx. 25 min at 0.01 mL/h. To achieve the flow rate variability of less than 4% required to meet the accuracy specification of ±2%, approx. 18 min averaging time for 0.1 mL/h and 80 min for 0.01 mL/h are required.

Figure 6:

The left part of the plots shows flow rate (5 s average) as a function of time for 0.01 and 0.1 mL/h. The right part shows limits of deviation of the average flow rate from the set value as functions of the observation window duration (trumpet curves).

Short-term flow variability – comparison of trumpet curves and PK-CV curves

The mechanism of syringe pumps causes oscillatory behaviour of the generated flow rate. These oscillations should be characterised and their potential impact on a patient should be under control. The IEC 60601-2-24 standard [16] prescribes characterisation of syringe pumps in terms of the so-called trumpet curves which give the maximal and minimal deviation of an average flow rate from the set value for given durations of the averaging period. Recently a new way of characterisation was proposed in the document AAMI TIR101:2021 [17] which focuses more on the impact on a patient and takes a pharmacokinetics model into account with a drug half-life (decay time) as a parameter. The purpose of the following paragraph is to compare the two approaches and to show how their outputs are related each other using the data obtained during the syringe pump tests.

Both assume that a syringe pump is tested with a gravimetric standard, i.e., the liquid from the pump is delivered at a certain flow rate into a beaker which is placed on a balance. The consecutive readings of the balance W _i are taken with a time step S, where the index i goes from 0 to n, where n+1 is the total number of readings during a selected test period T.

The approach of the IEC 60601-2-24 standard [16] is to evaluate possible variations of an averaged flow rate, taking the average value over period of time P (observation window), and to determine how these variations change with the change of P. The variations are expressed in terms of minima and maxima of the relative deviations of the averaged flow rate, which, plotted as functions of P, lead to the trumpet curve graph.

More precisely, according to the IEC 60601-2-24 standard [16], the volume flow rates are calculated over the sample periods of duration S as follows:

where ρ is the liquid density. Then the duration of an observation window p is selected and the average relative deviation of the flow rate from a set flow rate value Q_set is calculated over the observation windows of the duration P, which are consecutively shifted over the entire test period T with a step S, i.e., we obtain the following series of average relative deviations for a given value of P:

where m is the number of possible observation windows of duration P given as m = ( T − P ) / S + 1 . Then, from the series of deviations in equation (4) for given P a maximum, minimum and standard deviation can be calculated, i.e.

The values of minima and maxima as functions of P then define the trumpet curve. The standard deviation is later used for comparison with the PK-CV curves according to TIR101:2021 [17].

The philosophy of the AAMI TIR101:2021 [17] is different. It models a patients’ drug consumption in a simplified way, using a single-compartment pharmacokinetics model. The result is a variation of the drug level in the patient’s body rather than the variation of the pump flow rate itself. A brief description of the procedure is given below. Further details of the method can be found in the report [17].

The rate of consumption of a drug is assumed to be proportional to the amount of drug in a compartment (part of patient’s body). Denoting v as the drug volume in the compartment, the following equation applies d v dt = − k e v + Q , where k _e is a constant and Q is the pump flow rate. The constant k _e refers to the drug half-life T _D (also called decay time in [17], since k e = ln ( 2 ) / T D . The AAMI TIR101:2021 [17] then defines a factor B for the decrease in the amount of drug over a sampling period S as:

and models a time evolution of the drug volume in the compartment by a discrete recursive series v _i , which is related to the measured mass output series of the pump W _i as follows:

where the initial volume v 0 is given as v 0 = S . Q A ( 1 − B ) , where Q _A being an average pump flow rate over the entire test period. The volume v₀ is an estimation of the equilibrium volume of the drug in the compartment at a hypothetical constant pump flow rate Q _A . The variation of the drug level in the compartment is then expressed by a relative standard deviation of the v _i values, called a coefficient of variation CV, i.e.

The coefficient of variation CV is a function of the decay time T _D and the flow rate set in the syringe pump. In the technical report [17] it is recommended to plot the CV dependencies on the flow rate for several values of the decay time. However, in this paper the CV dependencies on the decay time are plotted for several values of the flow rate to allow comparison with the trumpet curves.

The mass-time data W _i for this test have been obtained using the gravimetric facility of CMI with a sampling time S of 1 s. Syringes with nominal volumes of 10 and 50 mL have been used and two flow rates (1 and 10 mL/h) have been tested.

Figure 7 shows the resulting trumpet curves given by maxima and minima of E _P and the comparison of the curves of the standard deviation of E _P with CV curves. The IEC 60601-2-24 standard [16] recommends calculating the trumpet curves for observation windows of 2, 5, 11, 19 and 31 min. In this analysis, the trumpet curves and the standard deviation curves of E _P were calculated for observation windows P ranging from 2 to 31 min, with window duration increased by 30 s step to see the oscillating dependencies on P in more detailed resolution. The CV curves are given for a range of decay times from 0.8 to 12.4 min with a step of 12 s, i.e., the range and step are scaled by a factor of 2.5 compared to the observation windows. The reason for such scaling is justified below.

Figure 7:

Trumpet curves given by maxima and minima of E_p and comparison of curves of standard deviation of E_p with CV curves for 2 syringes and 2 flow rates. The cases are ordered according to the plunger speed from the slowest to the fastest.

Figure 7 shows that the CV curve is much smoother than the curve of the standard deviation of E _P generated from the same mass-time data W _i . It can also be seen that the short-term instability in the movement of a pump plunger, which leads to certain variability in the average flow rate of the pump over an averaging period P, results in approximately the same variability in the volume of a delivered drug with half-life of P / 2.5 . The “variability” here is expressed as the standard deviation of the average flow rate or drug volume in a compartment. The factor 2.5 was estimated by matching the standard deviation of E _P and CV curves by purely visual comparison.

Tests on infusion device analyzer

In order to determine the best calibration procedure for both flow and pressure of an Infusion Device Analyzer (IDA), several tests were performed with the Fluke IDA-1S, mainly: determination of metrology flow rate error, use of different calibration liquids, different acquisition times (volumes), reproducibility, use of different calibration methods. The uncertainty components involved are also explained.

Different methods can be used to calibrate the IDA with respect to flow rate. The most common method is to use a high-quality reference syringe pump, Figure 8. This pump must be calibrated against a primary standard to ensure traceability to the SI units. Other calibration methods can also be used to calibrate the IDA, such as a calibrated reference flow meter or direct gravimetric measurements, although, the latter may not be the best option as the outlet of the IDA is not a continuous flow. All methods were used in this work.

Figure 8:

Calibration method with a high-quality syringe pump as reference.

Metrology flow error determination

The metrology flow error was determined by six laboratories at different flow rates: 0.5, 1, 5, 10, 25, 50 and 100 mL/h. The results can be seen in Figure 9.

Figure 9:

Flow error determination of the IDA.

All calibrations performed by the laboratories were based on the method with a syringe pump as reference. It was verified that the calibration results obtained by the six laboratories are all consistent as they are all within the uncertainties claimed by the laboratories (vertical bars). The uncertainty variability is mainly due to the acquisition time and the used reference standard. The largest deviations between the laboratories are observed at the lower flow rates. The uncertainty components are described below.

The first value obtained by the IDA should be discarded, which the device does automatically.

Acquisition time (volume)

The manufacturer recommends performing tests with a volume delivery of 10 or 20 mL, depending on the flow rates tested. At very low flow rates, the measurements take many hours, which may not be feasible for a calibration laboratory. Therefore, the tests were performed at different flow rates, and thus different acquisition times. The used volume was 20 mL. At lower flowrates, 1 and 0.5 mL/h the uncertainties are 1 and 2% respectively due to the poor display resolution (two decimal places).

The results are presented in Figure 10.

Figure 10:

Flow stability test regarding volume delivery.

As it can be seen from the figure, the deviation (measurement error) of the IDA for 2 mL or 3 mL volume measurements is already stable for all flow rates. This means that the measurement time can be shorter than recommended by the manufacturer.

Calibration with different liquids

To test the influence of the liquid type on the performance of the IDA, different liquids were used to calibrate the IDA at a flow rate of 1 mL/h. The liquids used were water, dobutamine, dopamine and a saline solution. The liquids were chosen as they are the most commonly used in hospital environment.

Figure 11 shows that the calibrations of the IDA with different liquids are consistent within the claimed uncertainties (vertical error bars). The uncertainty values are similar for all four liquids. It can be concluded that the IDA is not affected by the liquid properties.

Figure 11:

calibration of the IDA with different liquids.

Reproducibility

To determine the reproducibility of the IDA, a laboratory performed a calibration of the flow rate on different days. The results are presented in Figure 12.

Figure 12:

Reproducibility tests.

It can therefore be stated that the device is very reproducible. The deviations between the calibrations are very small. Above 10 mL/h, the largest deviation is 0.15%, which is less than the uncertainty of the calibration instrument of 0.61%. Below 10 mL/h, the deviation increases to 1%, but is still within the claimed uncertainty of 2.59%. At the lower flowrates, 1 mL/h and 0.5 mL/h the uncertainties due to the poor display resolution are 1 and 2%, respectively.

As expected, the reproducibility is lower at smaller flow rates. However, the deviation is still below the uncertainty, so that the number of repetitions can be reduce to a minimum.

Use of different calibration methods

One of the laboratories in the project group calibrated the IDA using three calibration methods at the same time. The tests were performed with a syringe pump that delivered water through a Coriolis flow meter upstream of the IDA and a balance downstream the IDA to collect the liquid. A glass ‘frit’ ensures that a constant flow can be generated without any disturbances caused by dripping. The results are shown in Figure 13.

Figure 13:

IDA calibration with different methods.

It can be observed that the three methods are consistent at higher flow rates, as the values are within the claimed uncertainties given by the laboratory. At lower flow rates, as expected, there is a large variability, also the syringe pump method is more stable over the entire range.

Syringe pump tests

The syringe pump is an important instrument in every hospital for delivering liquid to patients. In this work, several tests were conducted to characterise the flow rate behaviour of a specific BBraun Perfusor Space syringe pump using the gravimetric reference method in different laboratories.

The average flow rates generated by the syringe pump have been compared with the syringe filled with different liquids: distilled water, a saline solution, dopamine, dobutamine, propofol and gelaspan, at a set flow rate of 1 mL/h. No significant difference was found in the resulting flow rates of the different liquids. This shows that the test with distilled water, as specified in the IEC 60601-2-24 standard [16], is representative for the group of liquids considered here.

The average flow rate generated by the syringe pump was then measured for different plunger positions in a 50 mL syringe. These tests were carried out to verify whether it is possible to restrict the flow rate tests to specific syringe segments that are representative for the entire syringe volume. The syringe was divided into ten segments and it was found that the maximum deviation from the generated average flow rates across all the segments was about 1.4% for the lowest flow rate of 1 mL/h.

The performance of the syringe pump and gravimetric reference methods have been tested for extremely low flow rates of 0.1 and 0.01 mL/h, which is close to the lower range of the syringe pump as specified by the manufacturer. The three participating laboratories achieved good agreement in the average flow rate measured, demonstrating the metrological validity of their methods. The pulsating nature of the flow rate generated was visualised and it was shown that averaging times of at least 80 min for 0.01 mL/h and 18 min for 0.1 mL/h are required to achieve an average flow rate accuracy within ±2%.

In the last test, two methods for assessing short-term flow variability were compared. The first is based on the IEC 60601-2-24 standard [16], which defines the so-called trumpet curves. The second is based on the recently published technical information report AAMI TIR101:2021 [17], which uses a single-compartment pharmacokinetics model to define an alternative methodology for evaluating the effect of flow rate instability. It was shown that the predictions of the “flow rate variability” of the first approach (in terms of standard deviation of an average flow rate instead of its minimum and maximum) and the “compartment volume variability” of the second approach are quantitatively similar when the “observation window” is about 2.5 times the “drug decay time”.

IDA tests

The IDA is an instrument mainly used by hospitals to verify the performance of syringe pumps and peristaltic pumps. The traceability of this instrument is still a problem for most users due to the lack of information on subsequent calibration procedures in most European countries. The main objective of this work was to provide some relevant information about the use and performance of an IDA. Several tests were carried out: determination of flow rate error, use of different calibration liquids, different acquisition times (volumes), reproducibility, use of different calibration methods and detail description of the uncertainty components. The instrument was found to be repeatable and reproducible in flow measurements. It is possible to reduce the measuring time without compromising the accuracy of the measurements, the IDA is not affected by the properties of the calibration liquid, the use of a reference syringe pump method is recommended. The determined uncertainty values and errors are within the accuracy of the IDA, even at lower flow rates where largest errors were found.

Calibration of the IDA in an accredited laboratory ensures traceability to a commonly agreed standard/procedure. For flow rates, these are essentially mass and time. This traceability is transferred to the infusion device e.g. a syringe pump, when it is calibrated with the IDA as the reference. This is called the metrological infrastructure and ensures that devices used in a clinical environment operate within a documented margin of error (maximum permissible error) and uncertainty. In some cases, the uncertainty can be higher, e.g., general adult anaesthetics, than in others, e.g., treatment of premature infants. However, regardless of the intended use of the infusion device, it is always important to know the level of uncertainty.