Improvement of the post-analytical phase by means of an algorithm based autoveri ﬁ cation

Objectives: Autoveri ﬁ cation (AV) is releasing laboratory results using prede ﬁ ned rules. AV standardizes the veri ﬁ - cation of laboratory results, improves turnaround time (TAT), detects errors in the total test process, and enables e ﬀ ective use of laboratory sta ﬀ . In this study, we aimed to evaluate the outcomes of implementing the AV in a tertiary hospital. Methods: The study was performed in Gazi University Health Research and Application Hospital, Core Biochemistry Laboratory, between August 2017 and October 2019. Step by step, AV algorithms were designed and implemented via middleware for 29 clinical biochemistry tests. A comprehensive validation was performed before the AV system was run. Initially, AV system was tested with datasets and simulated patients (dry testing). Next, samples that may violate AV rules were tested anonymously with no-named trial barcodes (wet testing). Finally, validation of the system was performed with real patients, while the AV was running in the background but not active (i.e., while the manual veri ﬁ cation was still going on). After all these steps were successful, the system was started. Results: In the daytime, AV rates were ≥ 75 % for 23 of 29 tests. In night-shift, AV rates were ≥ 70 % for 16 of 25 tests. Report-based performance was found 26 % for daytime. TAT in the daytime decreased after AV implementation. Conclusions: Although this is the ﬁ rst time we have implemented the AV, a signi ﬁ cant percentage of the tests have been veri ﬁ ed. However, approaches that will increase the percentage of report-based veri ﬁ cation will enhance the e ﬃ ciency of autoveri ﬁ cation.


Introduction
The continuous rise in the number of individuals applying to hospitals, widespread use of laboratory tests in the diagnosis and follow-up, and the expansion of test panels have caused an increase in the workload of laboratories.The fact that the number of the clinical biochemists evaluating laboratory results is few making it difficult to evaluate each result with the same level of attention and to detect possible errors.Particularly, in large laboratories, there is a need for an approach to support the laboratory to this workload.
The clinical biochemist evaluates the test results in line with the intricate relationship of several variables, such as the patient's demographic data, clinical information, diagnosis, comorbidities, the relationship and compatibility of the test results with each other, and the previous results of the patient [1].Ultimately, a patient's report is verified in a truly short time frame according to the mental algorithms of the clinical biochemist.
Approximately 1,500 clinical biochemistry samples being delivered to Core Laboratory of Gazi University Health Research and Application Hospital (GUHRAH) in a day, which means only for the biochemistry panel approximately 20,000 tests were manually verified in daily.Verifying such a huge number of tests is vulnerable to error, especially if the clinical biochemist has additional tasks or the verification process is interrupted [2].Moreover, manual verification (MV) is a humdrum process and fatigue may develop.Thus, it may not always be possible to keep the concentration at the same level.Indeed, mostly test results are verified after a quick glance at the results [3].In addition, an issue to be mentioned is skills and experience of each clinical biochemist are not the same even in the same laboratory, thus may cause the lack of standardization and poor inter-observer agreement in releasing the test results [2,4].
Improvements in the post-analytical phase, such as autoverification (AV), may provide benefits on the aforementioned issues.AV standardizes the release of laboratory results, improves turnaround time (TAT) [3,[5][6][7], detects errors in total test process [2], permits more effective use of laboratory staff [8], and enhances the role of the clinical biochemist in the diagnostic processes by enabling focusing on the results that may be problematic [2].With the time gained by means of AV, the clinical biochemist will be more focused in other tasks in the laboratory, such as interpretation of results to physicians [9].
In recent years, developments in information technologies have responded to the needs of laboratory operations.There are several methods to harness AV.Laboratory information system (LIS) [7,10], expert systems [11,12], middlewares [3,10,13] and artificial intelligence [14] are may be used for AV process.A middleware is a software that makes it possible to verify the test results without human intervention, by using pre-defined algorithms designed by clinical biochemists [15].Middlewares act as a bridge between the LIS and autoanalyzer.Such softwares convert the rules in the algorithms into actions [15].These actions could be the verification of a result immediately, repeating the analysis, reflex testing, retest or auto-dilution (if there is a post-analytical storage system) or the addition of comments.
Autoverification algorithms can be established by using the quality control (QC) data, warning flags from instruments, serum indices, demographics and clinical information of the patients, critical values, result limit checks (i.e., release interval, AV range), reference ranges, delta checks, consistency checks, moving average alerts and the relationship between the tests [2,11,13,16].
In our study, we aimed to evaluate the outcomes of the AV in a tertiary hospital.A significant proportion of results were planned to verify via middleware according to rules and algorithms established by clinical biochemists.
We also aimed at improving quality and TAT, as well as to devote more time to the results that require interpretation.In addition, the benefits and experiences that AV provided to our laboratory are discussed.

Materials and methods
The study was conducted in GUHRAH, Core Biochemistry Laboratory between August 2017 and October 2019.The GUHRAH is a tertiary care academic medical centre which has approximately 1000-beds.For the study, Ethical Approval was obtained from the Gazi University Faculty of Medicine Clinical Research Ethics Committee (Date/Number: 2022/672).
The study workflow is presented in Supplementary Figure 1.First, an AV crew was established.The AV crew was comprised seven clinical biochemistry research assistants, two clinical biochemists, two workers from the manufacturer (Beckman Coulter) for middleware technical support (MTS), and two workers from LIS company.Meetings were held bi-monthly with the AV crew.In smaller groups, extra meetings were done as needed.

Reference change value (RCV) calculation and delta check implementation
The Clinical and Laboratory Standards Institute (CLSI) EP33 document was used for RCV calculation and delta check implementation [17].RCVs of clinical biochemistry tests were calculated (Table 1) [18,19].Bidirectional Z-score was used for calculations (1.96 for 95 % probability).RCV formula that used was below [20,21]: The analytical variation (CV A ) value in the formula was calculated using the results of internal quality controls.For the within-subject biological variation (CV I ) value in the formula, the CV I (sample type: serum) values available on Westgard's website were used [22].For sodium, RCV was accepted as an absolute value instead of a percentage change (Table 1).
Initially, considering patient safety, the time interval for delta check application was estimated as 8 h.Then, with safety checks and checking the data from the literature, the time interval of delta check was increased to one day, one week, and one month, respectively [5,16].After the performance of the AV system was evaluated, the time interval for delta check query was increased to three months.

Establishing algorithms and validation of the AV system
AV algorithms prepared by clinical biochemists based on the Boolean logic consisted of QC, critical values, serum indices, autoanalyzer flags, consistency checks, measurement intervals, related tests, delta checks, and AV range steps (Figure 1).During implementing the AV; CLSI AUTO10A, CLSI AUTO15, the College of American Pathologists (CAP) general checklist [23][24][25], and the document prepared by the Department of Examination and Diagnostic Services, Turkish Ministry of Health on the minimum requirements for laboratories that will apply AV were considered [26].

Yilmaz et al.: Process improvement via autoverification
At the beginning, the algorithms were implemented to the middleware (Remisol Advance 1.10, Beckman Coulter, Brea, CA, USA) for 22 clinical biochemistry tests which ran on AU5800 autoanalyzer (Beckman Coulter, Brea, CA, USA).During this process, MTS of the manufacturer were in close cooperation with LIS (Nucleus, Monad, Ankara, Turkey).
Initially, validation of the AV system was performed over datasets and simulated patients (dry testing).All steps in algorithms were checked with simulated patients.Simulated results were prepared for 29 tests.For each of the tests, the algorithms were checked with a total of two simulated results that will test each step in the algorithm in a positive and negative sense.Meanwhile, the patient samples that may violate AV rules were aliquoted and stored.Then, to test the AV system, these samples were tested anonymously with no-named trial barcodes which created from information technologist of LIS (wet testing).After that, validation of the AV system with real patients was performed in the background of LIS, while still the MV was going on and without interfering the MV.After all these steps were successful, the system was started.
Performance queries for the AV system, and adding new goals Performance of the system evaluated monthly.Test-based and reportbased (tube-based) AV rates were evaluated.Moreover, the day-shift and night-shift performance of the system was examined.In addition, AV performance for inpatients and outpatients was analyzed.
At the beginning, due to only 22 biochemistry tests and one autoanalyzer implemented to AV system, test and report-based queries were performed as if only 22 biochemistry tests and just one instrument (AU5800, Beckman Coulter, Brea, CA, USA) were running in the laboratory.Tests to be added later to the AV system such as iron, high density lipoprotein (HDL), triglyceride, total cholesterol, and the serum indices were not included in the first query.
New goals were set to improve the performance of the system.AV test panel expanded by adding seven more tests (HDL, triglyceride, total cholesterol, iron, and serum indices) to the AV algorithms.In addition, another biochemistry autoanalyzer (AU680, Beckman Coulter, Brea, CA, USA) running on the night-shifts and weekends implemented to the system (also, emergency department patients were running on AU680).
For the daytime; report-based performance of the system evaluated for three time points (Supplementary Figure 2).Since the implementation of AV, six days at three different time points (from three different months) were selected and the report-based AV rates were calculated by mean of these six days.The daytime and night-time intervals were described below: Daytime (day-shift): From 09:00 a.m. to 17:00 p.m. Night-time (night-shift): From 17:00 p.m. to 09:00 a.m. next day.In addition, the weekends and holidays are considered as night-time.

Statistical analysis
Statistical analyzes were performed using the SPSS 22.0 for Windows package program (IBM Inc., Chicago, IL, USA).For graphs, Prism ® 9.0 (GraphPad Software Inc., La Jolla, CA) was used.Significance was assumed for all values where p<0.05.All the reported p values are two sided.Normality assessment performed by Kolmogorov-Smirnov and Shapiro-Wilk tests.For evaluating AV system's effect on TAT, two equivalent three-month intervals before and after AV analysed with Mann-Whitney U test.

Results
RCVs of clinical biochemistry tests were calculated and also were implemented to the system (Table 1).Test-based and report-based performance of the AV system is presented (Figures 2 and 3).In the day-shift, the overall test-based AV rate (AV passing rate) was 81 %.The report-based AV rate for the day-shift was 28 , 30, and 19 % respectively, at three different time points (Supplementary Figure 2).In the daytime, the average report-based performance was found 26 %.
TAT in the daytime decreased after AV implementation, whereas considering night-shift TAT and overall TAT, there were no difference between before and after-AV (Table 2).Clinical biochemists gained 27 workdays in a year, thanks to autoverification (Supplementary Data 1).After implementing the second autoanalyzer (AU680) to the system, for inpatients report-based AV rate increased 16 % from 10 % (Supplementary Figure 3).In addition, a significant increase was seen on test-based AV rates of inpatients after implementing AU680 (Supplementary Table 1).Similarly, with the implementation of the second autoanalyzer, the test-based performance of the system increased during the day-shift (Table 3).

Discussion
With this study, we showed the significant progress we have made in the post-analytical phase through AV.Moreover, improvement on the daytime TAT was accomplished.Also, auto-comments via algorithms contributed to the standardization of the patient reports.
Particularly, in day-shift AV rates was satisfactory, overall AV passing rate was 81 %.In the day-shift, AV rates were ≥75 % for 23 of 29 tests.Considering there is a great test load during the day-shift, these results are substantial for our laboratory.AV rates of glucose, amylase, iron, HDL, total cholesterol and triglyceride were <75 % for both day-shift and outpatients subgroups.In night-shift conditions, AV rates were ≥70 % for 16 of 25 tests.AV rates of glucose, amylase, aspartate aminotransferase (AST), lactate dehydrogenase (LDH), creatinine, total protein, albumin, phosphorus, and magnesium were <70 %.AV rate of hemolysis was found to be lower in the night-shift, which may explain the lower AV rates in AST, LDH, and phosphorus.AV rates of all tests except direct bilirubin were higher in the day-shift than in the night-shift.AV rates of all tests except triglyceride were higher in outpatients than in inpatients.
Our tube-based (report-based) AV was not at the desired level.Furthermore, report-based AV rates decreased after adding new tests to the system.In case of report-based verification cannot be accomplished (i.e., some of the biochemistry tests in the report are verified), it may be necessary to examine the patient's other test results while performing MV.In this respect, report-based AV rate is important in the evaluation of AV performance.There are several reasons that report-based AV rate is not as high as we aimed.Mainly, this may be because we prefer to construct the AV system with extreme safety, and that we implemented the AV for the first time.Also, for the tests which were calculated a higher RCV compared to the values in the previous studies, we used the RCVs in the literature to make the AV system safer [18,27].As often practiced by laboratories first starting AV, we used the reference range as AV range [2].Owing to using reference interval as the AV range (i.e., releasing interval, limit check, verification decision limit), our AV rates are lower than in previous studies [10,28,29].Our lower report-based rate also may be due to our laboratory is in a tertiary hospital.This means due to the high number of hospitalized patients; we encounter more   pathological results.However, our report-based AV rate was in line with a recent study performed in our hospital, in which three different models of verification decision limit were established, then report-based AV rates calculated for each model [11].In the aforementioned publication, the report-based AV was 28 % (similar to our study) in the model with delta check applied and reference range as release interval [11].We think that, with experience, and after optimization of AV processes, we may reach higher report-based AV rates.AV algorithms for clinical biochemistry tests are more complex than hematology or some of other test panels in laboratory [30].Since we had no experience with AV before, we designed the process in the simplest and most basic way possible.But we think our AV rates are fairly well for a laboratory implementing AV for the first time on clinical biochemistry tests [3,5,10,31].AV is a dynamic process and broadening the releasing interval with the following revisions will probably increase our AV performance.It is aimed at taking actions that will increase the approval of all biochemistry tests on a report basis.
Indeed, we expected to see a rise in test-based AV rates after the second autoanalyzer's (AU680) implementation to the AV system.In our laboratory, the samples of inpatients were running on AU5800 during day-shifts, and on AU680 at night-shifts.Before AU680's integration to AV system, the results of inpatients whose samples were run at night-shift could not be recorded in the middleware's memory.This had caused problems in AV of inpatients' results, which were under follow-up.Also, after implementing the AU680 as the second autoanalyzer to the AV system, the report-based AV rate of inpatients increased from 10 to 16 %.
There are several studies showing improvement in TAT after AV [5,7].In line with the literature, we succeeded in a significant improvement in TAT during the daytime, which was a remarkable aspect for us.We needed AV to create such an improvement, as more tubes came into the laboratory during the daytime.But in the night-shift we did not observe such an effect.This may be due to a clinical biochemist is always on duty for MV during night-shifts and the number of tubes arriving at night-shifts is not high.In addition, autorepeat or dilution commands were added to algorithms for reducing TAT in need of retest or dilution.In such samples, TAT has improved thanks to AV and post-analytical storage system in our laboratory.
AV has also contributed to the lean process.The status of the AV system can be monitored online with a dashboard in the laboratory.When a test's AV percentage appears low in the dashboard, or AV fails on the same test repetitively, we detected easily an analytical shift existing in that test.
It has been shown that AV reduces the workload of laboratory staff [3,6,28].We calculated that the clinical biochemists responsible for verification saved 27 working days in a year, which corresponds to 14 % of the annual workload of biochemists.Also, we increased the number of lectures and trainings in the laboratory, in line with the decreasing workload.Not only did we more carefully consider the results held for manual review, we were also able to devote more time to other essential tasks that would add value to the laboratory.
Although there are common points between laboratories applying AV, each system should be unique.Autoanalyzer flags, measurement platforms, and measurement intervals of the analyzers used may be different.Patient characteristics (such as outpatient/inpatient, emergency, hemodialysis, oncology, transplant patients) vary by hospital.For this reason, it is recommended to design algorithms considering the instrument and institution features.In brief, each AV system should be centre-specific, updated, and dynamic.
Both international and local standards and guidelines should be considered for implementing AV.Conditions about AV may vary in each country.In our country, AV is optional, based on volunteerism, and merely it can be started with the consent of all clinical biochemists working in hospital.All clinical biochemists in the laboratory may not agree to start AV, thus can be counted as a discouraging factor.In our country, the regulations and the guidance of the Turkish Ministry of Health have to be followed while performing the AV.Before running the AV, strict validation of the AV system is mandatory for the patient safety, according to local regulations.In addition, after the notification of the start of AV to the Turkish Ministry of Health, laboratories are subject to audit.Indeed, the AV system implementation requires multidisciplinary work, and demands time and dedication.The installation process of the system, particularly the integration of the LIS and middleware took time.The change of the LIS firm was one of the major drawbacks for us, which caused a lag for a few months.Because purchasing processes are usually carried out every two or three years in state and university laboratories in Turkey, LIS or autoanalyzers in the laboratory may change.And usually change of LIS and autoanalyzers occur in different years.Thus, the TAT for implementation of the AV system should be faster, which is an important challenge.We had an advantage that the LIS had changed, but the autoanalyzers remained the same.
One strength of our work is that we evaluate the performance of the AV not only test but also on a report-basis.In addition, presenting the system's performance with outcomes, such as the decrease in TAT and workload of the biochemists, also reveals the importance of our study.Unlike numerous studies, we demonstrated AV performance according to shifts and inpatient/outpatient status.However, there are some limitations of our study.AV range was set as the reference interval for each test and this caused low AV rates for several tests.Our study and the findings in a recent publication show that choosing the reference interval as the AV range may be insufficient for some tests, such as the glucose, iron, and lipid panel [32].Since we had obtained testbased AV rates data only for six days (from different months) we could not perform a statistical analysis of AV rates according to hospitalization status (outpatient/inpatient).In addition, when calculating RCV, not using the different RCV formulas for some tests [33][34][35], and calculating RCVs with a uniform approach may be considered as another limitation of our study.
In conclusion, standardization of verification, early detection of laboratory errors, improvement of patient safety, shortening of TAT and focusing clinical biochemist on remaining important results were achieved by means of AV.Briefly, in the total test process, particularly in the postanalytical phase, a prominent improvement succeeded via AV.
Applying patient-based QC through the middleware, and thus helping to broaden the releasing range were identified as future goals.Besides, the addition of more complex related test rules was determined as a new target.Establishing different RCVs and new rules according to the departments in hospital, updating the RCVs values with different RCV formulas are also among our new goals.

Figure 3 :
Figure 3: Report-based performance of the algorithms.AV rates presented as % (percentage).

Figure 2 :
Figure 2: Test-based performance of the algorithms.AV rates presented as % (percentage).

Table  :
RCVs calculated by our laboratory, available in the literature, and implemented ones.

Table  :
Test-based AV rates on day-shift, before and after implementation of second autoanalyzer.