Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter November 26, 2020

Using machine learning to develop an autoverification system in a clinical biochemistry laboratory

  • Hongchun Wang , Huayang Wang , Jian Zhang , Xiaoli Li , Chengxi Sun and Yi Zhang EMAIL logo



Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated.


Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed.


The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory.


We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.

Corresponding author: Yi Zhang, Professor, Department of Clinical Laboratory, Qilu Hospital of Shandong University, #107 Wenhuaxi Road, Jinan, 250012, Shandong, P.R. China, Phone: +86-531-82166801, Fax: +86-531-86927544, E-mail:

Hongchun Wang and Huayang Wang contributed equally to this article.

Funding source: Natural Science Foundation of Shandong Province

Award Identifier / Grant number: ZR2017MH044

Funding source: Key Technology Research and Development Program of Shandong, China

Award Identifier / Grant number: 2019GSF108247

Funding source: National Natural Science Foundation of China

Award Identifier / Grant number: 81702815, 81972005

Funding source: Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan

Award Identifier / Grant number: 201805061


We would like to acknowledge Roche Diagnostics (Shanghai) Limited, Digital Solution team from CI and Solution Integration team from CPS & MD, for their digital expertise and technical supports.

  1. Research funding: National Natural Science Foundation of China (No. 81972005, 81702815), Natural Science Foundation of Shandong Province (No. ZR2017MH044), Key Technology Research and Development Program of Shandong, China (No. 2019GSF108247), Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan (No. 201805061).

  2. Author contributions: All authors have accepted responsibility for the entire content of this submitted manuscript and approved its submission.

  3. Conflicts of Interest: Authors state no conflict of interest.

  4. Ethical approval: This study has been approved by the Human Research Ethics Committee of Qilu Hospital of Shandong University (KYLL-2019-2-045).


1. Randell, EW, Yenice, S, Khine Wamono, AA, Orth, M. Autoverification of test results in the core clinical laboratory. Clin Biochem 2019;73:11–25. in Google Scholar PubMed

2. Autoverification of Clinical Laboratory Test Results, 1st ed. CLSI guideline AUTO10-A; 2006. [Online]. Available from: in Google Scholar

3. Autoverification of Medical Laboratory Results for Specific Disciplines, 1st ed. CLSI guideline AUTO15; 2019. [Online]. Available from: in Google Scholar

4. Valdiguie, PM, Rogari, E, Philippe, H. VALAB: expert system for validation of biochemical data. Clin Chem 1992;38:83–7. in Google Scholar

5. Cabitza, F, Banfi, G. Machine learning in laboratory medicine: waiting for the flood? Clin Chem Lab Med 2018;56:516–24. in Google Scholar PubMed

6. Obermeyer, Z, Emanuel, EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med 2016;375:1216–9. in Google Scholar PubMed PubMed Central

7. Wilkes, EH, Rumsby, G, Woodward, GM. Using machine learning to aid the interpretation of urine steroid profiles. Clin Chem 2018;64:1586–95. in Google Scholar PubMed

8. Luo, Y, Szolovits, P, Dighe, AS, Baron, JM. Using machine learning to predict laboratory test results. Am J Clin Pathol 2016;145:778–88. in Google Scholar PubMed

9. Somnay, YR, Craven, M, McCoy, KL, Carty, SE, Wang, TS, Greenberg, CC, et al.. Improving diagnostic recognition of primary hyperparathyroidism with machine learning. Surgery 2017;161:1113–21. in Google Scholar PubMed PubMed Central

10. Krasowski, MD, Davis, SR, Drees, D, Morris, C, Kulhavy, J, Crone, C, et al.. Autoverification in a core clinical chemistry laboratory at an academic medical center. J Pathol Inf 2014;5:13. in Google Scholar PubMed PubMed Central

11. Demirci, F, Akan, P, Kume, T, Sisman, AR, Erbayraktar, Z, Sevinc, S. Artificial neural network approach in laboratory test reporting: learning algorithms. Am J Clin Pathol 2016;146:227–37. in Google Scholar PubMed

12. Zhu, J, Wang, B, Guo, W, Pan, B. Establishment and optimization of an autoverification system for clinical biochemistry test results [in Chinese]. Chin J Clin Lab Sci 2018;36:704–8.Search in Google Scholar

13. Dabbura, I. K-means clustering: algorithm, applications, evaluation methods, and drawbacks; 2018. Available from: in Google Scholar

14. Autoverification of clinical laboratory quantitative test results [in Chinese], National Health Commission of PRC standard WS/T 616-2018; 2018. [Online]. Available from: in Google Scholar

15. Chawla, N, Bowyer, K, Hall, L, Kegelmeyer, W. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321–57. in Google Scholar

16. He, H, Bai, Y, Garcia, E, Li, S. ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). China: Hong Kong; 2008.Search in Google Scholar

17. Barber, D. Bayesian reasoning and machine learning. New York, NY: Cambridge University Press; 2012:697 p.10.1017/CBO9780511804779Search in Google Scholar

18. Guo, G, Wang, H, Bell, D, Bi, Y, Greer, K. KNN model-based approach in classification. In: Meersman, R, Tari, Z, Schmidt, DC, editors. On the move to meaningful internet systems. German: Springer; 2003.10.1007/978-3-540-39964-3_62Search in Google Scholar

19. Deng, H. An introduction to random forest; 2018. Available from: in Google Scholar

20. Chen, T. Introduction to boosted trees; 2014. Available from:∼tqchen/pdf/BoostedTree.pdf.Search in Google Scholar

21. Hastie, T, Tibshirani, R, Friedman, JH. The elements of statistical learning: data mining, inference, and prediction, 2nd ed. New York, NY: Springer; 2009:745 p.10.1007/978-0-387-84858-7Search in Google Scholar

22. Murdoch, WJ, Singh, C, Kumbier, K, Abbasi-Asl, R, Yu, B. Definitions, methods, and applications in interpretable machine learning. Proc Natl Acad Sci U S A 2019;116:22071–80. in Google Scholar PubMed PubMed Central

23. Richardson, A, Signor, BM, Lidbury, BA, Badrick, T. Clinical chemistry in higher dimensions: machine-learning and enhanced prediction from routine clinical chemistry data. Clin Biochem 2016;49:1213–20. in Google Scholar PubMed

24. Allen, RH. The evaluation and choice of laboratory equipment and reagents. Can Med Assoc J 1965;93:760–1.Search in Google Scholar

25. Doshi-Velez, F, Kim, B. Towards A rigorous science of interpretable machine learning. arXiv; 2017. Available from: in Google Scholar

26. Torke, N, Boral, L, Nguyen, T, Perri, A, Chakrin, A. Process improvement and operational efficiency through test result autoverification. Clin Chem 2005;51:2406–8. in Google Scholar PubMed

27. Chauhan, KP, Trivedi, AP, Patel, D, Gami, B, Haridas, N. Monitoring and root cause analysis of clinical biochemistry turn around time at an academic hospital. Indian J Clin Biochem 2014;29:505–9. in Google Scholar PubMed PubMed Central

28. Wang, Z, Peng, C, Kang, H, Fan, X, Mu, R, Zhou, L, et al.. Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory. BMC Med Inf Decis Making 2019;19:123. in Google Scholar PubMed PubMed Central

29. Kumar, Y, Sahoo, G. Prediction of different types of liver diseases using rule based classification model. Technol Health Care 2013;21:417–32. in Google Scholar PubMed

30. Ruthotto, L, Osher, SJ, Li, W, Nurbekyan, L, Fung, SW. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proc Natl Acad Sci U S A 2020;117:9183–93. in Google Scholar PubMed PubMed Central

Supplementary Material

The online version of this article offers supplementary material (

Received: 2020-05-14
Accepted: 2020-11-12
Published Online: 2020-11-26
Published in Print: 2021-04-27

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 7.12.2023 from
Scroll to top button