Abstract
Objectives
Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated.
Methods
Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed.
Results
The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory.
Conclusions
We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.
Funding source: Natural Science Foundation of Shandong Province
Award Identifier / Grant number: ZR2017MH044
Funding source: Key Technology Research and Development Program of Shandong, China
Award Identifier / Grant number: 2019GSF108247
Funding source: National Natural Science Foundation of China
Award Identifier / Grant number: 81702815, 81972005
Funding source: Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan
Award Identifier / Grant number: 201805061
Acknowledgments
We would like to acknowledge Roche Diagnostics (Shanghai) Limited, Digital Solution team from CI and Solution Integration team from CPS & MD, for their digital expertise and technical supports.
-
Research funding: National Natural Science Foundation of China (No. 81972005, 81702815), Natural Science Foundation of Shandong Province (No. ZR2017MH044), Key Technology Research and Development Program of Shandong, China (No. 2019GSF108247), Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan (No. 201805061).
-
Author contributions: All authors have accepted responsibility for the entire content of this submitted manuscript and approved its submission.
-
Conflicts of Interest: Authors state no conflict of interest.
-
Ethical approval: This study has been approved by the Human Research Ethics Committee of Qilu Hospital of Shandong University (KYLL-2019-2-045).
References
1. Randell, EW, Yenice, S, Khine Wamono, AA, Orth, M. Autoverification of test results in the core clinical laboratory. Clin Biochem 2019;73:11–25. https://doi.org/10.1016/j.clinbiochem.2019.08.002.Search in Google Scholar PubMed
2. Autoverification of Clinical Laboratory Test Results, 1st ed. CLSI guideline AUTO10-A; 2006. [Online]. Available from: https://clsi.org/standards/products/automation-and-informatics/documents/auto10/.Search in Google Scholar
3. Autoverification of Medical Laboratory Results for Specific Disciplines, 1st ed. CLSI guideline AUTO15; 2019. [Online]. Available from: https://clsi.org/standards/products/automation-and-informatics/documents/auto15/.Search in Google Scholar
4. Valdiguie, PM, Rogari, E, Philippe, H. VALAB: expert system for validation of biochemical data. Clin Chem 1992;38:83–7. https://doi.org/10.1093/clinchem/38.1.83.Search in Google Scholar
5. Cabitza, F, Banfi, G. Machine learning in laboratory medicine: waiting for the flood? Clin Chem Lab Med 2018;56:516–24. https://doi.org/10.1515/cclm-2017-0287.Search in Google Scholar PubMed
6. Obermeyer, Z, Emanuel, EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med 2016;375:1216–9. https://doi.org/10.1056/nejmp1606181.Search in Google Scholar PubMed PubMed Central
7. Wilkes, EH, Rumsby, G, Woodward, GM. Using machine learning to aid the interpretation of urine steroid profiles. Clin Chem 2018;64:1586–95. https://doi.org/10.1373/clinchem.2018.292201.Search in Google Scholar PubMed
8. Luo, Y, Szolovits, P, Dighe, AS, Baron, JM. Using machine learning to predict laboratory test results. Am J Clin Pathol 2016;145:778–88. https://doi.org/10.1093/ajcp/aqw064.Search in Google Scholar PubMed
9. Somnay, YR, Craven, M, McCoy, KL, Carty, SE, Wang, TS, Greenberg, CC, et al.. Improving diagnostic recognition of primary hyperparathyroidism with machine learning. Surgery 2017;161:1113–21. https://doi.org/10.1016/j.surg.2016.09.044.Search in Google Scholar PubMed PubMed Central
10. Krasowski, MD, Davis, SR, Drees, D, Morris, C, Kulhavy, J, Crone, C, et al.. Autoverification in a core clinical chemistry laboratory at an academic medical center. J Pathol Inf 2014;5:13. https://doi.org/10.4103/2153-3539.129450.Search in Google Scholar PubMed PubMed Central
11. Demirci, F, Akan, P, Kume, T, Sisman, AR, Erbayraktar, Z, Sevinc, S. Artificial neural network approach in laboratory test reporting: learning algorithms. Am J Clin Pathol 2016;146:227–37. https://doi.org/10.1093/ajcp/aqw104.Search in Google Scholar PubMed
12. Zhu, J, Wang, B, Guo, W, Pan, B. Establishment and optimization of an autoverification system for clinical biochemistry test results [in Chinese]. Chin J Clin Lab Sci 2018;36:704–8.Search in Google Scholar
13. Dabbura, I. K-means clustering: algorithm, applications, evaluation methods, and drawbacks; 2018. Available from: https://imaddabbura.github.io/post/kmeans-clustering/.Search in Google Scholar
14. Autoverification of clinical laboratory quantitative test results [in Chinese], National Health Commission of PRC standard WS/T 616-2018; 2018. [Online]. Available from: http://www.nhc.gov.cn/ewebeditor/uploadfile/2018/09/20180925121506686.pdf.Search in Google Scholar
15. Chawla, N, Bowyer, K, Hall, L, Kegelmeyer, W. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321–57. https://doi.org/10.1613/jair.953.Search in Google Scholar
16. He, H, Bai, Y, Garcia, E, Li, S. ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). China: Hong Kong; 2008.Search in Google Scholar
17. Barber, D. Bayesian reasoning and machine learning. New York, NY: Cambridge University Press; 2012:697 p.10.1017/CBO9780511804779Search in Google Scholar
18. Guo, G, Wang, H, Bell, D, Bi, Y, Greer, K. KNN model-based approach in classification. In: Meersman, R, Tari, Z, Schmidt, DC, editors. On the move to meaningful internet systems. German: Springer; 2003.10.1007/978-3-540-39964-3_62Search in Google Scholar
19. Deng, H. An introduction to random forest; 2018. Available from: https://towardsdatascience.com/random-forest-3a55c3aca46d.Search in Google Scholar
20. Chen, T. Introduction to boosted trees; 2014. Available from: https://homes.cs.washington.edu/∼tqchen/pdf/BoostedTree.pdf.Search in Google Scholar
21. Hastie, T, Tibshirani, R, Friedman, JH. The elements of statistical learning: data mining, inference, and prediction, 2nd ed. New York, NY: Springer; 2009:745 p.10.1007/978-0-387-84858-7Search in Google Scholar
22. Murdoch, WJ, Singh, C, Kumbier, K, Abbasi-Asl, R, Yu, B. Definitions, methods, and applications in interpretable machine learning. Proc Natl Acad Sci U S A 2019;116:22071–80. https://doi.org/10.1073/pnas.1900654116.Search in Google Scholar PubMed PubMed Central
23. Richardson, A, Signor, BM, Lidbury, BA, Badrick, T. Clinical chemistry in higher dimensions: machine-learning and enhanced prediction from routine clinical chemistry data. Clin Biochem 2016;49:1213–20. https://doi.org/10.1016/j.clinbiochem.2016.07.013.Search in Google Scholar PubMed
24. Allen, RH. The evaluation and choice of laboratory equipment and reagents. Can Med Assoc J 1965;93:760–1.Search in Google Scholar
25. Doshi-Velez, F, Kim, B. Towards A rigorous science of interpretable machine learning. arXiv; 2017. Available from: https://arxiv.org/abs/1702.08608v2.Search in Google Scholar
26. Torke, N, Boral, L, Nguyen, T, Perri, A, Chakrin, A. Process improvement and operational efficiency through test result autoverification. Clin Chem 2005;51:2406–8. https://doi.org/10.1373/clinchem.2005.054395.Search in Google Scholar PubMed
27. Chauhan, KP, Trivedi, AP, Patel, D, Gami, B, Haridas, N. Monitoring and root cause analysis of clinical biochemistry turn around time at an academic hospital. Indian J Clin Biochem 2014;29:505–9. https://doi.org/10.1007/s12291-013-0397-x.Search in Google Scholar PubMed PubMed Central
28. Wang, Z, Peng, C, Kang, H, Fan, X, Mu, R, Zhou, L, et al.. Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory. BMC Med Inf Decis Making 2019;19:123. https://doi.org/10.1186/s12911-019-0848-2.Search in Google Scholar PubMed PubMed Central
29. Kumar, Y, Sahoo, G. Prediction of different types of liver diseases using rule based classification model. Technol Health Care 2013;21:417–32. https://doi.org/10.3233/thc-130742.Search in Google Scholar PubMed
30. Ruthotto, L, Osher, SJ, Li, W, Nurbekyan, L, Fung, SW. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proc Natl Acad Sci U S A 2020;117:9183–93. https://doi.org/10.1073/pnas.1922204117.Search in Google Scholar PubMed PubMed Central
Supplementary Material
The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2020-0716).
© 2020 Walter de Gruyter GmbH, Berlin/Boston
