Application of machine learning approach in emergency department to support clinical decision making for SARS-CoV-2 infected patients

2.1 Study design and sample

This study retrospectively analyzed a cohort of SARS-CoV-2 infected patients presenting to ED, to collect a dataset for devising the ML models. Biochemical and clinical data of 779 adults (age > 18 years), affected by SARS-CoV-2, admitted during the first two waves to three hospitals of Lazio-Abruzzo area, were collected and analyzed. Missing data were managed by automatic imputation as described in the section below.

Demographic, clinical, and biochemical variables were collected by a physician, during the first presentation at each ED.

The clinical outcomes were safe discharge, disease severity at seventh day of hospitalization and mortality during hospitalization. Data were collected at three different times: at ED presentation (T0), at the seventh day of hospitalization (T7) and at hospital discharge (TD). The disease severity was interpreted by following the interim guidance released by the WHO [20]. WHO ordered the severity of the infectious disease in “mild” (WHO score 0), “moderate” for patients who no required supplemental oxygen (WHO score 1), and “severe” for all patients who needed non-invasive ventilation (NIV) or high-flow oxygen requirement or continuous positive airway pressure (CPAP) (WHO score 2), and “critical” for patients who needed invasive ventilation or multi-organ support (WHO score ≥3). The safe discharge was a composite endpoint considering for all enrolled patients: PaO₂/FiO₂ ratio > 274, PaO₂ > 65 mmHg, no oxygen supply during hospitalization, length of hospitalization < 10 days and no adverse events during hospitalization (endotracheal intubation, disease progression to severe or critical disease, admission to high dependency unit, intensive care unit, mortality). The cut-off of PaO₂/FiO₂ ratio > 274 was chosen considering the results reported in our previous manuscript [21]. The safe discharge outcome reported in the dataset was then revised, so to represent the correct decision that should have been taken at admission time. The mortality risk and the severity disease risk were retrospectively evaluated, applying respectively the 4C Mortality Score for COVID-19 (4C MS) and the quick COVID-19 Severity Index (qCSI) [22, 23].

The study was approved by the Internal Review Board of our Institution (IRB 06/2021, protocol number 28,958) and registered on clinicaltrial.gov, protocol number NCT04825301. The study was conformed to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement and should be considered a prediction model study type 2a [24].

2.2 Missing data imputation

To devise the models in case of missing data, the dataset was pre-processed through missing data imputation system [25, 26]. In detail, predictive mean matching for numeric data and logistic regression imputation for categorical data (with 2 levels) was used [27].

2.3 Predictors

To devise the ML models, both continuous and categorical variables were used. Heart rate (HR), respiratory rate-oxygenation (ROX) index, alanine aminotransferase (ALT), aspartate aminotransferase (AST), creatinine, C-reactive protein (CRP), lactic dehydrogenase (LDH), platelet count (PLT), white blood cells count (WBC), neutrophil count (NEU), lymphocyte count (LIN), eosinophil count (EOS), basophil count (BAS), neutrophil lymphocyte ratio (NLR), platelet lymphocyte ratio (PLR), international normalized ratio (INR), D-dimer (DD), standard blood gas analysis variables and PaO₂/FiO₂ ratio have been included as continuous variables. Sex at birth, fever (or not) and cough (or not) have been considered as categorical variables.

2.4 Outcomes

Primary outcome was the prediction (i.e., safe/risk) of safe discharge; secondary outcomes were the prediction of disease severity at seventh day (i.e., severe/mild) and mortality during hospitalization (i.e., present/absent).

2.5 Statistical analysis and ML models

The descriptive analyses consist of frequency tables, medians and IQRs. Data reported in Supplementary Table 1 have been analyzed using one way ANOVA with post-hoc Fisher LSD tests for multiple comparison.

Based on a different selection of predictors, several models with complete set of predictors and with set of predictors were devised, details are reported in Supplementary Table 3.

The ML task for all outcomes is binary classification. The following different models were tried from the available data: decision tree (DT), random forest (RF) and gradient boosting machines (GBM) [28–30]. For all models and outcomes, the dataset was split into train and test (with 75% of data going for training, 25% for testing), used 10-fold cross-validation, tuned each classifier according to its specific hyperparameters, calculated the confusion matrix (with the most useful associated statistics, i.e., accuracy, Cohen’s K, sensitivity, specificity) and the ROC curve (with the related AUC) [31]. To consider data imbalance, we also adopted RUSBoost as hybrid sampling/boosting algorithm [32].

In addition, to improve the explainability of the models, the variables importance was calculated and presented as graphs [33].

To compare the ML models (for the disease severity and mortality risk) with the standard clinical practice, the qCSI and 4C MS indexes was dichotomized. The confusion matrices (with the same associated statistics of the ML models) were calculated and finally, the results were compared with the predictions of the corresponding ML models, i.e., severity and mortality, respectively.

All statistical analyses and ML models discussed in the paper were calculated with RStudio 2022.07.2–576 running R4.2.1 for Linux, using the “caret” and “mice” packages [27, 34]. Due to the restrictions decided by the Internal Review Board, we are unable to publicly share the study data.

3.1 Sample description

The final dataset included 779 patients; demographic and clinical characteristics and principal biochemical markers for the study population are reported in Supplementary Table 1. Data record was not complete for 283 patients, whereas 496 patients have got a complete data record. The study population presented a median age of 65 years (IQR 55–75), 41.3% were female, more than three comorbidities were present in 19% of the cases. The median value of CRP was 4.84 mg/dL (IQR 1.58–10.89), of LDH was 302 IU/L (IQR 227–389.5), of platelet was 202 10³/mcl (IQR 160–265), of WBC was 6.27 10⁹/L (IQR 4.60–9.24), of neutrophil count was normal 4.55 10⁹/L/L (IQR 3.11–7.60), of lymphocytes count was 0.92 10⁹/L/L (IQR 0.64–1.33). D-Dimer was tested in D-Dimer Unit (DDU), its median value was 0.81 mcg/mL DDU (IQR 0.47–1.50). Respiratory parameter as PaO₂ had a median value of 70.2 mmHg (IQR 59–83), the PaO₂/FiO₂ ratio 308 (IQR 223–365) and the ROX index 20.4 (IQR 13–20.9). Patients with severe disease or deceased were elder and characterized by higher values of CRP, LDH, D-Dimer, and progressive lower values for the PaO₂/FiO₂ ratio and ROX index, compared to the safe discharge population.

At presentation in ED, 72.6% of patients reported fever and 54.1% referred cough; the clinical spectrum of disease of the whole cohort is outlined in Supplementary Table 2. At presentation in ED, SARS-CoV-2 patients presented mild (26.3%), moderate (38.7%), severe (28.6%), critical (6.4%) symptoms. In Supplementary Table 2 are reported the distribution of safe discharge, severe disease and died population considering the WHO interim guidance (see material and methods).

3.2 Machine learning models

As previously described, based on a different selection of predictors we devised several models, the results of all the prediction models developed are reported in Supplementary in Table 4.

Tables 1a and 1b report the main statistics as coming from the confusion matrices for the best classifiers selected: the ROX index models and the PaO₂/FiO₂ ratio models. Figure 1 instead contains all ROC curves with the related AUCs, for the best classifiers (in terms of AUC), for the different outcomes. Finally, Figure 2 reports the importance of the first ten variables, ranked by importance, for the different outcomes, for the best classifiers (in terms of AUC).

Figure 1:

ROC curves of the best models (in terms of AUC) for (A) safe discharge: (a) PaO₂/FiO₂ ratio RUSBoost model – RF, (b) ROX index RUSBoost model – RF; (B) disease severity: (c) PaO₂/FiO₂ ratio model – RF, (d) ROX index RUSBoost model – RF; (C) mortality: (e) PaO₂/FiO₂ ratio RUSBoost model – GBM, (f) ROX index RUSBoost model – RF.

Figure 2:

Variables ranked by importance for the different outcomes for (A) safe discharge: (a) PaO₂/FiO₂ ratio model – RF, (b) ROX index model – GBM; (B) disease severity: (c) PaO₂/FiO₂ ratio model – RF, (d) ROX index model – RF; (C) mortality: (e) PaO₂/FiO₂ ratio model – GBM, (f) ROX index model – RF.

PaO₂/FiO₂ ratio models. For safe discharge, RF is the most accurate in almost all the statistics, except for sensitivity, where GBM performs slightly better. The results for disease severity showed that RF performed with the best AUC; conversely, GBM has slightly superior sensitivity and accuracy. Regarding mortality during hospitalization, GBM reached the best AUC and specificity, on the other hand, RF performed the best accuracy and sensitivity (Table 1a and Figure 1).

ROX index models. For safe discharge, GBM has the best AUC. On the other hand, RF demonstrated the highest sensitivity and accuracy. The results for disease severity showed that RF is the most accurate model, with the highest specificity and the best AUC. Conversely, GBM has the best accuracy. In connection with mortality during hospitalization, RF is the best classifier in terms of all statistics, except for specificity, where GBM performs slightly better (Table 1b and Figure 1).

3.3 Comparison of ML models with standard clinical practice

Table 2 contains the main statistics of the best classifiers and the indexes commonly used in clinical practice, for both disease severity and mortality. Figure 3 depicts the comparison in terms of ROC curves and AUCs. As mentioned earlier, given that the indexes could be computed only for the complete cases, the comparison is reported with the models devised from the same subset of observations. As previously mentioned, the main statistics for complete cases are reported in Supplementary Table 4. The results show that our models for predicting mortality outperforms the 4C MS in all statistics. On the other hand, with respect to qCSI, our models for predicting disease severity showed higher accuracy, specificity and AUC, but a smaller sensitivity.

Figure 3:

ROC curves and AUCs for (A) disease severity: (a) quick Covid Severity Index (qCSI), (b) ROX index model – GBM, (c) PaO₂/FiO₂ ratio model – GBM; (B) Mortality: (d) 4C Mortality Score (4C MS), (e) ROX index model – RT, (f) PaO₂/FiO₂ ratio model – RF.

3.4 External validation

The best prediction models were selected and integrated into a system; further details are described in the paragraph below (see COVID-19 Decision Support System). During the fourth wave, the system was used by the physicians of different institutions and was applied to patients with infection by SARS-CoV-2. So far, the validation cohort is made up of 57 patients. The mortality risk prediction model showed an accuracy of 0.94 for both GBM and RF models, whereas the models about safe discharge returned an accuracy of 0.70 and 0.60 (RF and GBM respectively).

Therefore, as expected, the system confirms its high sensitivity, but a lower overall accuracy [34, 35].

3.5 COVID-19 Decision Support System (C19DSS)

We developed the COVID-19 Decision Support System (C19DSS) to enable physicians to effectively use the previous models. The system is made up of a smartphone app used by clinicians, and a server that provides the “intelligence” to the app.

In short, the app is made up of four activities (see Supplementary Figure 1). The first activity is the dashboard, that summarizes the database content and the server connection status. The second activity contains the patient list, plus a set of filters and a button to add a new patient. The third screen shows how to enter the laboratory/clinical data of a new patient, and the button to request the classification to the server. The fourth screen depicts how to edit the patient data, as well as request the classification to the server or delete the patient from the local database.

When the classification process is activated, the app opens an encrypted connection to the server, sends the laboratory data and the ID of the patient (so, no personal data is communicated over the network) to the classification service. Then, the server applies the correct model, depending on the request and on the received data, stores the received data for further analyses, and returns the classification results to the app. The activities briefly described above were designed, developed, tested and redesigned following the UCD methodology, i.e., physicians were involved from the very beginning phases of the design, and then we adapted and improved the design/implementation according to consecutive cycles of usability tests. Physicians reported that the application was easy ed intuitive to use; the process of data entry and classification did not hamper the normal ED work routine. More details on the system, its architecture and on the usability results can be found in a further papers of our research group [19].