A new machine-learning-based prediction of survival in patients with end-stage liver disease

Study population

In a retrospective cohort study, we followed 778 consecutive patients, who were recruited during the evaluation process for liver transplantation at University Hospital Leipzig from November 2012 to June 2015. For each patient, we recorded 44 variables, including age, sex, etiology of liver disease, complications as listed in Table 1, and 25 laboratory measurements.

We excluded 124 patients from our analysis (Figure 1). One was younger than 18 years, 24 had had a liver transplant before, and in 99 cases the follow-up data were missing or invalid.

Figure 1:

Flowchart showing inclusion and exclusion of records.

The Ethics Committee at the Leipzig University Faculty of Medicine approved the retrospective usage of the data for our study (reference number: 039/14ff).

MELD scores

MELD and MELD-Na were calculated as described in [3] using the following formulas:

Creatinine, bilirubin, and INR values lower than 1.0 mg/dL were set to 1.0 mg/dL. The maximum allowed creatinine was 4.0 mg/dL. If the patients received dialysis, creatinine was set to 4.0 mg/dL. For MELD-Na, MELD was calculated as above and recalculated if greater than 11 using:

Serum sodium values lower than 125 mmol/L or higher than 137 mmol/L were set to 125 mmol/L and 137 mmol/L, respectively.

MELD 3.0 was calculated as described in [5]:

In contrast to MELD-Na the maximum allowed creatinine was set to 3.0 mg/dL.

The MELD-Plus7 risk score was calculated as described in [10]:

The MELD-Plus9 score was ignored because the length of stay information was not available for our cohort, and the model itself was not superior to MELD-Plus7 in its original publication [10].

Data processing and machine-learning

All data processing, statistical, and machine-learning analyses were performed using R version 4.2.0 [12]. Prior to the analyses, all laboratory values were zlog transformed to approximate a normal distribution as described in [13] and implemented in [14].

where LL and UL are the lower and upper age-, and sex-related reference limits of the corresponding laboratory diagnostic, respectively. Missing laboratory measurements were replaced with a zlog value of zero. In a similar manner, where information about complications or comorbidities was missing, we treated them as not present.

We compared 13 different statistical and machine-learning algorithms using the mlr3 framework [15], [16], [17], [18]. The algorithms used were designed for analysis of survival data: Cox proportional hazards regression model [19], [20], [21]; penalized regressions, namely, two different implementations of the lasso (least absolute shrinkage and selection operator) regression [22], [23], [24], [25], [26], two different implementations of the ridge regression [23], [24], [25], [26], and the elastic net regression [23, 24]; two different random forest implementations [27], [28], [29], [30]; extreme gradient boosting [31, 32]; two different support vector machines [33, 34]; and two neural networks [35], [36], [37].

We used nested resampling to avoid feature- and model-selection bias to obtain reliable metrics to choose the best model. We thus applied a 5-fold inner and 3-fold outer cross-validation that was repeated 10 times. We searched the hyperparameter space using a grid search with a resolution of 10. The models were evaluated using the concordance index, which is a measure of rank correlation between predicted and observed events [38]. Subsequently, we selected the model with the highest median concordance index. If multiple models were comparable, we chose the simpler (more penalized) one.

After selecting the final model, we trained and validated it as described in [39]. Briefly, we developed our model using a 3-fold cross-validation that was repeated 10 times and finally predicted survival at 90 days. The predictions were sorted and grouped into equal-sized intervals so that there were 50 patients in each interval. For each group, the mean survival probability and the difference to the Kaplan–Meier estimate was calculated. The same procedure was repeated for 200 bootstrap samples. Subsequently, the differences between predicted survival and Kaplan–Meier estimates of the bootstrap samples were averaged and added to the original model difference to obtain a bias-corrected estimate between predicted and observed survival.

The final model was compared against the MELD, MELD-Na, MELD 3.0, and MELD-Plus7 scores using the area under the time-dependent receiver operating characteristic curve (AUROC) based on the nonparametric inverse probability of censoring weighting estimator (IPCW) [40, 41]. Testing was applied as described in [40]. A p<0.05 was considered a statistically significant difference.

Further we determined the variable importance by counting how often the variable was chosen during the feature selection in the bootstrap process described above and by the logrank value returned by one random forest implementation [28].

Summary tables were built using the gtsummary package [42, 43]

Baseline characteristics

In this study, we analyzed a cohort of 654 patients evaluated for liver transplantation (Table 1). Of these, 414 (63%) were men. The median (IQR) follow-up time was 191 (38, 384) days. Within the follow-up time, 60 (9.2%) patients received a liver transplant and were censored for the analysis beyond this time point. The median (IQR) age of the cohort was 58 (52, 64) years.

Most of the patients presented with cirrhosis (594 [91%]). The remaining patients had acute liver failure (8 [1.2%]) or chronic liver insufficiency (51 [7.8%]). At evaluation time, the median (IQR) MELD, MELD-Na, and MELD 3.0 scores were 11 (8, 17), 11 (8, 20), and 12 (8, 19), respectively. Within 90 days, 81 (14%) patients had died. In nearly two thirds of all patients (411 [63%]), alcohol was the main cause of their end-stage liver disease. The remaining patients had viral infections – hepatitis B (20 [3.1%]) or C (45 [6.9%] – autoimmune hepatitis (31 [4.7%]), primary sclerosing cirrhosis (16 [2.4%]), primary biliary cirrhosis (17 [2.6%]), unknown causes (non-alcoholic steatohepatitis (48 [7.3%]), or cryptogenic hepatitis (68 [10%])). The most common complications among all patients were gastrointestinal bleedings in 162 (25%) patients, followed by hepatocellular carcinoma (HCC) in 121 (19%) and spontaneous bacterial peritonitis (SBP) in 91 (14%) patients. Just 33 (5.1%) patients required renal dialysis.

MELD scores

We divided our cohort into 5 MELD-Na-score risk categories, with 257, 234, 105, 38, and 11 patients, respectively. As shown in Table 2, our observed 90-day mortality was 1.2–2.2 times higher than the predicted one [44]. Only in the group with the lowest MELD-Na scores was the mortality much lower.

AMPEL MELD

The penalized regression and random forest models resulted in a similar high median performance in the benchmark process (see Supplementary Material). In favor of simpler models, we chose a lasso-based approach for the final model development. The classical Cox model, the data and computational intense algorithms extreme gradient boosting, the support vector machines, and the neural networks yielded a worse performance.

The resulting AMPEL model of end-stage liver disease (AMELD) uses 6 predictors. Besides the classical MELD variables INR and (direct) bilirubin, the lasso regression selected cystatin C over creatinine, as well as IL-6, total protein, and cholinesterase (Table 3).

The 90-day survival estimate of a given patient can be calculated as shown below:

where S ₀ is the baseline survival estimate from Table 4 and LP is the linear predictor of the coefficients in Table 3.

Model comparison

AMELD offers the best AUROC 0.961 (95% CI, 0.942–0.980) of all competitors (Figure 2). However, after correction for multiple testing, the difference against the second-best model, MELD 3.0 (AUROC 0.936 [95% CI, 0.910–0.963]), is not statistical significant (p=0.15; see Table 5 for complete comparison).

Figure 2:

Receiver operating characteristic (ROC) curve. Area under the time-dependent ROC curve (AUC) based on the nonparametric inverse probability of censoring weighting estimate (IPCW) for AMELD, MELD, MELD-Na, MELD 3.0, and MELD-Plus7, as described in [40].

Figure 3 depicts the calibration curve for AMELD and the competing MELD scores. While the prediction for low-risk patients is similar to other scores, AMELD predicts the medium-risk patients more accurately.

Figure 3:

Calibration of AMELD in comparison to MELD, MELD-Na, MELD 3.0, and MELD-Plus7. The points mark the mean predicted survival of 50 patients per interval against their observed mean survival. The error bars correspond to the 95% confidence interval of the survival estimate.

Variable importance

The AMELD selected features IL-6, total protein, INR, cholinesterase, cystatin C, and direct bilirubin are the 6 highest ranked features in the bootstrap process. In the random forest logrank they are among the 12 highest ranked features out of 42 features. Sodium and albumin introduced by MELD-Na and MELD 3.0 are on position 18 and 12 for the bootstrap and 20 and 5 for the random forest variable importance ranking, respectively (see Supplement for the complete rankings).

Limitations

Our study has several limitations. Principally, it is a single-center, retrospective analysis lacking an external validation cohort. The data were collected during the evaluation process for liver transplantation at University Hospital Leipzig. There was no controlled follow-up process. We excluded all patients who had already undergone liver transplantation, which could artificially lower the disease severity of the cohort. Furthermore, most of our patients suffered from ethyltoxic liver cirrhosis (63%), which may have limited the transfer to other etiologies.

With a median MELD-Na score of 11, our cohort was healthier than those of previous studies that reported a score of 14–18.5 [4, 10, 44]. However, the 90-day mortality rate of 81 (14%) was comparable to the mortality rates seen in earlier studies, ranging from 6 to 30% [1, 4, 10]. Although our cohort was healthier, with a medium mortality rate, the classical MELD score underestimates the observed mortality by 1.2–2.2 times. Unfortunately, our sample size was low in the high-risk patient category, resulting in a poor but better than MELD-based prediction performance by our AMELD for very ill patients.