Prediction of COVID-19 deterioration in high-risk patients at diagnosis: an early warning score for advanced COVID-19 developed by machine learning

The prediction algorithm and SACOV-19 were developed and validated on patient data from LEOSS, the multicenter international COVID-19 registry comprising over 7000 patients collected in more than 100 study sites (http://​www.​leoss.​net). Inclusion criteria for LEOSS were a laboratory confirmed SARS-CoV-2 infection from any respiratory material and clinical information available on follow-up until the end of the treatment (recovery or death). The day of the first SARS-CoV-2 diagnosis was referred to as the baseline time point. Documentation in LEOSS was performed retrospectively and anonymous. All patients’ variables, and also rational data such as age, BMI or laboratory data was collected in categories. Due to the anonymous data collection, written informed consent of the participants was waived by the respective ethics committees. For patients, recruited in Turkey, informed consent was obtained from the participants upon request of the national ethics committee. To reduce the risk of re-identification, the data was additionally anonymized using the principles used for the LEOSS Public Use File (PUF) we described earlier [22]. Approval for LEOSS data collection and analysis was obtained by the applicable local ethics committees of all participating centers and registered at the German Clinical Trials Registry (DRKS, No. S00021145).

In this study, patients were included who were asymptomatic or exhibited mild symptoms (symptoms of the upper respiratory tract, fever, nausea, emesis or diarrhea) at baseline.

Progression to a complicated or severe stage of COVID-19 during medical consultation/observational period was set as the endpoint (denoted as advanced COVID-19 stage). Since COVID-19 is a multi-organ disease, any incident organ failure during the disease was considered a complication. It was defined by the occurrence of at least one of the following symptoms during the observational period (complicated or critical COVID-19 stage according to LEOSS criteria [21]): need for new oxygen supplementation due to clinical deterioration, oxygen saturation (SO₂) at room air < 90%, partial pressure of oxygen (PaO₂) at room air < 70 mmHg, clinically meaningful increase of oxygen supplementation compared to prior oxygen home therapy, increase of aspartate aminotransferase (AST) or alanine aminotransferase (ALT) > 5 × ULN (upper limit of normal), new cardiac arrhythmia, new pericardial effusion > 1 cm or new heart failure with pulmonary edema, congestive hepatopathy or peripheral edema, catecholamine therapy, life-threatening cardiac arrhythmia, liver failure with an INR > 3.5 (Quick < 50%), a qSOFA score of ≥ 2 or acute renal failure with need of dialysis. The baseline data comprised patient characteristics, symptoms, co-morbidities, known microbiological colonization, preexisting medication, and laboratory and vital parameters.

Using the data from the discovery cohort, patients were randomly separated into an endpoint-balanced training (80%) and a test set (20%). Endpoint balancing was achieved by stratification of the classes by inducing the sampling rate of patients progressing to advanced COVID-19 and reducing the sampling rate of patients not progressing to advanced COVID-19. Binary variables were defined for all baseline patient characteristics. To note, since in the LEOSS database also rational variables were given in categories, no information was lost by this binarization. Missing values or data documented as “unknown”, “not measured” or “not detected” were incorporated in the design of the binary variables. For details of the binary variable computation see Supplementary Text 1. These binary variables were used in the following data processing. The base predictor was constructed using the H2O.ai platform (https://​www.​h2o.​ai) selecting automatically (with h2o.automl) the best suitable machine learning method on the training set. To save computational time, the selection of methods was limited to random forests, gradient boosting machines (gbm), extreme gradient boosting (XGBoost) and StackedEnsemble. The parameters of each method were optimized employing an internal tenfold cross-validation on the training set. The optimal method was then applied to the test set to assess the final performance. In each loop, the best performing predictor was identified from all obtained predictors using the performance measure logloss. The selection of predictors was based on the area under the curve (AUC > 0.75) and logloss < 0.50. A schematic representation of the procedure is shown in Supplementary Figure S1. Variables associated with the "base predictor" were selected according to their scaled importance above 0.05 to obtain the “slim predictor” which based on a reduced set of variables (n = 61). To obtain the best performing predictor based on a minimalistic set of variables, variables of the “slim predictor” were ranked according to their scaled importance. Of these, a smaller set of variables (n = 60) was selected by leaving out the lowest ranking variable, a new predictor trained on the training set and its performance evaluated on the test set. Again, the lowest ranking variable on the remaining set of variables was removed, a new predictor generated and tested in the same way. This procedure was repeated until no variable remained. Out of these predictors, the minimalistic predictor was selected showing the best tradeoff between good performance and minimal set of variables (see “Results”, XGBoost predictors). The robustness of the minimalistic predictor was evaluated by constructing supplementary (mutated) predictors leaving out one variable at a time. To estimate the robustness, the performance of these mutated predictors was compared to the performance of the minimalistic (wildtype) predictor. For the minimalistic predictor, a graphical user interface was implemented in R using the package Shiny and ggplot2. The computational core consists of functionalities employing the packages h2o and lime.

We estimated the discriminative power of each individual patient variable using the discovery set. The predictive power of each variable was estimated based on balanced accuracy. Patients with missing values for the tested variable were omitted. To identify the score (SACOV-19), we used the variables selected for the minimalistic predictor and combined up to a maximum of 16 variables into a predictive score. Each selected variable counted + 1. Together with a threshold T, the score predicted an advanced COVID-19 stage if at least T many of the (binary) variable values of the evaluated score equaled “yes” (+ 1) for a concrete patient. Varying the threshold from 0 to the length of the score, we computed the AUC for each score. We started with computing all scores of lengths two and stored the best 1000 of them according to their AUC. Next, the variables of each of these 1000 scores of lengths two were combined with one of the remaining variables. Doing this for all remaining variables yielded a list of scores of lengths three. Subsequently, we selected the 1000 best scores according to their AUC. This dynamic-programming-like procedure was repeated until a list of 1000 best scores of lengths 16 was compiled. Note, that this heuristic works in reasonable computational time. The rationale for this procedure was that we assumed that sub-scores of well performing scores also perform good. Indeed, we observed that bests-of-lists of length 200 (instead of 1000) yet comprised all the best scores. Out of the list of 16 best scores (with length 1–16), the optimal score was determined by selecting the score with the highest AUC on the test set of the 16 optimal scores. All data processing, modeling and assessment of performances was performed using R (version 3.6.3). Confidence intervals for the odds ratios were calculated using the package “fmsb_0.7.0” [24]. Further used packages were dplyr_1.0.5, h2o_3.30.0.7, lime_0.5.2, ggplot2_3.3.3, liqueueR_0.0.1, arsenal_3.6.2, caret_6.0-86, flexdashboard_0.5.2 and shiny_1.6.0.

We included 3487 out of 6360 patients enrolled in LEOSS in our study, 1223/2819 patients for model discovery and 2264/3541 patients for validation (for details of the selection of patients, see “Methods” and Supplementary Figure S2). The analyzed patients were obtained from 117 LEOSS study sites located in Germany (94.8%, 3307/3487), Turkey (1.9% 66/3487), Belgium (0.8%, 29/3487), Switzerland (0.7%, 25/3487), the United Kingdom (0.7%, 25/3487), Latvia (0.7%, 24/3487), Spain (0.2%, 8/3487), Austria (0.06%, 2/3487), and Italy (0.03%, 1/3487). Patients were recruited either at university hospitals (60.6%, 2113/3487), community hospitals (36.5%, 1274/3487) or medical practices (2.8%, 100/3487). 91.5% of patients (3176/3470; 17 with missing information) were hospitalized during the observation period. In 74.1% of the patients (2345/3165; 322 with missing information) the first positive SARS-CoV-2 test (at baseline) was performed in an inpatient setting. 19.8% of the patients (582/2939; 548 with missing information) were documented as asymptomatic at baseline. Asymptomatic patients had predominantly more documented co-morbidities [21]. The clinical stage of 35.2% (1229/3487) patients worsened to the Advanced COVID-19 stage. The median days from the date of baseline to start of Advanced COVID-19 was five days (inter quartile range 2–7 days). An overview of the patient characteristics and clinical conditions at baseline of the validation cohort is given in Table 1. This data was kept untouched during machine learning and developing the SACOV-19 score. The patient characteristics of the discovery cohort are shown in Supplementary Table S1.

Our goal was to develop a predictor as the basis for deriving a score aiding the front-line physician identifying patients at risk developing Advanced COVID-19. We compiled 472 baseline patient variables (being present to the treating physician) as input for obtaining the “base predictor” and trained machines on data of the discovery cohort. Evaluating the performance on a test set (taken from the discovery cohort) (n = 244), the “base predictor” revealed decent performance (AUC = 0.79 ± 0.11, OR = 7.65 [95% CI 4.13–14.19]) (Supplementary Table S3). Next, we focused on a smaller set of variables for the prediction to simplify the interpretation and to improve generalizability. We obtained an optimized predictor based on n = 61 variables (slim predictor) showing an AUC of 0.80 ± 0.01, OR = 9.14 [95% CI 4.90–17.05] on the test set. Though the new predictor showed a similar performance as the base predictor, it consisted of a considerable reduced number of variables (Table S2, Fig. 1B). To further reduce the number of variables, we computed predictors by iteratively removing variables with minor importance leading to the minimalistic predictor with a similar performance (AUC = 0.80 ± 0.01, OR = 8.20 [95% CI 4.51–14.88], Table S2, Fig. 1C, Supplementary Figures S3A–B). The minimalistic predictor was based on the variables body-mass index (BMI > 24.9 kg/m²), smoking habit (smoker/former smoker), presence of acute kidney injury, dyspnea, oxygen saturation level (< 96%), body temperature (two thresholds, i.e. > 37.3 °C and > 38.9 °C), respiratory rate (two thresholds, > 16/min and > 21/min), C-reactive protein (CRP, 2 thresholds, > 29 and > 119 mg/L), creatinine (≥ ULN, upper limit of normal), LDH (≥ ULN), AST (≥ ULN), gamma-GT (≥ ULN), lymphocyte counts (≥ 3000/µL), and neutrophil counts (≥ 3000/µL). Age was employed along three different thresholds (> 55, > 65 and > 75 years) reflecting the continuously increasing risk with increasing age. Using the (unseen) data from the validation cohort, the minimalistic predictor showed an AUC = 0.71, OR = 4.41 [95% CI 3.57–5.46]. Receiver operating characteristics (ROC) curve of the minimalistic predictor are shown in Fig. 1D. Further performance values are shown in Table S1. A predictor is estimated to be robust if it performs similar under varying input conditions [25]. We constructed predictors by randomly dropping single variables. We observed that this did not influence the performance (Fig. 2A), reflecting the robustness of the minimalistic predictor. Hitherto, the results were based on patients containing missing values. To assess the impact of missing values on the predictive power, we applied the minimalistic predictor to data of patients without missing values for any of the 20 patient variables. We observed a slightly better prediction performance (on the validation set AUC = 0.77 ± 0.02, OR = 6.78 [95% CI 2.74–16.65] and balanced accuracy: 0.72 ± 0.01 using n = 124 patients, Table S3, Fig. 2B, Supplementary Figures S3C).

To summarize, we constructed and internally validated a minimalistic predictor based on 20 patient variables comprising patient characteristics such as age and body mass index, but also vital parameters such as body temperature, respiration and lung parameters, several blood laboratory parameters such as CRP, LDH and creatinine levels, and acute kidney injury at diagnosis. The predictor showed good and stable performance in predicting the development to the advanced COVID-19 stage.

For clinical implementation, we developed an early warning score. Starting with the 20 variables from the minimalistic predictor, we applied the score optimization procedure (described in “Methods”) and identified a predictive score (SACOV-19) based on 11 patient characteristics or 14 binary variables including three binary variables originating from the same categorical variables. The performance was similar as for the machine learning-based predictors (AUC 0.80 ± 0.01) for the discovery set. For the validation set, the AUC was 0.73 ± 0.01. The composition of SACOV-19 is shown in Table 2. A high sensitivity is of particular clinical relevance reducing misclassification of patients in need of hospitalization and close monitoring and who possibly could benefit for medical treatment. This can be achieved using lower thresholds. Selecting a threshold of four, we obtained a sensitivity of 0.90 and an absolute risk reduction of ARR = 0.34 for the validation cohort. 45.7% (717/1,570) of patients with a score of at least 4 deteriorated to advanced COVID-19.

Removing patients with a missing value in at least one of the 14 binary variables, improved the performance for the discovery (AUC = 0.83 ± 0.02, n = 120 patients) and the validation set (AUC = 0.75 ± 0.02, n = 153 patients) (Fig. 2C, Supplementary Figures S3D). Table 3 shows the performances for three different thresholds. To test if our score only works within a hospital setting, we computed the performance also for outpatients and asymptomatic patients. For outpatients (n = 28, after removal of patients with at least one NA in the score variables) the sensitivity was 82% and specificity 53%. For the asymptotic patients the sensitivity was 67% with a specificity of 81% (threshold = 4, n = 29 after removal of patients with at least one NA in the score variables). However, both results show only the tendency as their lower confidence values were not above one, assumedly due to the low patient numbers. To evaluate the predictive power of single variables, we computed their individual performance as a predictor to develop an advanced COVID-19 stage. Table 3 shows the results. The best single variable was oxygen saturation (SO2) smaller than 96% with an AUC of 0.63 ± 0.01 (OR = 3.07 [95% CI 2.34–4.04]). Notably, the top five discriminating variables (oxygen saturation, age, CRP, LDH and temperature) are all part of the minimalistic predictor and of SACOV-19 showing the consistency of the results and the principal relevance of these five variables.

In summary, using the preselected variables from the minimalistic predictor enabled to define a clinical score comprising eleven patient variables with a good performance which is comparable to the machine learning-based predictors.

To illustrate the performance of the minimalistic predictor, we designed a graphical user interface for a quick entry of the values of potential patient variables, followed by the prediction of the investigated endpoint. The web interface (http://​www.​klinikum.​uni-muenchen.​de/​Medizinische-Klinik-und-Poliklinik-II/​de/​sacov19app/​index.​html, login: user, password: sacov19) provides the user with the model-based estimated probability of the patient to develop an advanced COVID-19 stage, the odds ratio, SACOV-19 and the model prediction. Moreover, it provides several graphical presentations to illustrate the impact of the specific variables on the decision. Supplementary Figure S4 and movie M1 shows the web front-end and illustrates its usage (for scientific use).