Development and validation of nomogram to predict severe illness requiring intensive care follow up in hospitalized COVID-19 cases

This retrospective cohort study was carried out in Ankara City Hospital, set apart as the main pandemic response center in Ankara with 3810 beds, of which 696 are intensive care beds. The ethical approval was obtained from Ankara City Hospital Ethical Committee 1. Verbal consent was obtained after the patients were informed that their medical records would only be used in scientific studies after anonymization of their personal information. All patients older than 18 years who were hospitalized with the diagnosis of COVID-19 infection between March 15, 2020, and June 15, 2020, were included in the study. Only COVID-19 patients with the definite diagnosis were included in the study. The diagnosis was confirmed with polymerase chain reaction (PCR) for SARS-CoV-2 performed based on the protocol established by the World Health Organization (WHO) interim guideline [3]. Patients were monitored up to June 30, 2020, the final date of follow up. Patients with a negative SARS-CoV-2 test even if typical chest computed tomography (CT) findings and those who were still in hospital at the moment of final date of follow up (if no death or discharge) were excluded.

The patients with a severe and critical illness were candidate to ICU follow-up based on the WHO COVID-19 disease severity classification. Patients with pneumonia and one of the following: > 30 breaths/min; severe respiratory distress; or O₂ saturation (SpO₂) < 90% on room air were considered severe. Patients were considered critical if they had acute respiratory distress syndrome (ARDS) or other respiratory failure requiring mechanical ventilation, or septic shock, and/or organ failure requiring ICU follow up [1]. The decision of ICU admission was made by intensive care specialists. ICU admission criteria were respiratory rate ≥ 30, SpO₂ < 90% or partial oxygen pressure (PaO₂) < 70 mmHg on room air despite nasal oxygen support of 5 lt/min or above, PaO₂/fraction of inspired oxygen (FiO₂) < 300. The primary outcome was defined as severe illness that required ICU follow up. The patients were classified as cases who required ICU follow up and those who did not require ICU follow up based on disease severity.

To collect data, a special form was created for COVID-19 patients, containing information of patients at the admission and follow up. The parameters included in special patient forms were age, gender, smoking status, comorbid diseases, the symptoms of fever and dyspnea, oxygen saturation (SpO₂), quick sequential organ failure assessment (qSOFA) at admission. The forms also included following laboratory and radiological tests: complete blood counts, serum biochemistry, C-reactive protein (CRP), procalcitonin (PCT), coagulation tests, ferritin, D-dimer, troponin I, and chest CT. The clinical outcomes were defined as requirement of ICU or discharge from hospital.

Data were collected prospectively. In case of patient death or discharge, all the missing laboratory records in patient files were completed from the hospital database and registered in an electronic recording system and uploaded collaboratively to an online database created specifically for COVID-19 patients. Data cut-off for the study was June 30, 2020. Data were recorded to the system by the physicians who followed up the patients from different departments including infectious disease, internal medicine, respiratory disease, and anesthesiology and reanimation. After patient records were compiled, the data was checked by two independent controllers who were infectious disease physicians. Patients with more than 30% missing data were not included in the study.

For the development of a functional nomogram, patient data obtained on the day of hospitalization were used. The predictors were selected from the factors that affect the prognosis of the patients such as age and the presence of comorbidities, and clinical features and easily accessible, practical, and quickly performed laboratory parameters. The potential predictive parameters were determined as age, gender, the presence of fever and dyspnea, and qSOFA on admission, clinical risk factors (comorbidities including hypertension, coronary arterial disease, diabetes mellitus, chronic lung disease, malignity, and number of comorbidities), SpO2 and laboratory parameters which are found significant covariates on COVID-19 infection including white blood cell (WBC), monocyte, neutrophil to lymphocyte ratio (NLR), hemoglobin (HGB), platelet count, urea, creatinine, glomerular filtration rate (GFR), albumin, aspartate transaminase (AST), alanine transaminase (ALT), lactate dehydrogenase (LDH), creatine kinase (CK), troponin I, CRP, PCT, ferritin, prothrombin time (PT), activated partial thromboplastin time (aPTT), D-dimer, international normalized ratio (INR) and fibrinogen, and presence of bilateral infiltration on chest CT. A total of 35 predictors were included in the construction of nomogram in the beginning. After determining potential predictors, their association with ICU hospitalization was investigated.

Statistical analyses were performed using R software version 4.0.4 (R Foundation for Statistical Computing, Vienna, Austria) and IBM SPSS Statistics version 23.0 for Windows (IBM Corporation, Armonk, NY). Missing data pattern and mechanism were evaluated using R packages VIM, mice, MissMech, and BaylorEdPsych [4‐7]. Listwise deletion (complete-case method) was applied for handling missing data due to the MNAR mechanisms in numerical measurements. Laboratory parameters were discretized using Receiver Operating Characteristic (ROC) analysis. Thereafter, Youden’s J Index was used for determining optimal cut-off points of the numerical variables in predicting ICU admission. Descriptive statistics were presented as median with interquartile range (IQR) for continuous variables since the distribution of the variables were skewed and contains extreme values. Frequency and percentages were presented as descriptive statistics for categorical variables. Mann–Whitney U test was used in the comparison of continuous variables between patients admitted to ICU and those without need for ICU follow up due to the violation of the parametric test assumptions. Pearson χ2 test was used for testing independence between ICU admission status and other categorical variables when test requirements were satisfied. Otherwise, Fisher’s Exact test was used.

To estimate ICU admission status, univariate logistic models were constructed using bootstrap sampling with 1000 samples and bootstrap estimated p values were evaluated. The variables with bootstrapped p-value below 0.25 as considered candidate variable for multivariable analysis [8]. Numeric variables which were included in the multivariable analysis are evaluated for linearity in logit and multicollinearity was investigated using Variance inflation factor (VIF) before applying the variable selection method [9]. Both univariate and multivariable logistic regression analysis were carried out using R rms package [10]. Fast backward elimination method for variable selection was carried out with bootstrap sampling (1000 successful bootstrap samples) to develop a parsimonious model for predicting ICU admission. Estimations obtained from multivariable model was based on penalized maximum likelihood estimations with best penalty parameter obtained using pentrace function in R rms package. In addition, final model is selected according to the Akaike Information Criteria (AIC).

Selected variables were represented as odds ratio (OR) with bootstrapped 95% confidence interval (CI) and two-tailed p-values. Discrimination was evaluated using bias-corrected Harrell’s Concordance index (C-index). Bias-corrected Harrel’s C-index was calculated from rms package validate function with 1000 successful bootstrap samples. Validated model is checked for multicollinearity. Hence, VIF values of all the predictor variables in the multivariable model were below 5. In addition, linearity in logit assumption was satisfied. Nomogram was constructed based on the final validated model for estimating the admission to ICU and provided a quantitative tool for physicians to assess the individual probability of ICU admission. In addition, model’s discriminative power was evaluated with ROC analysis using R pROC package [11]. The Area Under the Curve (AUC) was obtain with 95% Hereafter in the article, ‘corrected C-index” will be used to indicate that AUC value/C-index was obtained from bootstrap samples and bias-corrected, and “AUC” will be used to indicate crude AUC value/C-index which was obtained from one ROC Curve.

Calibration plots were developed to assess the predictive accuracy and agreement between predicted and observed ICU admission with 1000 bootstrap samples and calibration curve analyses were performed in addition to Hosmer–Lemeshow goodness of fit evaluation. In addition, both the unreliability test and the calibration test are performed to evaluate good calibration.

In addition, false-negative (i.e. not admitting the patient in the ICU when the patient needs intensive care) is far more harmful than the false positive (i.e. admitting the patient in the ICU when the patient doesn’t need intensive care at all) in the present study. Therefore, decision curve analysis was performed since the possibility of ICU need of the patient is more crucial for patients' well-being. In a decision curve analysis, a low-risk threshold probability might indicate that delaying the ICU admission is far more harmful than early admission; a higher threshold might indicate that waiting the parameters to reach critical levels is relatively more harmful than unnecessary ICU admission.

All analysis related to the evaluating classification performance and calibration of the prognostic accuracy of the nomogram model were performed according to TRIPOD guidelines [12].

Study included 1022 patients with laboratory confirmed COVID-19. The dataset consists of these 1022 patient records contained missing data ranged between 0.1% and 42%. Variables which consist of more than 20% percent of missing data were excluded from the analysis. The proportion of missing data ranged from 0.1% to 9% after exclusion of the variables which consist of more than %20 missing observations (smoking status, myoglobin, symptom duration before hospital admission). The MCAR test in the R MissMech package [4‐7] was used to assess whether the missing data mechanism is Missing Completely at Random (MCAR). MCAR hypothesis was rejected at 0.05 level. Therefore, after list-wise deletion 686 cases out of 1022 patient remained for further analysis. Brief overview of the missing data structure is represented graphically (Fig. 1).

The demographic characteristics, comorbidities, and initial laboratory parameters of patients are shown in Table 1. Of the 686 patients, 104 (15.2%) required ICU follow up during hospitalization. There was no difference in gender between the groups (p = 0.057), 52.4% of patients who did not need ICU follow up and 62.5% of those who needed ICU follow up were male. The median age was higher in the patients requiring ICU follow up (67, IQR 30–54) than those who did not need ICU follow up (42, IQR 54–76) (p < 0.001). The patients who required ICU follow up had significantly higher rates of hypertension (40.4% vs 12.8%), coronary arterial disease (20.2% vs 6.4%), diabetes mellitus (29.3% vs 10.1%), chronic pulmonary disease (21.2% vs 6.4%), and malignity (9.6% vs 1.9%) compared to those who did not need ICU admission (for all, p < 0.001). Fever (48.1% vs 34.0%, p = 0.006) and dyspnea (59.9% vs 19.4%, p < 0.001) were significantly more frequent on admission in the patients who needed ICU follow up. Laboratory features of two groups were compared. All initial parameters except ALT and aPTT were significantly different between the two groups (for all parameters, p < 0.001).

Univariate analysis indicates that common laboratory features have possible effect on the patient’s requirement of intensive care as well as the patient characteristics such as age, gender and comorbidities (Table 2).

A total of 35 predictor were chosen for the development of nomogram predicting of ICU admission in hospitalized patients with COVID-19. All predictors have p-value below 0.25. Therefore, they were all considered as candidate variables for multivariable analysis except QSOFA score due to sparsity and quasi-complete separation problem (All the patients whose QSOFA score equals 2 had been admitted to the ICU).

The nomogram was constructed using the data obtained from 686 patients’ records. Afterwards, all laboratory parameters were discretized by using optimal cut-off points obtained from ROC analysis with Youden’s J Index. To construct multivariable nomogram model for estimating ICU admission status of inpatients, first candidate variables were selected using univariate analysis (Table 2). All variables had p value below 0.25 and considered for the multivariable analysis. After candidate variables were chosen, multivariable model was constructed with 34 variables including demographical, clinical and laboratory parameters. The parameters included in the initial model were age, gender, hypertension, coronary arterial disease, diabetes mellitus, chronic pulmonary disease, malignity, number of comorbidities, fever and dyspnea on admission, SpO2, WBC, monocyte, NLR, HGB, platelet count, urea, creatinine, GFR, AST, ALT, albumin, LDH, CK, CRP, PCT, ferritin, troponin, PT, aPTT, INR, D-dimer, fibrinogen and bilateral infiltration on CT. Final model was selected according to the Akaike Information Criteria (AIC), and only SpO2, LDH, CRP, PCT and troponin, which were shown to be independent risk factors for predicting ICU admission, were included. The present nomogram calculates the risk for requirement of ICU in hospital admission of patients using these 5 parameters (Fig. 2). Additional information on score assignment for each variable and calculation the risk for ICU admission based on total point is shown in Additional file 1. According to the nomogram model, the risk of ICU admission is 4.4 (95% CI 2.48–7.72) times higher in patients with oxygen saturation equal and below 94.5% compared to the patients with oxygen saturation above 94.5% (p < 0.0001). In addition, the risk is 3.1 (95% CI 1.76–5.53) times higher in patients with LDH level equal and above 286.5 U/L compared to the patients with LDH level below 286.5 U/L (p < 0.0001) while it is 2.5 (95% CI 1.37–4.63) times higher in patients with CRP level equal and above 0.0275 g/L compared to the patients with CRP level below 0.0275 g/L (p = 0.0029). Whereas the risk is 3.4 (95% CI 1.89–5.94) times higher in patients with PCT level equal and above 0.085 pg/mL compared to the patients with PCT level below 0.085 pg/mL (p < 0.0001), it is 3.6 (95% CI 2.03–6.22) times higher in patients with troponin level equal and above 5.9 ng/L compared to the patients with troponin level below 5.9 ng/L (p < 0.0001) (Table 3).

Based on these 5 independent risk factors included in the nomogram for ICU admission, a web-based calculation tool was constructed. The clinician can easily access the calculation tool using the online website at https://​achcovid19.​com/​prj/​f?​p=​126:​1.

The nomogram model had a significantly high predictive value for the development of ICU needs in hospitalized patients. The model had an AUC of 0.93 (0.902–0.950) (Fig. 3a). In addition, we evaluate the validation of the final model using bootstrap resampling method and obtained corrected C-index of the nomogram as 0.91 (95% CI 0.899–0.947) which implies exceptionally good discriminative value for differentiating inpatients who needed ICU follow up from those who did not.

Furthermore, the decision curve revealed that when threshold probability is between 0.15 and 0.85, predicting ICU admission by using our nomogram model would provide higher net benefit than the admitting all the patients to the ICU (All) or admitting none of the patients to the ICU (None) (Fig. 4).

Hosmer–Lemeshow goodness of fit test indicated that there is no significant difference between the predictive calibration curve and the ideal curve for predicting the ICU status of the patients (X-squared = 4.3284, df = 8, p-value = 0.8263) (Fig. 3b). The calibration curves and Hosmer–Lemeshow test results indicates that the nomogram model is calibrated. In addition, significance of miscalibration of the model is evaluated using unreliability test (p = 0.8197) and calibration test (p = 0.1703) which indicates that the model is statistically well-calibrated.