Background
Complete separation | Quazi-complete separation | ||||||
---|---|---|---|---|---|---|---|
Y | Y | ||||||
1 | 0 | 1 | 0 | ||||
X | A | 10 | 0 | X | A | 10 | 0 |
B | 0 | 10 | B | 2 | 8 |
Methods
Maximum likelihood based logistic regression model
Penalized methods for logistic regression model
Firth’s penalized method
Penalized method based on logF prior
Ridge penalized method
Evaluating predictive performance
Software
Results
Example data: stress echocardiography data
Simulation study
Assessing the properties of the regression coefficients
Estimates | Relative bias (%) | MSE | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Coefficient | Prev.(%) | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE |
5.5 | 0.33 | 0.29 | 0.30 | 0.28 | 0.26 | 10.84 | -3.56 | -1.29 | -6.82 | -14.72 | 0.25 | 0.19 | 0.20 | 0.18 | 0.10 | |
11.5 | 0.33 | 0.31 | 0.32 | 0.31 | 0.26 | 8.71 | 2.68 | 5.46 | 2.44 | -12.23 | 0.10 | 0.09 | 0.10 | 0.09 | 0.07 | |
β
1
| 20.4 | 0.30 | 0.29 | 0.29 | 0.29 | 0.24 | 0.14 | -4.54 | -1.81 | -3.62 | -19.50 | 0.07 | 0.06 | 0.06 | 0.06 | 0.05 |
39.6 | 0.31 | 0.30 | 0.31 | 0.30 | 0.25 | 4.13 | -0.14 | 2.62 | 1.19 | -16.32 | 0.04 | 0.04 | 0.04 | 0.04 | 0.03 | |
59.9 | 0.31 | 0.29 | 0.30 | 0.30 | 0.25 | 1.68 | -2.48 | 0.20 | -1.19 | -16.87 | 0.05 | 0.04 | 0.04 | 0.04 | 0.04 | |
5.5 | 0.80 | 0.87 | 0.86 | 0.71 | 0.58 | -11.19 | -3.41 | -4.19 | -21.26 | -35.13 | 0.66 | 0.76 | 0.75 | 0.50 | 0.36 | |
11.5 | 0.98 | 0.91 | 0.91 | 0.82 | 0.78 | 8.92 | 1.50 | 0.67 | -8.85 | -13.36 | 0.48 | 0.42 | 0.42 | 0.33 | 0.29 | |
β
2
| 20.4 | 0.95 | 0.90 | 0.89 | 0.84 | 0.76 | 5.34 | 0.02 | -0.74 | -6.12 | -15.61 | 0.27 | 0.23 | 0.23 | 0.21 | 0.23 |
39.6 | 0.92 | 0.89 | 0.89 | 0.86 | 0.75 | 2.19 | -0.74 | -1.50 | -4.94 | -16.14 | 0.18 | 0.16 | 0.16 | 0.15 | 0.18 | |
59.9 | 0.92 | 0.89 | 0.89 | 0.86 | 0.75 | 2.36 | -0.56 | -1.33 | -4.77 | -16.36 | 0.15 | 0.14 | 0.14 | 0.13 | 0.17 |
Assessing the predictive performance
Stress echocardiography simulation
-
Fit the following logistic regression model for the response “any cardiac event” with Firth’s penalized method (to avoid bias in the estimate of the regression coefficient) to obtain the true model:$$\begin{array}{*{20}l} {}\text{logit(Pr(Cevent=1))}&=\beta_{0}+\beta_{1}{dobef}+ \beta_{2} wma +\beta_{3} posse\\&\quad+\beta_{4} bsef+\beta_{5} ht+\beta_{6} age \end{array} $$
-
To create a training data, choose the EPV and prevalence (prev), and then calculate sample size for the respective EPV given the number of predictors p as \(n=\frac {EPV \times p}{prev}\). Sample with replacement the n values of the covariates in the true model from original data. For each of the n values of the covariates, simulate new responses from Bernoulli distribution with the probability calculated from the fitted model. However, replace the value of β 0 by -0.65 to confirm the prevalence of the response (prev), on an average, 15.5% for all EPV scenarios.
-
With this combination, check and record if separation occurred due to any of the binary covariates (‘posse’ or ‘wma’, or ‘ht’ or combination of them). Otherwise to create separation, enlarge the true value of the respective coefficient of the binary covariate to some extents. Note that the chances of separation is expected to increase with decreasing EPV value.
-
To create a test dataset, sample with replacement m×n (m times of the original data of size n=558, we considered m=2) values of the covariates. Then simulate the corresponding new responses from the same true model used for simulating training data.
-
Repeat the steps (ii)-(iii) to produce 1000 training and 1000 test datasets.
-
Fit the risk models (using MLE and all types of penalized regression methods under study) to each of the training data sets and check whether convergence was achieved. Then evaluate their predictive performance (if convergence achieved) by means of calibration slope, AUC, root Brier score, and average predictive probability (APP) using the corresponding test dataset. Summarize the predictive performance over the number of simulations for which convergence is achieved.
Further simulation
-
For creating training data, choose the EPV and prevalence (prev) and calculate sample size (n) for the given EPV value and the number of predictors using the same formula previously used.
-
For each observation in training data, first simulate three continuous predictors (X 1,X 2,X 3) independently from standard normal distribution and two binary predictors (X 4,X 5) independently from Bernoulli distribution one with low (20%) and the other with high (60%) prevalence.
-
Simulate the corresponding responses from Bernoulli dis-tribution with probability calculated from the true model:$$\begin{array}{*{20}l}{}\text{logit}(\pi)=\beta_{0}+\beta_{1} X_{1} +\beta_{2} X_{2}+ \beta_{3} X_{3}+\beta_{4} X_{4}+\beta_{5}X_{5}. \end{array} $$For the model with weak predictive ability, the values of the regression coefficient were set as β 0=−1.5β 1=0.2,β 2=0.5β 3=−0.03,β 4=0.05 and β 5=−0.6 and for the model with strong predictive ability, the corresponding true values were set as β 0=−3.5,β 1=1.2,β 2=−0.9β 3=0.9,β 4=1.2 and β 5=1.2. In each case, the value of the β 0 confirms the desired prevalence of the response. With this combination, the binary covarites X 5 in the model with weak predictive ability and X 4 in the model with strong predictive ability create separation in some of the simulations. Check and record if separation occurred.
-
Create test data with size 1000 (much larger than the training data) for the similar level of EPV and prevalence. For each observation in the test data, simulate the same predictors as in the test data and the corresponding response from the same true model.
-
Repeat the steps (ii)-(iv) to produce 1000 training and 1000 test datasets
-
Fit risk models (using all methods) using training data, count if convergence was achieved for the respective model, and evaluate their predictive performance (if convergence was achieved in training data) using test data as before. Finally summarize the predictive performance over the number of simulations for which convergence is achieved.
Model with weak predictive ability | |||||||||||
Calibration slope, Max MCE=0.0235 | AUC, Max MCE=0.0012 | ||||||||||
EPV (N) | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | |
2(67) | Mean | 0.367 | 0.414 | 0.383 | 0.424 | 1.029 | 0.606 | 0.605 | 0.605 | 0.607 | 0.628 |
SD | 0.277 | 0.303 | 0.281 | 0.302 | 0.847 | 0.060 | 0.058 | 0.059 | 0.059 | 0.042 | |
3(100) | Mean | 0.472 | 0.512 | 0.487 | 0.517 | 1.027 | 0.613 | 0.613 | 0.613 | 0.614 | 0.626 |
SD | 0.305 | 0.326 | 0.311 | 0.324 | 0.757 | 0.054 | 0.054 | 0.054 | 0.054 | 0.041 | |
5(167) | Mean | 0.621 | 0.658 | 0.637 | 0.658 | 1.055 | 0.629 | 0.630 | 0.630 | 0.630 | 0.635 |
SD | 0.317 | 0.328 | 0.317 | 0.323 | 0.667 | 0.046 | 0.046 | 0.046 | 0.046 | 0.039 | |
10(334) | Mean | 0.797 | 0.814 | 0.801 | 0.812 | 1.076 | 0.645 | 0.645 | 0.645 | 0.646 | 0.646 |
SD | 0.286 | 0.289 | 0.282 | 0.286 | 0.504 | 0.037 | 0.037 | 0.037 | 0.037 | 0.035 | |
root Brier Score, Max MCE=0.0007 | APP (True 0.152), Max MCE=0.0015 | ||||||||||
EPV(N) | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | |
2(67) | Mean | 0.370 | 0.369 | 0.367 | 0.365 | 0.360 | 0.159 | 0.178 | 0.154 | 0.153 | 0.156 |
SD | 0.022 | 0.019 | 0.019 | 0.018 | 0.017 | 0.045 | 0.041 | 0.044 | 0.044 | 0.044 | |
3(100) | Mean | 0.363 | 0.362 | 0.361 | 0.360 | 0.358 | 0.156 | 0.171 | 0.154 | 0.154 | 0.155 |
SD | 0.018 | 0.017 | 0.017 | 0.017 | 0.016 | 0.035 | 0.033 | 0.035 | 0.035 | 0.035 | |
5 (167) | Mean | 0.357 | 0.357 | 0.357 | 0.356 | 0.355 | 0.153 | 0.163 | 0.153 | 0.153 | 0.152 |
SD | 0.017 | 0.016 | 0.017 | 0.016 | 0.016 | 0.028 | 0.027 | 0.027 | 0.027 | 0.027 | |
10 (334) | Mean | 0.354 | 0.354 | 0.354 | 0.354 | 0.354 | 0.151 | 0.157 | 0.151 | 0.151 | 0.151 |
SD | 0.016 | 0.015 | 0.016 | 0.016 | 0.015 | 0.020 | 0.019 | 0.020 | 0.020 | 0.019 | |
Model with strong predictive ability | |||||||||||
Calibration slope, Max MCE=0.0344 | AUC, Max MCE=0.0024 | ||||||||||
EPV (N) | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | |
2(67) | Mean | 0.659 | 0.825 | 0.784 | 0.890 | 1.252 | 0.831 | 0.831 | 0.832 | 0.834 | 0.832 |
SD | 0.296 | 0.310 | 0.268 | 0.273 | 0.742 | 0.039 | 0.039 | 0.038 | 0.037 | 0.037 | |
3 (100) | Mean | 0.774 | 0.888 | 0.857 | 0.931 | 1.125 | 0.845 | 0.845 | 0.846 | 0.846 | 0.845 |
SD | 0.236 | 0.251 | 0.231 | 0.233 | 0.292 | 0.028 | 0.028 | 0.028 | 0.028 | 0.028 | |
5(167) | Mean | 0.868 | 0.934 | 0.917 | 0.963 | 1.066 | 0.854 | 0.854 | 0.854 | 0.855 | 0.854 |
SD | 0.218 | 0.226 | 0.216 | 0.217 | 0.224 | 0.024 | 0.023 | 0.023 | 0.023 | 0.023 | |
10(334) | Mean | 0.933 | 0.959 | 0.955 | 0.979 | 1.016 | 0.860 | 0.860 | 0.860 | 0.860 | 0.860 |
SD | 0.167 | 0.169 | 0.166 | 0.167 | 0.159 | 0.022 | 0.022 | 0.022 | 0.022 | 0.022 | |
root Brier Score, Max MCE=0.0009 | APP (True 0.162), Max MCE=0.0014 | ||||||||||
EPV (N) | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | |
2 (67) | Mean | 0.338 | 0.331 | 0.330 | 0.327 | 0.328 | 0.172 | 0.182 | 0.164 | 0.163 | 0.167 |
SD | 0.030 | 0.022 | 0.021 | 0.020 | 0.019 | 0.045 | 0.040 | 0.042 | 0.042 | 0.042 | |
3(100) | Mean | 0.323 | 0.321 | 0.321 | 0.320 | 0.320 | 0.165 | 0.175 | 0.163 | 0.163 | 0.164 |
SD | 0.018 | 0.016 | 0.017 | 0.016 | 0.016 | 0.033 | 0.032 | 0.033 | 0.033 | 0.032 | |
5(167) | Mean | 0.316 | 0.315 | 0.315 | 0.315 | 0.315 | 0.163 | 0.170 | 0.163 | 0.163 | 0.163 |
SD | 0.016 | 0.015 | 0.015 | 0.015 | 0.015 | 0.026 | 0.025 | 0.025 | 0.025 | 0.025 | |
10(334) | Mean | 0.310 | 0.310 | 0.310 | 0.310 | 0.310 | 0.163 | 0.166 | 0.163 | 0.163 | 0.163 |
SD | 0.016 | 0.015 | 0.015 | 0.015 | 0.015 | 0.019 | 0.019 | 0.019 | 0.019 | 0.019 |
Illustration using stress echocardiography data
Modeling the risk of MI | |||||
MLE | FIRTH | logF(1,1) | logF(2,2) | RIDGE | |
Dobutamine EF | -0.0503 | -0.0492 | -0.0508 | -0.0513 | -0.0413 |
(0.0183) | (0.0178) | (0.0183) | (0.0182) | (0.0161) | |
Positive DSE | 1.272 | 1.241 | 1.185 | 1.109 | 0.994 |
(0.549) | (0.531) | (0.533) | (0.518) | (0.469) | |
Hist. of HT | 1.115 | 0.923 | 0.973 | 0.866 | 0.657 |
(0.789) | (0.716) | (0.716) | (0.662) | (0.542) | |
Intercept | -1.253 | -1.028 | -1.057 | -0.901 | -1.243 |
(1.351) | (1.284) | (1.307) | (1.275) | (1.132) | |
Modeling the risk of CABG | |||||
Dobutamine EF | -0.0634 | -0.0506 | -0.0518 | -0.0523 | -0.0420 |
(0.0181) | (0.0177) | (0.0181) | (0.0181) | (0.0161) | |
Positive DSE | 1.568 | 1.190 | 1.137 | 1.068 | 0.971 |
(0.551) | (0.529) | (0.531) | (0.516) | (0.468) | |
Intercept | 0.272 | -0.224 | -0.206 | -0.145 | -0.683 |
(1.122) | (1.120) | (1.140) | (1.131) | (1.029) | |
Modeling the risk of PTCA | |||||
Positive DSE | 0.825 | 0.820 | 0.770 | 0.722 | 0.579 |
(0.498) | (0.481) | (0.483) | (0.470) | (0.409) | |
Base EF | -0.0381 | -0.0375 | -0.0389 | -0.0396 | -0.0306 |
(0.0204) | (0.0198) | (0.0202) | (0.0201) | (0.0168) | |
Hist. of MI | 1.168 | 1.125 | 1.118 | 1.072 | 0.867 |
(0.533) | (0.517) | (0.515) | (0.499) | (0.412) | |
Hist of PTCA | 1.304 | 1.310 | 1.211 | 1.127 | 1.087 |
(0.617) | (0.591) | (0.602) | (0.588) | (0.555) | |
Intercept | -1.754 | -1.661 | -1.646 | -1.548 | -1.809 |
(1.184) | (1.151) | (1.167) | (1.153) | (0.968) | |
Modeling the risk of cardiac death | |||||
Positive DSE | 1.084 | 1.061 | 1.026 | 0.974 | 0.873 |
(0.489) | (0.474) | (0.478) | (0.467) | (0.436) | |
Hist. of DM | 1.083 | 1.047 | 1.025 | 0.973 | 0.784 |
(0.495) | (0.480) | (0.481) | (0.468) | (0.419) | |
Age | 0.0347 | 0.0328 | 0.0344 | 0.0342 | 0.0229 |
(0.0240) | (0.0236) | (0.0238) | (0.0236) | (0.0188) | |
Intercept | -6.040 | -5.783 | -5.965 | -5.899 | -4.960 |
(1.787) | (1.750) | (1.769) | (1.753) | (1.379) | |
Positive DSE | 1.064 | 1.045 | 1.047 | 0.989 | 0.940 |
(0.332) | (0.327) | (0.327) | (0.264) | (0.248) | |
Dobutamine EF | -0.0381 | -0.0372 | -0.0384 | -0.0364 | -0.0333 |
(0.0131) | (0.0128) | (0.0130) | (0.0103) | (0.00941) | |
Modeling the risk of any cardiac event | |||||
Rest WMA | -0.779 | -0.758 | -0.755 | -0.839 | -0.762 |
(0.429) | (0.421) | (0.419) | (0.336) | (0.293) | |
Hist. of HT | 0.823 | 0.781 | 0.794 | 0.728 | 0.632 |
(0.393) | (0.384) | (0.384) | (0.302) | (0.276) | |
Intercept | 0.107 | 0.115 | 0.151 | 0.0102 | -0.0980 |
(0.833) | (0.818) | (0.829) | (0.655) | (0.611) |
Models for predicting the risk of MI (EPV ≈ 7) | ||||
Methods | Calibration Slope | AUC | Brier Score | APP |
MLE | 0.696(0.258) | 0.768(0.051) | 0.047 | 0.051 |
Firth | 0.706(0.260) | 0.766(0.052) | 0.049 | 0.057 |
logF(1,1) | 0.713 (0.265) | 0.769(0.051) | 0.048 | 0.052 |
logF(2,2) | 0.723 (0.271) | 0.769(0.051) | 0.048 | 0.052 |
RIDGE | 0.772(0.309) | 0.762(0.053) | 0.047 | 0.050 |
Models for predicting the risk of CABG (EPV ≈ 10) | ||||
MLE | 0.912(0.219) | 0.814(0.046) | 0.057 | 0.056 |
Firth | 0.909 (0.217) | 0.814 (0.046) | 0.056 | 0.059 |
logF(1,1) | 0.921(0.221) | 0.814(0.046) | 0.056 | 0.055 |
logF(2,2) | 0.926(0.223) | 0.813(0.046) | 0.057 | 0.055 |
RIDGE | 0.886(0.217) | 0.814(0.046) | 0.057 | 0.055 |
Models for predicting the risk of PTCA (EPV ≈ 5) | ||||
MLE | 0.718 (0.291) | 0.730(0.108) | 0.034 | 0.061 |
Firth | 0.721(0.279) | 0.729(0.108) | 0.035 | 0.066 |
logF(1,1) | 0.721(0.298) | 0.728(0.107) | 0.034 | 0.061 |
logF(2,2) | 0.720(0.305) | 0.728(0.107) | 0.034 | 0.061 |
RIDGE | 0.774(0.544) | 0.727(0.107) | 0.033 | 0.061 |
Models for predicting the risk of cardiac death (EPV ≈ 6) | ||||
MLE | 0.661(0.529) | 0.688(0.121) | 0.024 | 0.062 |
Firth | 0.680(0.545) | 0.688 (0.121) | 0.024 | 0.067 |
logF(1,1) | 0.645(0.535) | 0.687(0.120) | 0.024 | 0.062 |
logF(2,2) | 0.623 (0.538) | 0.687 (0.120) | 0.024 | 0.061 |
RIDGE | 0.665 (0.608) | 0.684 (0.121) | 0.023 | 0.062 |
Models for predicting the risk of any cardiac event (EPV ≈ 15) | ||||
MLE | 0.942(0.206) | 0.771(0.044) | 0.059 | 0.164 |
Firth | 0.946(0.207) | 0.767 (0.044) | 0.059 | 0.167 |
logF(1,1) | 0.945(0.206) | 0.770(0.044) | 0.058 | 0.164 |
logF(2,2) | 0.946(0.207) | 0.770 (0.044) | 0.058 | 0.164 |
RIDGE | 1.004(0.222) | 0.769(0.044) | 0.056 | 0.165 |