Background
Methods
Monte carlo simulation study
Data-generating process
-
Sample size, ranging from 1,000 down to 40 (1000, 900, 800, 700, 600, 500, 400, 300, 200, 180, 160, 140, 120, 100, 80, 60, 40)
-
Treatment effect , successively fixed at 0 (null hypothesis), 0.41 and 0.92 (alternative hypotheses of moderate or strong treatment effect) corresponding to conditional ORs fixed at 1, 1.5 and 2.5, respectively
-
Strength of the association between the covariates and both the treatment and the outcome, with a and b fixed at 0.41 and 0.92, corresponding to moderate and strong association, respectively.
Analysis of simulated data sets
Propensity score models
Propensity score based matching
Inverse-probability-of-treatment weighting
Treatment effect estimates
gee
for R, Vincent J Carey, Thomas Lumley and Brian Ripley). Then, a weighted logistic model using a generalized linear model adapted to data from a complex survey design, with inverse-probability weighting and design-based standard errors applied (package svyGLM
for R, Thomas Lumley).Model performance criteria
Results
Simulation results
Full fitted models
N | OR(a) = 1.5, OR(b) = 1.5 | OR(a) = 2.5, OR(b) =1.5 | OR(a) = 1.5, OR(b) = 2.5 | OR(a) = 2.5, OR(b) = 2.5 | |||||
---|---|---|---|---|---|---|---|---|---|
PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | ||
40
|
Type I error
| 0.055 | 0.048 | 0.053 | 0.056 | 0.052 | 0.039 | 0.052 | 0.047 |
Bias
| 0.05 | 0.057 | 0.049 | 0.07 | 0.059 | 0.055 | 0.048 | 0.079 | |
%
| 5.6 | 6.5 | 5.4 | 7.8 | 7.1 | 6.6 | 5.7 | 9.4 | |
Variance
| 0.875 | 0.593 | 0.918 | 0.694 | 0.873 | 0.576 | 0.903 | 0.683 | |
MSE
| 0.878 | 0.597 | 0.92 | 0.699 | 0.876 | 0.579 | 0.905 | 0.690 | |
60
|
Type I error
| 0.045 | 0.046 | 0.047 | 0.046 | 0.044 | 0.039 | 0.044 | 0.039 |
Bias
| 0.058 | 0.036 | 0.058 | 0.049 | 0.048 | 0.022 | 0.05 | 0.033 | |
%
| 6.4 | 4 | 6.4 | 5.4 | 5.7 | 2.6 | 6 | 3.9 | |
Variance
| 0.511 | 0.36 | 0.553 | 0.414 | 0.484 | 0.329 | 0.516 | 0.388 | |
MSE
| 0.514 | 0.362 | 0.556 | 0.416 | 0.486 | 0.33 | 0.519 | 0.389 | |
100
|
Type I error
| 0.049 | 0.046 | 0.047 | 0.046 | 0.047 | 0.034 | 0.04 | 0.036 |
Bias
| 0.023 | 0.018 | 0.02 | 0.022 | 0.021 | 0.015 | 0.025 | 0.016 | |
%
| 2.6 | 2 | 2.2 | 2.5 | 2.5 | 1.8 | 2.9 | 1.9 | |
Variance
| 0.254 | 0.194 | 0.282 | 0.223 | 0.242 | 0.177 | 0.261 | 0.204 | |
MSE
| 0.255 | 0.194 | 0.283 | 0.223 | 0.243 | 0.177 | 0.262 | 0.204 | |
500
|
Type I error
| 0.049 | 0.046 | 0.05 | 0.048 | 0.041 | 0.038 | 0.042 | 0.037 |
Bias
| 0.01 | 0.007 | 0.012 | 0.008 | 0.01 | 0.005 | 0.013 | 0.005 | |
%
| 1.1 | 0.8 | 1.3 | 0.9 | 1.2 | 0.6 | 1.5 | 0.6 | |
Variance
| 0.04 | 0.034 | 0.045 | 0.038 | 0.037 | 0.032 | 0.042 | 0.035 | |
MSE
| 0.04 | 0.034 | 0.045 | 0.038 | 0.037 | 0.032 | 0.042 | 0.035 |
Selected fitted models
Model 1 | Model 2 | Model 3 | Model 4 | Model 5 | Model 6 | Model 7 | Model 8 | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
N | X2 | X3 | X4 | X2,X3 | X2,X4 | X3,X4 | X2,X3,X4 | X1,X2,X3,X4 | |||||||||
PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | PSm | IPTW | ||
40
|
Type I error
| 0.062 | 0.041 | 0.059 | 0.051 | 0.058 | 0.040 | 0.060 | 0.047 | 0.053 | 0.036 | 0.048 | 0.045 | 0.053 | 0.044 | 0.052 | 0.047 |
Bias
| 0.273 | 0.235 | 0.265 | 0.251 | 0.086 | 0.06 | 0.259 | 0.25 | 0.075 | 0.061 | 0.064 | 0.071 | 0.075 | 0.074 | 0.048 | 0.079 | |
Relative bias
| 32.6 | 28.1 | 31.7 | 30 | 10.3 | 7.2 | 31 | 29.9 | 9 | 7.3 | 7.6 | 8.5 | 9 | 8.8 | 5.7 | 9.4 | |
Variance
| 0.809 | 0.508 | 0.872 | 0.566 | 0.862 | 0.551 | 0.874 | 0.573 | 0.843 | 0.56 | 0.907 | 0.634 | 0.904 | 0.644 | 0.903 | 0.683 | |
MSE
| 0.884 | 0.563 | 0.942 | 0.629 | 0.869 | 0.555 | 0.941 | 0.635 | 0.848 | 0.563 | 0.911 | 0.639 | 0.909 | 0.649 | 0.905 | 0.69 | |
60
|
Type I error
| 0.055 | 0.050 | 0.058 | 0.057 | 0.045 | 0.038 | 0.052 | 0.053 | 0.043 | 0.037 | 0.049 | 0.042 | 0.044 | 0.041 | 0.044 | 0.039 |
Bias
| 0.239 | 0.203 | 0.235 | 0.217 | 0.041 | 0.02 | 0.245 | 0.219 | 0.047 | 0.022 | 0.051 | 0.028 | 0.059 | 0.03 | 0.05 | 0.033 | |
Relative bias
| 28.6 | 24.3 | 28.1 | 25.9 | 4.9 | 2.4 | 29.3 | 26.2 | 5.6 | 2.6 | 6.1 | 3.3 | 7.1 | 3.6 | 6 | 3.9 | |
Variance
| 0.466 | 0.317 | 0.52 | 0.354 | 0.494 | 0.339 | 0.532 | 0.348 | 0.477 | 0.333 | 0.524 | 0.379 | 0.514 | 0.375 | 0.516 | 0.388 | |
MSE
| 0.523 | 0.359 | 0.575 | 0.401 | 0.495 | 0.339 | 0.592 | 0.396 | 0.479 | 0.334 | 0.527 | 0.38 | 0.517 | 0.376 | 0.519 | 0.389 | |
100
|
Type I error
| 0.062 | 0.057 | 0.065 | 0.066 | 0.052 | 0.039 | 0.057 | 0.062 | 0.042 | 0.034 | 0.046 | 0.041 | 0.044 | 0.037 | 0.04 | 0.036 |
Bias
| 0.209 | 0.193 | 0.212 | 0.203 | 0.029 | 0.013 | 0.215 | 0.203 | 0.029 | 0.013 | 0.028 | 0.016 | 0.028 | 0.016 | 0.025 | 0.016 | |
Relative bias
| 25 | 23.1 | 25.3 | 24.3 | 3.5 | 1.6 | 25.7 | 24.3 | 3.5 | 1.6 | 3.3 | 1.9 | 3.3 | 1.9 | 2.9 | 1.9 | |
Variance
| 0.239 | 0.175 | 0.254 | 0.194 | 0.242 | 0.186 | 0.256 | 0.188 | 0.239 | 0.18 | 0.259 | 0.205 | 0.253 | 0.199 | 0.261 | 0.204 | |
MSE
| 0.282 | 0.212 | 0.299 | 0.235 | 0.243 | 0.186 | 0.302 | 0.23 | 0.24 | 0.18 | 0.259 | 0.205 | 0.254 | 0.2 | 0.262 | 0.204 | |
500
|
Type I error
| 0.148 | 0.155 | 0.150 | 0.162 | 0.051 | 0.045 | 0.146 | 0.157 | 0.046 | 0.040 | 0.046 | 0.042 | 0.045 | 0.037 | 0.042 | 0.037 |
Bias
| 0.187 | 0.182 | 0.192 | 0.191 | 0.014 | 0.005 | 0.193 | 0.191 | 0.015 | 0.005 | 0.014 | 0.005 | 0.014 | 0.005 | 0.013 | 0.005 | |
Relative bias
| 22.3 | 21.8 | 22.9 | 22.8 | 1.7 | 0.6 | 23.1 | 22.8 | 1.8 | 0.6 | 1.7 | 0.6 | 1.7 | 0.6 | 1.5 | 0.6 | |
Variance
| 0.037 | 0.032 | 0.042 | 0.036 | 0.04 | 0.034 | 0.041 | 0.034 | 0.039 | 0.033 | 0.044 | 0.037 | 0.042 | 0.035 | 0.042 | 0.035 | |
MSE
| 0.072 | 0.066 | 0.079 | 0.072 | 0.04 | 0.034 | 0.078 | 0.071 | 0.039 | 0.033 | 0.044 | 0.037 | 0.042 | 0.035 | 0.042 | 0.035 |
Illustration to a real observational dataset
Model | Adjustment in the original set | PS-matched sample | IPT-Weighted sample | ||
---|---|---|---|---|---|
Covariates | OR (95%CI) p-value | No pairs | OR (95%CI) p-value | Sum of weights | OR (95%CI) p-value |
X1 = age
| 0.44 (0.14;1.42) p = 0.17 | 22 | 0.27 (0.06;1.23) p = 0.091 | 327.5 | 0.39 (0.11;1.32) p = 0.13 |
X2 = beta2micro
| 0.47 (0.15;1.47) p = 0.19 | 23 | 0.27 (0.05;1.41) p = 0.12 | 330.0 | 0.48 (0.14;1.65) p = 0.25 |
X3 = time to relapse
| 0.24 (0.07;0.84) p = 0.026 | 23 | 0.19 (0.05;0.65) p = 0.0088 | 349.1 | 0.27 (0.08;0.92) p = 0.039 |
X1 + X2
| 0.49 (0.15;1.57) p = 0.23 | 22 | 0.48 (0.09;2.58) p = 0.39 | 317.9 | 0.41 (0.12;1.36) p = 0.15 |
X1 + X3
| 0.22 (0.06;0.85) p = 0.028 | 18 | 0.20 (0.03;1.17) p = 0.073 | 456.0 | 0.23 (0.05;1.00) p = 0.052 |
X2 + X3
| 0.26 (0.07;0.93) p = 0.039 | 23 | 0.19 (0.05;0.68) p = 0.011 | 340.8 | 0.26 (0.08;0.86) p = 0.028 |
X1 + X2 + X3
| 0.24 (0.06;0.93) p = 0.040 | 20 | 0.41 (0.08;2.1) p = 0.28 | 432.1 | 0.21 (0.05;0.86) p = 0.031 |