Background
Methods
Risk adjustment methods
Fixed and random effects logistic regression
Generalized propensity score methods
Simulation study
Data source
Data generation
Scenarios
-
The number of providers, K: 10, 20, 30, 40, or 50.Table 1Parameter Settings of Scenarios Studied Through SimulationsScenarioKp ··P(out)HSmin(n k )P(nmin)110-500.100.22210000.52500.03-0.200.22210000.53500.100.08-0.42210000.54500.100.21-4210000.55500.100.221-210000.56500.100.222500-10000.57500.100.2225000.5-1
-
The average incidence of mortality, p··: 3, 10 or 20%.
-
The percentage of true outliers, P(out): 8, 20 or 40%. This ensured an equal number of true outliers selected from both outlier distribution for each K studied.
-
The amount of standard deviations the outlier random intercept distribution was shifted, H: 1, 2, 3, or 4.
-
Outliers were either drawn from both outlier distributions (S=2) or only from the below-average performance distribution (S=1).
-
Half of the providers were allocated either min(n k )=500 or min(n k )=1000 patients, while the other half were always of size max(n k )=1000.
Statistical analysis
Classification performance
Results
Discussion
Measure |
K
|
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 10 | 0.00, 1.00 | 0.00, 0.50 | 0.00, 0.50 | 0.00, 0.50 | 0.00, 0.50 | 0.00, 0.50 |
20 | 0.25, 1.00 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | |
30 | 0.33, 1.00 | 0.00, 0.33 | 0.00, 0.33 | 0.00, 0.17 | 0.00, 0.33 | 0.00, 0.33 | |
40 | 0.50, 1.00 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | 0.00, 0.25 | |
50 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
Specificity | 10 | 0.50, 1.00 | 1.00, 1.00 | 1.00, 1.00 | 0.88, 1.00 | 0.88, 1.00 | 0.88, 1.00 |
20 | 0.56, 0.88 | 0.94, 1.00 | 0.94, 1.00 | 0.88, 1.00 | 0.94, 1.00 | 0.88, 1.00 | |
30 | 0.54, 0.88 | 0.96, 1.00 | 0.96, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.92, 1.00 | |
40 | 0.56, 0.84 | 0.97, 1.00 | 0.97, 1.00 | 0.91, 1.00 | 0.94, 1.00 | 0.91, 1.00 | |
50 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
PPV | 10 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 |
20 | 0.20, 0.67 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
30 | 0.25, 0.58 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
40 | 0.26, 0.55 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
50 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 10 | 0.75, 1.00 | 0.80, 0.89 | 0.80, 0.89 | 0.78, 0.89 | 0.78, 0.89 | 0.78, 0.89 |
20 | 0.80, 1.00 | 0.80, 0.84 | 0.80, 0.84 | 0.78, 0.84 | 0.79, 0.84 | 0.78, 0.84 | |
30 | 0.82, 1.00 | 0.79, 0.86 | 0.79, 0.86 | 0.79, 0.83 | 0.79, 0.86 | 0.79, 0.85 | |
40 | 0.84, 1.00 | 0.79, 0.84 | 0.79, 0.84 | 0.78, 0.84 | 0.79, 0.84 | 0.78, 0.84 | |
50 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |
Measure |
p
..
|
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 0.03 | 0.20, 0.70 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.10 |
0.10 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
0.20 | 0.68, 1.00 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.30 | |
Specificity | 0.03 | 0.75, 0.95 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.95, 1.00 |
0.10 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
0.20 | 0.45, 0.72 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
PPV | 0.03 | 0.18, 0.67 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 |
0.10 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
0.20 | 0.26, 0.43 | 0.50, 1.00 | 0.50, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 0.03 | 0.80, 0.92 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.82 |
0.10 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 | |
0.20 | 0.86, 1.00 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.85 |
Measure | P(out) |
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 0.03 | 0.20, 0.70 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.10 |
0.10 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
0.20 | 0.68, 1.00 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.30 | |
Specificity | 0.03 | 0.75, 0.95 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.95, 1.00 |
0.10 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
0.20 | 0.45, 0.72 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
PPV | 0.03 | 0.18, 0.67 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 |
0.10 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
0.20 | 0.26, 0.43 | 0.50, 1.00 | 0.50, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 0.03 | 0.80, 0.92 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.82 |
0.10 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 | |
0.20 | 0.86, 1.00 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.85 |
Conclusions
Appendix
Confidence interval tables
Measure |
H
|
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 1 | 0.20, 0.70 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.10 | 0.00, 0.20 |
2 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
3 | 0.80, 1.00 | 0.10, 0.40 | 0.10, 0.40 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.30 | |
4 | 0.90, 1.00 | 0.20, 0.50 | 0.20, 0.50 | 0.00, 0.30 | 0.00, 0.40 | 0.00, 0.40 | |
Specificity | 1 | 0.57, 0.82 | 0.95, 1.00 | 0.95, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.90, 0.97 |
2 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
3 | 0.57, 0.82 | 0.97, 1.00 | 0.97, 1.00 | 0.95, 1.00 | 0.95, 1.00 | 0.92, 1.00 | |
4 | 0.55, 0.80 | 1.00, 1.00 | 1.00, 1.00 | 0.95, 1.00 | 0.97, 1.00 | 0.95, 1.00 | |
PPV | 1 | 0.11, 0.41 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 0.50 |
2 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
3 | 0.33, 0.56 | 0.75, 1.00 | 0.75, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
4 | 0.36, 0.56 | 1.00, 1.00 | 1.00, 1.00 | 0.00, 1.00 | 0.33, 1.00 | 0.00, 1.00 | |
NPV | 1 | 0.76, 0.90 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.82 | 0.78, 0.83 |
2 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 | |
3 | 0.92, 1.00 | 0.81, 0.87 | 0.82, 0.87 | 0.79, 0.85 | 0.80, 0.85 | 0.79, 0.85 | |
4 | 0.96, 1.00 | 0.83, 0.89 | 0.83, 0.89 | 0.80, 0.85 | 0.80, 0.87 | 0.79, 0.87 |
Measure |
S
|
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 1 | 0.40, 0.80 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.10 | 0.00, 0.20 | 0.00, 0.20 |
2 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
Specificity | 1 | 0.55, 0.78 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.90, 0.97 |
2 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
PPV | 1 | 0.20, 0.41 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 0.67 | 0.00, 1.00 | 0.00, 0.67 |
2 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 1 | 0.80, 0.93 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.82 | 0.79, 0.83 | 0.78, 0.83 |
2 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |
Measure | min(n
k
) |
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 500 | 0.40, 0.90 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 |
1000 | 0.50, 0.90 | 0.00, 0.30 | 0.00, 0.30 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
Specificity | 500 | 0.62, 0.85 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.90, 1.00 |
1000 | 0.57, 0.80 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.95, 1.00 | 0.90, 1.00 | |
PPV | 500 | 0.26, 0.56 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 |
1000 | 0.27, 0.53 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 500 | 0.83, 0.97 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |
1000 | 0.84, 0.97 | 0.80, 0.85 | 0.80, 0.85 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |
Measure | P(nmin) |
L
R
F
|
L
R
R
|
g
P
S
A
|
g
P
S
W
|
g
P
S
WT
|
g
P
S
MWS
|
---|---|---|---|---|---|---|---|
Sensitivity | 0.5 | 0.40, 0.90 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 |
1 | 0.30, 0.80 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | 0.00, 0.20 | |
Specificity | 0.5 | 0.62, 0.85 | 0.97, 1.00 | 0.97, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.90, 1.00 |
1 | 0.62, 0.85 | 0.98, 1.00 | 0.98, 1.00 | 0.92, 1.00 | 0.92, 1.00 | 0.92, 1.00 | |
PPV | 0.5 | 0.26, 0.56 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 |
1 | 0.21, 0.50 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | 0.00, 1.00 | |
NPV | 0.5 | 0.83, 0.97 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |
1 | 0.81, 0.94 | 0.80, 0.83 | 0.80, 0.83 | 0.79, 0.83 | 0.79, 0.83 | 0.79, 0.83 |