Background
Methods
Motivating example
Bayesian multi-Arm multi-stage design
The priors
Decision thresholds
Results
Illustrative case study
Prior | ESS | Posterior mean | Decision criteria | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Arm | A | B | C | A | B | C | A | B | C | B | C | B | C |
Rule 1 | Rule 1 | Rule 1 | Rule 2 | Rule 2 | Rule 3 | Rule 3 | |||||||
MLE | 0 | 0 | 0 | 0.3750 | 0.3250 | 0.4000 | |||||||
Non informative | 1 | 1 | 1 | 0.3780 | 0.3293 | 0.4024 | 0.1505 | 0.3576 | 0.0863 | 0.3198 | 0.5906 | 0.0286 | 0.1197 |
2 | 2 | 2 | 0.3810 | 0.3333 | 0.4048 | 0.1384 | 0.3346 | 0.0789 | 0.3223 | 0.5894 | 0.0281 | 0.1161 | |
Sk eptical | 10 | 1 | 1 | 0.3600 | 0.3244 | 0.3976 | 0.1900 | 0.3833 | 0.0971 | 0.3575 | 0.6437 | 0.0310 | 0.1340 |
10 | 5 | 1 | 0.3600 | 0.3222 | 0.3976 | 0.1900 | 0.3885 | 0.0971 | 0.3465 | 0.6437 | 0.0262 | 0.1340 | |
10 | 1 | 5 | 0.3600 | 0.3244 | 0.3889 | 0.1900 | 0.3833 | 0.1074 | 0.3575 | 0.6148 | 0.0310 | 0.1099 | |
Enthusiastic | 10 | 1 | 1 | 0.3600 | 0.3222 | 0.3889 | 0.1900 | 0.3885 | 0.1074 | 0.3465 | 0.6148 | 0.0262 | 0.1099 |
10 | 5 | 1 | 0.3600 | 0.3280 | 0.4012 | 0.1900 | 0.3640 | 0.0889 | 0.3716 | 0.6570 | 0.0338 | 0.1422 | |
10 | 1 | 5 | 0.3600 | 0.3389 | 0.4012 | 0.1900 | 0.2996 | 0.0889 | 0.4128 | 0.6570 | 0.0393 | 0.1422 |
Simulation study
Simulation settings
Simulation results
Threshold calibration
Sample size | True benefit | Posterior mean estimate biases | Mean square errors | Decision criterion 1 | Criterion 2 | Criterion 3 | |||
---|---|---|---|---|---|---|---|---|---|
dB | pA
| pB
| pA
| pB
| A | B | B | B | |
40
| 0.00 | 0.0086 | 0.0090 | 0.0048 | 0.0048 | 0.4850 | 0.4835 | 0.5012 | 0.1414 |
0.05 | 0.0082 | 0.0078 | 0.0048 | 0.0052 | 0.4854 | 0.2982 | 0.6358 | 0.2399 | |
0.10 | 0.0100 | 0.0042 | 0.0050 | 0.0054 | 0.4798 | 0.1632 | 0.7440 | 0.3518 | |
0.15 | 0.0097 | 0.0014 | 0.0049 | 0.0055 | 0.4798 | 0.0753 | 0.8352 | 0.4803 | |
0.20 | 0.0098 | −0.0001 | 0.0049 | 0.0057 | 0.4795 | 0.0297 | 0.9028 | 0.6132 | |
0.25 | 0.0096 | −0.0013 | 0.0049 | 0.0056 | 0.4800 | 0.0093 | 0.9477 | 0.7322 | |
0.30 | 0.0095 | −0.0049 | 0.0049 | 0.0055 | 0.4804 | 0.0027 | 0.9731 | 0.8271 | |
0.35 | 0.0088 | −0.0070 | 0.0048 | 0.0051 | 0.4841 | 0.0005 | 0.9886 | 0.8999 | |
0.40 | 0.0109 | −0.0087 | 0.0048 | 0.0047 | 0.4749 | 0.0001 | 0.9958 | 0.9477 | |
0.45 | 0.0092 | −0.0116 | 0.0049 | 0.0043 | 0.4808 | 0.0000 | 0.9984 | 0.9755 | |
100
| 0.00 | 0.0042 | 0.0032 | 0.0020 | 0.0020 | 0.4863 | 0.4925 | 0.4960 | 0.0460 |
0.05 | 0.0037 | 0.0030 | 0.0020 | 0.0022 | 0.4885 | 0.2156 | 0.7042 | 0.1363 | |
0.10 | 0.0040 | 0.0014 | 0.0020 | 0.0023 | 0.4866 | 0.0661 | 0.8503 | 0.2896 | |
0.15 | 0.0038 | 0.0012 | 0.0020 | 0.0024 | 0.4881 | 0.0128 | 0.9404 | 0.4925 | |
0.20 | 0.0040 | −0.0003 | 0.0020 | 0.0024 | 0.4858 | 0.0015 | 0.9807 | 0.6875 | |
0.25 | 0.0027 | −0.0009 | 0.0020 | 0.0023 | 0.4947 | 0.0001 | 0.9950 | 0.8473 | |
0.30 | 0.0046 | −0.0017 | 0.0020 | 0.0023 | 0.4836 | 0.0000 | 0.9989 | 0.9362 | |
0.35 | 0.0042 | −0.0032 | 0.0020 | 0.0022 | 0.4856 | 0.0000 | 0.9998 | 0.9792 | |
0.40 | 0.0032 | −0.0037 | 0.0020 | 0.0021 | 0.4922 | 0.0000 | 1.0000 | 0.9952 | |
0.45 | 0.0036 | −0.0047 | 0.0020 | 0.0018 | 0.4897 | 0.0000 | 1.0000 | 0.9992 |
Assessing false decision rates
Sample size | True benefit | Posterior mean estimate biases | Enrolled sample sizes | % Early stopping | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
dB | dC | pA
| pB
| pC
| nA
| nB
| nC
| A | % early B | % early C | |
40 | −0.20 | 0.00 | −0.0051 | 0.0305 | −0.0063 | 36.3504 | 15.0121 | 36.3180 | 15.32 % | 96.09 % | 15.15 % |
−0.15 | 0.00 | −0.0058 | 0.0061 | −0.0057 | 36.2548 | 21.3829 | 36.3305 | 15.43 % | 79.69 % | 15.19 % | |
−0.10 | 0.00 | −0.0063 | −0.0066 | −0.0062 | 36.3540 | 27.9657 | 36.3594 | 15.20 % | 53.98 % | 15.36 % | |
−0.05 | 0.00 | −0.0056 | −0.0076 | −0.0072 | 36.2719 | 33.0725 | 36.2428 | 15.44 % | 30.66 % | 15.86 % | |
0.00
|
0.00
|
−0.0050
|
−0.0057
|
−0.0063
|
36.2726
|
36.4040
|
36.4653
|
15.77 %
|
15.11 %
|
14.85 %
| |
0.05 | 0.00 | −0.0073 | −0.0034 | −0.0049 | 36.2420 | 38.1107 | 36.3738 | 15.75 % | 7.33 % | 15.23 % | |
0.10 | 0.00 | −0.0077 | −0.0011 | −0.0049 | 36.2214 | 39.1845 | 36.3761 | 15.91 % | 2.92 % | 14.88 % | |
0.15 | 0.00 | −0.0044 | −0.0008 | −0.0070 | 36.4279 | 39.6107 | 36.2580 | 14.86 % | 1.35 % | 15.47 % | |
0.20 | 0.00 | −0.0044 | −0.0016 | −0.0060 | 36.3945 | 39.8325 | 36.3640 | 15.06 % | 0.55 % | 15.29 % | |
100 | −0.20 | 0.00 | −0.0206 | 0.0305 | −0.0205 | 84.2873 | 15.4339 | 84.3159 | 22.80 % | 99.99 % | 22.82 % |
−0.15 | 0.00 | −0.0188 | −0.0007 | −0.0203 | 84.7484 | 24.5031 | 84.4050 | 21.95 % | 99.27 % | 22.22 % | |
−0.10 | 0.00 | −0.0202 | −0.0212 | −0.0197 | 84.4487 | 43.6230 | 84.5338 | 22.31 % | 85.84 % | 22.44 % | |
−0.05 | 0.00 | −0.0191 | −0.0276 | −0.0199 | 84.8712 | 66.7741 | 84.3454 | 21.99 % | 52.46 % | 22.57 % | |
0.00
|
0.00
|
−0.0216
|
−0.0207
|
−0.0216
|
84.0536
|
84.2135
|
83.8033
|
22.86 %
|
22.67 %
|
23.36 %
| |
0.05 | 0.00 | −0.0203 | −0.0115 | −0.0196 | 84.6003 | 93.1786 | 84.7271 | 22.28 % | 8.66 % | 22.24 % | |
0.10 | 0.00 | −0.0194 | −0.0050 | −0.0202 | 84.6521 | 97.3946 | 84.1845 | 21.94 % | 3.08 % | 22.76 % | |
0.15 | 0.00 | −0.0194 | −0.0023 | −0.0202 | 84.8085 | 98.9283 | 84.6060 | 22.00 % | 1.20 % | 22.06 % | |
0.20 | 0.00 | −0.0199 | −0.0024 | −0.0215 | 84.5757 | 99.4983 | 84.1044 | 22.02 % | 0.56 % | 22.94 % |
Sample size | True benefit | Posterior mean estimate biases | Average sample sizes | % Early stopping | ||||||
---|---|---|---|---|---|---|---|---|---|---|
n | dB | dC | pA
| pB
| pC
| A | B | C | B | C |
40 | −0.20 | 0.00 | 0.0446 | 0.0410 | −0.0036 | 34.7688 | 19.0005 | 34.1757 | 83.18 % | 20.90 % |
−0.15 | 0.00 | 0.0407 | 0.0196 | −0.0049 | 35.4230 | 23.9111 | 34.1465 | 65.96 % | 21.12 % | |
−0.10 | 0.00 | 0.0384 | 0.0076 | −0.0045 | 36.0178 | 28.2528 | 33.9857 | 47.31 % | 21.51 % | |
−0.05 | 0.00 | 0.0325 | 0.0004 | −0.0039 | 36.8988 | 31.7086 | 34.2757 | 31.84 % | 20.67 % | |
0.00
|
0.00
|
0.0289
|
−0.0046
|
−0.0034
|
37.5750
|
34.3312
|
34.2628
|
20.68 %
|
20.55 %
| |
0.05 | 0.00 | 0.0262 | −0.0053 | −0.0046 | 38.1404 | 35.9159 | 34.2368 | 13.61 % | 21.01 % | |
0.10 | 0.00 | 0.0212 | −0.0068 | −0.0038 | 38.6097 | 37.1804 | 34.2661 | 8.79 % | 20.81 % | |
0.15 | 0.00 | 0.0184 | −0.0079 | −0.0043 | 39.0075 | 38.0515 | 34.3098 | 5.79 | 20.69 % | |
0.20 | 0.00 | 0.0163 | −0.0098 | −0.0051 | 39.2281 | 38.5864 | 34.2477 | 4.01 % | 20.85 % | |
100 | −0.20 | 0.00 | 0.0458 | 0.0362 | −0.0144 | 80.7525 | 22.0750 | 80.0766 | 99.05 % | 26.36 % |
−0.15 | 0.00 | 0.0427 | 0.0103 | −0.0153 | 81.9307 | 34.6011 | 79.9595 | 91.50 % | 26.58 % | |
−0.10 | 0.00 | 0.0418 | −0.0063 | −0.0152 | 83.5055 | 51.2404 | 79.0897 | 71.48 % | 27.30 % | |
−0.05 | 0.00 | 0.0344 | −0.0142 | −0.0153 | 87.6493 | 67.8997 | 80.1142 | 46.17 % | 26.49 % | |
0.00
|
0.00
|
0.0277
|
−0.0161
|
−0.0157
|
90.8189
|
79.7603
|
79.7672
|
26.62 %
|
26.82 %
| |
0.05 | 0.00 | 0.0247 | −0.0149 | −0.0154 | 93.1423 | 86.5031 | 79.4970 | 15.96 % | 26.65 % | |
0.10 | 0.00 | 0.0202 | −0.0130 | −0.0157 | 95.3857 | 91.6455 | 78.7749 | 9.26 % | 27.54 % | |
0.15 | 0.00 | 0.0154 | −0.0119 | −0.0155 | 96.8064 | 94.2754 | 79.4074 | 6.15 % | 27.04 % | |
0.20 | 0.00 | 0.0112 | −0.0100 | −0.0147 | 97.8693 | 96.1853 | 80.1669 | 4.03 % | 26.14 % |
Sample size | True benefit | Posterior mean estimate biases | Average sample sizes | % Early stopping | ||||||
---|---|---|---|---|---|---|---|---|---|---|
dB | dC | pA
| pB
| pC
| A | B | C | B | C | |
40 | −0.15 | 0.00 | 0.0092 | 0.0182 | 0.0254 | 39.9621 | 39.8136 | 37.9895 | 0.54 % | 6.45 % |
−0.05 | 0.00 | 0.0088 | 0.0237 | 0.0272 | 39.8054 | 38.8511 | 37.8795 | 3.56 % | 6.90 % | |
0.00
|
0.00
|
0.0085
|
0.0254
|
0.0270
|
39.6095
|
37.9435
|
37.9783
|
6.69 %
|
6.53 %
| |
0.05 | 0.00 | 0.0089 | 0.0324 | 0.0250 | 39.5103 | 36.5882 | 38.0288 | 11.66 % | 6.37 % | |
0.10 | 0.00 | 0.0062 | 0.0384 | 0.0264 | 39.2740 | 34.3440 | 38.0311 | 19.85 % | 6.37 % | |
0.15
|
0.00
|
0.0056
|
0.0416
|
0.0254
|
39.0452
|
31.9203
|
38.0610
|
29.96 %
|
6.22 %
| |
0.20 | 0.00 | 0.0049 | 0.0451 | 0.0256 | 38.8153 | 28.6999 | 37.9511 | 42.93 % | 6.61 % | |
0.25 | 0.00 | 0.0051 | 0.0423 | 0.0254 | 38.5798 | 25.0534 | 37.9815 | 57.41 % | 6.60 % | |
0.30 | 0.00 | 0.0052 | 0.0369 | 0.0250 | 38.4489 | 21.2180 | 38.0583 | 71.85 % | 6.33 % | |
0.35 | 0.00 | 0.0037 | 0.0261 | 0.0270 | 38.2338 | 17.3342 | 37.9999 | 83.87 % | 6.49 % | |
0.40 | 0.00 | 0.0039 | 0.0098 | 0.0265 | 38.0706 | 14.1594 | 37.8966 | 92.38 % | 6.79 % | |
0.45 | 0.00 | 0.0035 | −0.0115 | 0.0266 | 38.0738 | 11.1299 | 37.9628 | 97.64 % | 6.54 % | |
100 | −0.15 | 0.00 | 0.0029 | 0.0093 | 0.0226 | 99.8389 | 99.4843 | 94.1617 | 0.55 % | 6.49 % |
−0.05 | 0.00 | 0.0031 | 0.0169 | 0.0237 | 99.3256 | 96.7377 | 93.7367 | 3.55 % | 6.94 % | |
0.00
|
0.00
|
0.0020
|
0.0233
|
0.0222
|
99.0139
|
93.7839
|
94.1902
|
6.91 %
|
6.48 %
| |
0.05 | 0.00 | 0.0013 | 0.0322 | 0.0224 | 98.3338 | 89.3276 | 94.1794 | 12.36 % | 6.48 % | |
0.10 | 0.00 | 0.0012 | 0.0405 | 0.0227 | 97.6239 | 82.2664 | 94.1533 | 21.90 % | 6.50 % | |
0.15
|
0.00
|
−0.0009
|
0.0509
|
0.0234
|
96.5788
|
71.2643
|
94.0376
|
37.94 %
|
6.60 %
| |
0.20 | 0.00 | −0.0025 | 0.0578 | 0.0235 | 95.7497 | 56.8519 | 94.0065 | 59.48 % | 6.66 % | |
0.25 | 0.00 | −0.0025 | 0.0577 | 0.0230 | 95.1262 | 41.9689 | 93.9749 | 79.99 % | 6.71 % | |
0.30 | 0.00 | −0.0030 | 0.0473 | 0.0235 | 94.5634 | 29.7273 | 93.9678 | 92.99 % | 6.71 % | |
0.40 | 0.00 | −0.0025 | 0.0119 | 0.0239 | 94.0066 | 15.1126 | 93.8206 | 99.84 % | 6.79 % | |
0.45 | 0.00 | −0.0027 | −0.0110 | 0.0227 | 94.2995 | 11.7989 | 94.1851 | 100.00 % | 6.48 % |
Sample size | True benefit | Posterior mean estimate biases | Enrolled sample sizes | % Early stopping | ||||||
---|---|---|---|---|---|---|---|---|---|---|
dB | dC | pA
| pB
| pC
| A | B | C | B | C | |
40 | −0.20 | 0.00 | 0.0625 | 0.0585 | −0.0123 | 30.6959 | 13.8404 | 29.9078 | 92.23 % | 34.59 % |
−0.15 | 0.00 | 0.0601 | 0.0311 | −0.0120 | 31.2312 | 18.3044 | 29.5173 | 79.84 % | 35.56 % | |
−0.10 | 0.00 | 0.0567 | 0.0107 | −0.0119 | 32.3427 | 22.4957 | 29.6238 | 63.89 % | 35.43 % | |
−0.05 | 0.00 | 0.0506 | −0.0027 | −0.0126 | 33.5772 | 26.5306 | 29.7194 | 48.34 % | 35.68 % | |
0.00
|
0.00
|
0.0480
|
−0.0122
|
−0.0123
|
34.5352
|
29.3818
|
29.5852
|
35.89 %
|
35.74 %
| |
0.05 | 0.00 | 0.0386 | −0.0166 | −0.0109 | 35.7946 | 32.2020 | 29.7191 | 25.14 % | 35.03 % | |
0.10 | 0.00 | 0.0337 | −0.0178 | −0.0107 | 36.7805 | 34.2756 | 29.8086 | 17.75 % | 34.52 % | |
0.15 | 0.00 | 0.0300 | −0.0176 | −0.0122 | 37.6091 | 35.9146 | 29.5542 | 12.22 % | 35.64 % | |
0.20 | 0.00 | 0.0268 | −0.0180 | −0.0111 | 38.1120 | 36.8978 | 29.8112 | 8.92 % | 34.91 % | |
100 | −0.20 | 0.00 | 0.0704 | 0.0581 | −0.0229 | 66.7855 | 15.4632 | 65.8818 | 99.60 % | 42.83 % |
−0.15 | 0.00 | 0.0673 | 0.0252 | −0.0230 | 68.2876 | 23.4374 | 66.0635 | 96.28 % | 42.43 % | |
−0.10 | 0.00 | 0.0601 | 0.0007 | −0.0229 | 71.7312 | 36.8674 | 66.5112 | 84.36 % | 42.43 % | |
−0.05 | 0.00 | 0.0571 | −0.0160 | −0.0231 | 75.3970 | 51.5977 | 65.8379 | 64.18 % | 43.02 % | |
0.00
|
0.00
|
0.0467
|
−0.0237
|
−0.0240
|
81.2572
|
66.4872
|
65.8151
|
42.25 %
|
42.95 %
| |
0.05 | 0.00 | 0.0379 | −0.0255 | −0.0233 | 86.5612 | 76.7592 | 66.9099 | 27.22 % | 41.95 % | |
0.10 | 0.00 | 0.0338 | −0.0241 | −0.0242 | 89.8261 | 83.4433 | 64.7309 | 18.46 % | 43.92 % | |
0.15 | 0.00 | 0.0279 | −0.0229 | −0.0232 | 92.3335 | 88.1690 | 66.3140 | 12.66 % | 42.48 % | |
0.20 | 0.00 | 0.0234 | −0.0208 | −0.0245 | 94.1323 | 91.5214 | 65.5106 | 8.98 % | 43.34 % |
Sample size | True benefit | Posterior mean estimate biases | Enrolled sample sizes | % Early stopping | ||||||
---|---|---|---|---|---|---|---|---|---|---|
dB | dC | pA | pB | pC | A | B | C | B | C | |
40 | −0.15 | 0.00 | 0.0083 | 0.0320 | 0.0562 | 39.5748 | 38.6963 | 33.8148 | 3.64 % | 18.64 % |
−0.05 | 0.00 | 0.0063 | 0.0495 | 0.0568 | 38.7270 | 35.8578 | 33.7310 | 12.09 % | 18.95 % | |
0.00
|
0.00
|
0.0049
|
0.0555
|
0.0581
|
38.1052
|
33.9250
|
33.5867
|
18.38 %
|
19.20 %
| |
0.05 | 0.00 | 0.0016 | 0.0632 | 0.0563 | 37.5141 | 31.3987 | 33.8662 | 26.92 % | 18.51 % | |
0.10 | 0.00 | −0.0008 | 0.0691 | 0.0570 | 36.8763 | 28.0621 | 33.8537 | 38.65 % | 18.55 % | |
0.15
|
0.00
|
0.0001
|
0.0720
|
0.0578
|
35.9576
|
24.3662
|
33.6137
|
51.87 %
|
19.22 %
| |
0.20 | 0.00 | −0.0002 | 0.0673 | 0.0561 | 35.5246 | 20.7723 | 33.8744 | 64.82 % | 18.40 % | |
0.25 | 0.00 | −0.0017 | 0.0575 | 0.0568 | 34.9601 | 16.9213 | 33.7372 | 77.56 % | 18.91 % | |
0.30 | 0.00 | −0.0043 | 0.0409 | 0.0575 | 34.3862 | 13.5320 | 33.6114 | 87.33 % | 19.19 % | |
0.35 | 0.00 | −0.0016 | 0.0196 | 0.0560 | 34.5481 | 10.7458 | 34.0309 | 94.31 % | 17.94 % | |
0.40 | 0.00 | −0.0033 | −0.0069 | 0.0547 | 34.2171 | 8.5161 | 33.9205 | 97.80 % | 18.43 % | |
0.45 | 0.00 | −0.0023 | −0.0372 | 0.0542 | 34.1941 | 6.8534 | 33.9756 | 99.23 % | 18.33 % | |
100 | −0.05 | 0.00 | 0.0032 | 0.0229 | 0.0559 | 98.8583 | 96.4597 | 82.6701 | 3.69 % | 18.83 % |
−0.05 | 0.00 | −0.0006 | 0.0433 | 0.0563 | 96.1819 | 89.1834 | 82.5699 | 11.51 % | 18.92 % | |
0.00
|
0.00
|
−0.0038
|
0.0577
|
0.0571
|
94.3563
|
82.2688
|
82.2694
|
19.21 %
|
19.14 %
| |
0.05 | 0.00 | −0.0060 | 0.0689 | 0.0570 | 92.3215 | 73.7509 | 82.5095 | 29.51 % | 19.02 % | |
0.10 | 0.00 | −0.0075 | 0.0778 | 0.0567 | 89.5994 | 63.0008 | 82.5545 | 43.27 % | 18.97 % | |
0.15
|
0.00
|
−0.0088
|
0.0822
|
0.0574
|
87.6736
|
49.5431
|
82.5278
|
61.69 %
|
18.88 %
| |
0.20 | 0.00 | −0.0102 | 0.0771 | 0.0530 | 86.2653 | 37.2856 | 83.2202 | 77.99 % | 18.15 % | |
0.25 | 0.00 | −0.0113 | 0.0636 | 0.0558 | 84.2998 | 25.2895 | 82.5265 | 91.74 % | 19.02 % | |
0.30 | 0.00 | −0.0112 | 0.0448 | 0.0576 | 83.4348 | 17.4124 | 82.3130 | 97.39 % | 19.14 % | |
0.35 | 0.00 | −0.0114 | 0.0204 | 0.0558 | 83.0775 | 11.9089 | 82.5449 | 99.58 % | 18.89 % | |
0.40 | 0.00 | −0.0102 | −0.0073 | 0.0552 | 83.3131 | 9.0623 | 82.9624 | 99.95 % | 18.47 % | |
0.45 | 0.00 | −0.0111 | −0.0375 | 0.0585 | 82.5830 | 6.9659 | 82.3697 | 100.00 % | 19.09 % |