The minimum sample size needed for a discrete-choice experiment (DCE) depends on the specific hypotheses to be tested. |
DCE practitioners should realize that a small size effect may still be meaningful, but that a limited sample size prevents detection of such small effects. |
Policy makers should not make a decision on non-significant outcomes without considering whether the study had a reasonable power to detect the anticipated outcome. |
1 Introduction
2 Literature Review
2.1 Methods
2.2 Literature Review Results
Item |
N (%) |
---|---|
Country of origina
| |
UK | 16 (23) |
USA | 13 (19) |
Canada | 10 (14) |
Australia | 7 (10) |
Germany | 6 (9) |
Netherlands | 4 (6) |
Denmark | 3 (4) |
Other | 19 (28) |
Number of attributesa
| |
2–3 | 5 (7) |
4–5 | 24 (35) |
6 | 25 (36) |
7–9 | 17 (25) |
>9 | 3 (4) |
Number of choices per respondent | |
8 or fewer | 14 (20) |
9–16 choices | 47 (68) |
More than 16 choices | 5 (7) |
Not clearly reported | 3 (4) |
Sample size useda
| |
<100 | 22 (32) |
100–300 | 28 (41) |
300–600 | 17 (25) |
600–1,000 | 10 (14) |
>1,000 | 6 (9) |
Sample size method useda
| |
Parametric approach | 4 (6) |
Louviere et al. [6] | 3 (4) |
Rose and Bliemer [21] | 1 (1) |
Rule of thumb | 9 (13) |
5 (7) | |
Pearmain et al. [30] | 2 (3) |
Lancsar and Louviere [9] | 3 (4) |
Referring to studies | 8 (12) |
Review studies | 3 (4) |
Applied studies | 5 (7) |
Not (clearly) reported | 49 (71) |
2.3 Comment on the State of Play
3 Determining Required Sample Sizes for Discrete-Choice Experiments (DCEs): Theory
3.1 Required Elements for Estimating Minimum Sample Size
-
Significance level (α)
-
Statistical power level (1−β)
-
Statistical model used in the DCE analysis [e.g., multinomial logit (MNL) model, mixed logit (MIXL) model, generalized multinomial logit (G-MNL) model]
-
Initial belief about the parameter values
-
The DCE design.
3.1.1 Significance Level (α)
3.1.2 Statistical Power Level (1−β)
3.1.3 Statistical Model Used in the DCE Analysis
3.1.4 Initial Belief About the Parameter Values
3.1.5 DCE Design
3.2 Sample Size Calculation for DCEs
4 Determining Required Sample Sizes for DCEs: A Practical Example
Parameter label | Initial belief parameter value | DCE design code | ||
---|---|---|---|---|
Alternative | Alternative label | |||
Constant (i.e., alternative specific constant for drug treatment; intercept) | A | 1.23 | ||
Alternative 1 | Drug treatment alternative I | 1 | ||
Alternative 2 | Drug treatment alternative II | 1 | ||
Alternative 3 | Opt-out alternative | 0 | ||
Attribute | Attribute levels | |||
Drug administration | Tablet once a month | |||
Tablet once a week | B1 | –0.31 | 1 | |
Injection every 4 months | B2 | –0.21 | 1 | |
Injection once a month | B3 | –0.44 | 1 | |
Effectiveness ( %) | C | 0.028 | ||
5 | 5 | |||
10 | 10 | |||
25 | 25 | |||
50 | 50 | |||
Side effect nausea | D | –1.10 | ||
No | 0 | |||
Yes | 1 | |||
Treatment duration (years) | E | –0.04 | ||
1 | 1 | |||
2 | 2 | |||
5 | 5 | |||
10 | 10 | |||
Cost (€) | F | –0.0015 | ||
0 | 0 | |||
120 | 120 | |||
240 | 240 | |||
720 | 720 |
-
The DCE design should be coded in a text-file in such a way that it can be read correctly into R. That is, the DCE design should contain one row for each alternative. So, there should be nalts × nchoices rows (see Table 3 as an example for our illustration, which contains 48 rows (i.e., 3 alternatives × 16 choice sets); rows 1–3 correspond to choice set 1, rows 4–6 correspond to choice set 2, etc.)Table 3DCE designChoice taskAlternativeConstantI. Route of drug administrationII. EffectivenessIII. NauseaIV. DurationV. CostsAB1B2B3CDEF11110050101201210101011240130000000021100151572022100010010023000000003110002511024032110050017203300000000..............................16110101001072016210012511016300000000Each row should contain the coded attribute levels for that alternative. See Table 3 for how the DCE design for our illustration was coded (columns A–F). For example, row 1 corresponds to the first preventive drug treatment alternative in choice set 1: a drug treatment alternative (value 1, column A) that should be taken as a tablet every week (value 1, column B1), which will result in a 5 % reduction of a hip fracture (value 5, column C) without side effects (value 0, column D), for which the drug treatment duration will be 10 years (value 10, column E) and out-of-pocket costs of €120 are required (value 120, column F). Be aware that only the DCE design (i.e., the ‘white part’ of Table 3) should be in a text file, so that it can be read correctly in R (Box 5)
-
The results of the minimum sample size required to obtain the desired power level for finding an effect when testing at a specific confidence level for each parameter are shown in Table 4. To illustrate the impact of the probability that we will find a significant effect given a specific effect size, we also computed the required sample size for the statistical power level 1−β equal to 0.6, 0.7, and 0.9. Additionally, we also computed the required sample sizes assuming a significance level α of 0.1, 0.025, and 0.01Table 4Minimum sample size required to obtain the desired power level 1−β for finding an effect when testing at a specific confidence level 1−αα =1−β =ConstantI. Route of drug administrationII. EffectivenessIII. NauseaIV. DurationV. CostsAB1B2B3CDEF0.10.62287213211730.050.634311119212740.0250.645815126323660.010.667920535534980.10.733910017212440.050.745614525323560.0250.767319033434670.010.7796250436360100.10.845313924323350.050.867319033434670.0250.879324142535890.010.89119308537474120.10.967820235534980.050.98102263456464100.0250.910125323567478130.010.91215440069959716As can be seen from Table 4, one needs a minimum sample size of 190 respondents with a statistical power of 0.8 and assuming an α = 0.05, whether ‘injection every 4 months’ is significantly different from ‘tablet once a month (reference attribute level)’ (Table 4, column B2). If a smaller sample size of, for example, 111 respondents were to be used and no significant result to be found for this parameter, one has a statistical power of 0.6, assuming an α = 0.05, to conclude that respondents do not prefer ‘tablet every month’ over ‘injection every 4 months’. As a proof of principle, we compared the standard errors and confidence intervals from the actual study [12] against the predicted standard errors and confidence intervals. The results showed that they were quite similar (Table 5), which gives further evidence that our sample size calculation makes sense.Table 5Parameter estimates and precision from an actual discrete-choice experiment study [12] relative to those predicted by the sample size calculationsAttributeMNL results actual study (N = 117)aPredicted results based on 117 subjectsParameter valueSE95 % CISE95 % CIConstant (drug treatment)1.230.2180.81–1.660.1091.02–1.45Drug administration (base level tablet once a month):Tablet once a week–0.310.070−0.45 to −0.170.099–0.50 to –0.12Injection every 4 months–0.210.097−0.41 to −0.020.108–0.43 to –0.01Injection once a month–0.440.100−0.64 to −0.250.094–0.63 to –0.26Effectiveness (1 % risk reduction)0.030.0030.02–0.030.0020.02–0.03Side effect nausea–1.100.104−1.30 to −0.890.065–1.22 to –0.97Treatment duration (1 year)–0.040.010−0.06 to −0.020.010–0.06 to –0.02Cost (€1)–0.00150.0002−0.002 to −0.0010.0002–0.002 to –0.001