Background
Methods
Development of the guidance
Scope of the guidance
Results
Choosing the primary outcome
Specifying the target difference
Name
|
Description
|
Difference specified
|
---|---|---|
Anchor
| The outcome of interest can be ‘anchored’ by using either a patient’s or health professional’s judgement to define an important difference. This may be achieved by comparing a patient’s health before and after treatment, and then linking this change to participants who showed improvement and/or deterioration using a more familiar outcome (for which either patients or health professionals more readily agree on what amount of change constitutes an important difference). An outcome can be anchored to another which more is known about. Contrasts between patients (such as individuals with varying severity of a disease) can also be used to determine a meaningful difference. | Important |
Distribution
| Approaches that determine a value based upon distributional variation. A common approach is to use a value that is larger than the inherent imprecision in the measurement and therefore likely to represent a minimal level for a noticeable difference. | Important |
Health economic
| Approaches that use principles of economic evaluation. These typically include both resource cost and health outcomes, and define a threshold value for the cost of a unit of health effect that a decision-maker is willing to pay, to estimate the overall net benefit of treatment. The net benefit can be analysed in a frequentist framework or take the form of a (typically Bayesian) decision-theoretic value of information analysis. Due to difficulties in implementing a value of information analysis simpler heuristic frameworks, also based on the principles of economic evaluation, have been proposed. | Important |
Opinion-seeking
| The target difference can be based on opinions elicited from health professionals, patients or others. Possible approaches include forming a panel of experts, surveying the membership of a professional or patient body or interviewing individuals. This elicitation process can be explicitly framed within a trial context. | Important and/or realistic |
Pilot study
| A pilot (or preliminary) study may be carried out where there is little evidence, or even experience, to guide expectations and determine an appropriate target difference for the trial. In a similar manner, a phase two study could be used to inform a phase three study, though this would need to take account of methodological differences (such as inclusion criteria and outcomes), which should be reflected in the target difference. | Realistic |
Review of evidence base
| The target difference can be derived using current evidence on the research question. Ideally, this would be from a systematic review or meta-analysis of RCTs. In the absence of randomised evidence, evidence from observational studies could be used in a similar manner. An alternative approach is to undertake a review of studies in which an important difference was determined. | Important and/or realistic |
Standardised effect size
| The magnitude of the effect on a standardised scale defines the value of the difference. For a continuous outcome, the standardised difference (most commonly expressed as Cohen’s d effect size) can be used. Cohen’s cutoff values of 0.2, 0.5 and 0.8 for small, medium and large effects, respectively, are often used. Thus a medium effect corresponds simply to a change in the outcome of 0.5 SDs. Binary or survival (time-to-event) outcome metrics (such as an odds, risk or hazard ratio) can be utilised in a similar manner, though no widely recognised cutoff values exist. Cohen’s cutoff values approximate odds ratios of 1.44, 2.48 and 4.27, respectively. Corresponding risk ratio values vary according to the control group event proportion. | Important |
Reporting the sample size calculation and target difference
Item no.
|
Item description
|
Protocol
|
Report
|
---|---|---|---|
1
| State any divergence from the conventional approach. | √ | √ |
2
| State the primary outcome (and any other outcome which the study sample size calculation is based upon), or state why there is not one. | √ | √ |
3
| √ | ||
4
| State the values used for statistical parameters (such as significance level and power). | √ | √ |
5
| State the underlying basis for determining the target difference: | √ | |
a. an important difference as judged by a stakeholder, | |||
b. a realistic difference based upon current knowledge or | |||
c. both an important and realistic difference. | |||
6
| Express the target difference according to the outcome type: | √ | √ |
a. Binary: state the target difference as an absolute and/or relative effect, along with the intervention and control group proportions. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation. | |||
b. Continuous: state the target mean difference on the natural scale, the common SD and standardised effect size (mean difference/SD). It is preferable to also provide the anticipated control group mean even though it is not required for the sample size calculation. | |||
c. Time-to-event (survival): state the target difference as an absolute and/or relative difference, provide the control group event proportion, along with the intervention and control group survival distributions. Additionally, the planned length of follow-up should be stated along with the assumed accrual pattern. If both an absolute and a relative difference are provided, clarify if either takes primacy in terms of the sample size calculation. | |||
7
| Explain the choice of target difference: specify and reference any formal method used or relevant previous research. | √ | |
8
| State the sample size based upon the assumptions specified above (for a time-to-event outcome, the number of events required should also be stated). If any factors are incorporated which alter the required sample size (such as allowance for loss-to-follow-up) they should also be specified along with the final sample size. | √ | √ |
9
| Reference the trial protocol | √ |
Primary outcome type
|
Example text
|
---|---|
Binary
| Men After Prostate Surgery (MAPS) radical prostatectomy trial [28]: The primary outcome is continence. The sample size was based upon a target difference of 15% absolute difference (85 versus 70%). This magnitude of a target difference was determined as both a realistic and important difference from discussion between clinicians and the project management group, and inspection of the proportion of urinary incontinence in the trials included in the Cochrane systematic review [29]. The control group proportion is also based upon the observed proportion in the RCTs in this review. Setting the statistical significance to the two-sided 5% level and seeking 90% power, 174 participants per group are required; 348 in total. |
Continuous
| Full-thickness macular hole and Internal Limiting Membrane peeling Study (FILMS) [7]: The primary outcome is ETDRS (Early Treatment Diabetic Retinopathy Study) distance visual acuity. A target difference of a mean difference of five letters, with a common standard deviation (SD) of 12, was assumed as five letters is equivalent to one line on a visual acuity chart and is viewed as an important difference by patients and clinicians. The SD value was based upon two previous studies; one RCT and one observational comparative study [30,31]. This target difference is equivalent to a standardised effect size of 0.42. Setting the statistical significance to the two-sided 5% level and seeking 90% power, 123 participants per group are required; 246 in total. |
Time to event
| Arterial Revascularisation Trial (ART) [32]: The primary outcome is all-cause mortality. The sample size was based upon a target difference of 5% in 10-year mortality with a control group mortality of 25%. Both the difference and control group mortality proportions are realistic based upon a systematic review of observational (cohort) studies [33]. Setting the statistical significance to the two-sided 5% level and seeking 90% power, 1464 participants per group are required; 2,928 participants (651 events) in total. Participants will be followed for 10 years. |