Sample size estimation for randomised controlled trials with repeated assessment of patient-reported outcomes: what correlation between baseline and follow-up outcomes should we assume?

In a two-group study with a Normally distributed outcome, comparing POST-randomisation mean outcomes between two groups, the number of subjects per group n_POST assuming equal sample sizes and equal standard deviations (SDs) per group for a two-sided significance level α and power 1 – β is [4]:

where:

δ is the target or anticipated difference in mean outcomes between the two groups

σ is the SD of the outcome post-randomisation (which is assumed to be the same in both groups)

Z_{1 – α/2} and Z_{1 – β} are the appropriate values from the standard normal distribution for the 100 (1 – α/2) and 100 (1 – β) percentiles respectively.

Consider a two-group study with a Normally distributed outcome, with a single baseline and single post-randomisation assessment of outcomes. Comparing mean outcomes between two groups, adjusted for the baseline or pre-randomisation value of the outcome, using an ANCOVA model for the number of subjects per group n_ANCOVA (assuming equal sample sizes and equal SDs, at baseline and post-randomisation, per group) for a two-sided significance level α and power 1 – β is:

Here, ρ denotes the correlation between the baseline and post-randomisation outcomes and σ is the post-randomisation SD, which is assumed to be the same as the baseline SD [1, 5]. Machin et al. [5] refer to the (1 – ρ²) term as the ’design effect’ (DE).

In a two-group study with a Normally distributed outcome, comparing the mean change in outcomes (i.e. post-randomisation outcome – baseline) between two groups, the number of subjects per group n_CHANGE (assuming equal sample sizes and equal SDs, at baseline and post-randomisation, per group) for a two-sided significance level α and power 1 – β is:

Here, δ_c is the target or anticipated difference in mean change in outcomes between the two groups and σ is the post-randomisation SD that is assumed to be the same as the baseline SD. If the expected mean values of the baseline outcomes are the same in both groups, which is likely in an RCT, then δc is the same as δ.

Figure 1 shows the relationship between the total sample size and the correlation between the baseline and post-randomisation outcomes, for the three methods of sample size estimation (POST, CHANGE and ANCOVA) with a 5% two-sided significance level, 90% power, a target difference (a difference in post-treatment means or a difference in mean changes) of 0.50 and an SD of 1.0. Figure 1 shows how the total sample size is constant for POST irrespective of the baseline and post-randomisation follow-up correlation; the sample size declines as the correlation increases for ANCOVA and CHANGE; and that for correlations above 0.5 the sample size for ANCOVA is always the lowest and is less than or equal to the sample size for CHANGE.

The SELF study [6] was a multicentre, pragmatic, unblinded, parallel-group randomised control superiority trial designed to evaluate the clinical effectiveness of a self-managed single exercise programme versus usual physiotherapy treatment for rotator cuff tendinopathy (pain or weakness in the shoulder muscles). The intervention was a programme of self-managed exercise prescribed by a physiotherapist in relation to the most symptomatic shoulder movement. The control group received usual physiotherapy treatment. The primary outcome measure was the total score on the Shoulder Pain and Disability Index (SPADI) at 3 months post-randomisation. The SPADI Shoulder Score ranges from 0, being the best outcome (less disability), to 100 the worst (greater disability).

The original sample size calculation for the SELF trial assumed that a 10-point difference in the mean 3 months post-randomisation SPADI scores between the intervention and control groups would be regarded as a minimum clinical important difference (MCID). It assumed an SD of 24 points, a power of 80% and a (two-sided) significance level of 5%, meaning that using the POST sample size formula, 91 participants per group were required (182 in total). However, in light of new information from an external pilot study, the investigators undertook a sample size re-estimation (SSR) calculation, which was approved by the ethics committee. The new information related to a narrower estimate of population variance from an external pilot RCT (n = 24) of 16.8 points on the SPADI and, additionally, a correlation between baseline and 3 months SPADI scores of 0.5. Using the ANCOVA sample size formula, with an SD of 17 points; correlation between baseline and 3 months SPADI scores of 0.50, 80% power, 5% two-sided significance and a MCID (as before) of 10 points, it was estimated that 34 participants per group were required (68 in total). This contrasts with a sample size of 45 per group using the POST means formula with the revised SD of 17 points. Thus, with a correlation of 0.50 between baseline and follow-up, using the ANCOVA method for sample size estimation, we can reduce the sample size by approximately 25% (i.e. 1–0.5²) compared to the POST treatment means method.

Should the method of sample size estimation mirror the proposed method of statistical analysis (of the outcome data)? That is, if an ANCOVA model is likely to be used in the statistical analysis of the collected outcome data, should an ANCOVA method that allows for the correlation also be used in the sample size estimation method? And if so, what correlation (between baseline and follow-up outcomes) should be assumed and used in the sample size estimation? Other factors/parameters in the sample size estimation method being unchanged, an assumed correlation of 0.70 (between baseline and follow-up outcomes) means that we can halve the require sample size at the study design stage, if we used an ANCOVA method compared to a comparison of POST treatment means method. It is, however, paramount to assess how realistic a correlation of 0.50 or 0.70 between baseline and post-randomisation outcomes is, and to make evidence-based assumptions on these values, as an overestimated correlation could result in an underpowered study. The aim of this paper is to estimate the observed correlations between baseline and post-randomisation follow-up PROMs from a number of RCTs, bridging a gap in the evidence.

This was a secondary analysis of RCTs with continuous patient-reported outcomes (both primary and secondary) undertaken in the School of Health and Related Research (ScHARR) at the University of Sheffield published between 1998 and 2019. Secondary ethics approval was gained through the University of Sheffield ScHARR Ethics Committee (Reference 024041).

For each included trial, the correlation between baseline and post-randomisation outcomes was calculated using the Pearson correlation coefficient [7]. Given a set of n pairs of observations (x₁, y₁), (x₂, y₂), …, (x_n, y_n), with means \( \overline{x} \) and \( \overline{y} \) respectively, then the Pearson correlation coefficient r is given by:

with a standard error SE(r) = \( SE(r)=\sqrt{\frac{1-{r}^2}{n-2}} \).

A variety of summary statistics for the baseline and post-randomisation correlations were calculated, including (1) the unweighted sample mean and median; (2) a weighted sample mean, using the fixed effect inverse variance method [4], and (3) a sample mean with allowance for clustering by trial derived from a multilevel mixed-effects linear model with a random effect for the trial using restricted maximum likelihood estimation (REML) [8]. The correlations were calculated overall and then split by trial, outcome and time point.

Table 1 shows a summary of the 20 RCTs included in the analysis. Various outcome measures were used in the trials for both the primary and secondary outcomes. Table 2 provides a brief description of the outcome measures and how they were scaled. Three of the outcome measures, the Clinical Outcomes in Routine Evaluation - Outcome Measure (CORE-OM), Pelvic Organ Prolapse/Urinary Incontinence Sexual Questionnaire (PISQ-31) and SPADI, have a total score and various subscales: both were included in the analysis. The 20 included RCTs had sample sizes (at baseline) ranging from 49 to 2659 participants. The time points for the post-randomisation to follow-up assessments ranged from 7 days to 24 months. The maximum sample size for the baseline follow-up correlations ranged from 39 to 2659 participants. Four-hundred and sixty-four correlations between baseline and follow-up were estimated in the 20 trials. Table 1 shows, for example, that the Leg Ulcer trial (Trial 1) had 9 outcomes all assessed at 2 post-randomisation time points (3 and 12 months), giving a total of 2 × 9 = 18 correlations. The median number of outcomes per trial was 9 and ranged from 1 (in the 3Mg trial) to 15 (AIM-High, PLINY and IPSU). The median number of correlations calculated per trial was 16.5 and ranged from 1 (in the 3Mg trial) to 65 (in the DiPALS trial). The median number of post-randomisation follow-up time points across the 20 trials was 2.5 and ranged from 1 to 6.

Figure 2 shows a histogram of the 464 estimated baseline to follow-up correlations. The histogram is reasonably symmetrical, and the overall mean correlation was 0.50 (median of 0.51). The baseline to follow-up correlations ranged from − 0.13 to 0.91 with an interquartile range of 0.41 to 0.60. Since the sample sizes for the trials varied from 49 to 2659 participants, a weighted estimate of the mean correlation, using the inverse variance method, was 0.51. Since the 464 correlation estimates were from 20 trials and the correlations were nested or clustered with trials, the estimated mean correlation after allowing for clustering by trial, using a multilevel mixed-effects linear regression model (with a random effect or intercept for the trial), was 0.49 (95% confidence interval [CI] 0.45 to 0.53). These other summary estimates were very similar to the simple unweighted mean value of 0.50.

Table 3 shows the baseline to post-randomisation follow-up correlations aggregated by trial. The largest average correlations per trial showed a mean of 0.67 observed in the PLINY trial; the lowest average correlations were observed in the POLAR trial. The trial with the widest range of correlations was the PRACTICE trial. Figure 3 shows a box and whisker plot of how the observed baseline to follow-up correlations varied across the 20 RCTs along with the overall median correlation. There was considerable intertrial variation in the correlations, and it should be noted that some of the trials had less than or equal to six baseline to follow-up correlations estimated (3Mg [N = 1 outcome and correlation], BEADS [N = 3], Homeopathy [N = 5] and PRACTICE [N = 6]).

The time points for the post-randomisation follow-up assessments ranged from 7 days to 24 months. Table 4 shows the baseline to post-randomisation follow-up correlations by post-randomisation follow-up time point. Figure 4 shows a scatter plot of the baseline to follow-up correlations by post-randomisation follow-up time point for the 464 correlations from the 20 trials. Although it is not obvious from the scatter plot, a multilevel mixed-effects linear regression model (with a random intercept for the trial) suggests a small decline in the baseline to post-randomisation follow-up correlations the further the time points are apart. The estimated regression coefficient from the model was − 0.003 (95% CI − 0.006 to − 0.001; P = 0.005). This implies that for every unit or 1-month increase in the time from baseline to the post-randomisation follow-up the correlation declines by 0.003 point. Figures 5 and 6 show how the correlations change over time for the Short Form Health Survey (SF-36) outcomes (282 correlations and 12 trials) and the EuroQol five dimension scale (EQ-5D) Utility score outcome (29 correlations and 12 trials). A similar pattern to the overall pattern is observed for these specific outcomes with a small decline (0.003 for the SF-36 and 0.002 for the EQ-5D) in baseline to follow-up correlations over time.

Table 5 shows the baseline to post-randomisation correlations by outcome. The SF-36 was the most popular outcome and used in 12 out of the 20 trials. The correlations for SF-36 outcomes and its various dimensions (12 trials and n = 282 correlations) showed a mean of 0.51 (median 0.53), range 0.06 to 0.91. The second most popular outcome was the EQ-5D, which was used in 12 of the trials as well. Correlations for EQ-5D outcomes only (12 trials and n = 50 correlations) showed a mean of 0.49 (median 0.51), range − 0.13 to 0 87. Three of the outcome measures, the CORE-OM, PISQ-31 and SPADI, in Table 5 have a total score and various subscales. There was no clear pattern in the correlations and no reliable evidence that the total scale score correlated more highly than an individual subscale score.