To pool outcome data for trials that compare the same intervention with the same comparator, we will use random effects meta-analyses, which are conservative in that they consider both within and among study differences in calculating the error term used in the analysis[
11]. We will pool cross-over trials with parallel design RCTs, using methods outlined in the Cochrane handbook to derive effect estimates[
72]. Specifically, we will perform a paired
t-test for each cross-over trial if either of the following are available: 1) the individual participant data; 2) the mean and SD (or standard error) of the participant-specific differences between the intervention and control measurements; 3) the mean difference and one of the following: (i) a t-statistic from a paired
t-test, (ii) a p-value from a paired
t-test, or (iii) a confidence interval from a paired analysis; or 4) a graph of measurements of the intervention arm and control arm from which we can extract individual data values (pending that the matched measurements for each individual can be identified)[
72]. If these data are not available, we will approximate paired analyses by first calculating the mean difference for the paired analysis (MD = M
E - M
C) and the standard error of the mean difference:
, where N represents the number of participants in the trial, and SD
diff represents the standard deviation of within-participant differences between the intervention and control measurements[
72]. If the standard error or standard deviation of within-participant differences is not available, we will impute the standard deviation using methods outlined in the Cochrane Handbook[
73].
Ensuring interpretable results from pooled estimates of effect
We will use a number of approaches to facilitate interpretable results from our meta-analyses. For trials that report dichotomous outcomes, we will calculate the odds ratio (OR) to inform relative effectiveness. We will acquire estimates of baseline risk from observational studies or, if not available, from the median of the control group from eligible RCTs.
When pooling across trials that report continuous outcomes using the same instrument, we will calculate the weighted mean difference (WMD), which maintains the original unit of measurement and represents the average difference between groups[
74]. Once the WMD has been calculated, we will contextualize this value by noting the corresponding minimally important difference (MID) - the smallest change in instrument score that patients perceive is important. We will prioritize use of anchor-based MIDs when available, and calculate distribution-based MIDs when they are not.
Contextualizing the WMD through the MID can be misleading because clinicians may interpret a WMD less than the MID as suggesting that no patient obtains an important benefit, which is not accurate. Therefore, we will generate an estimate of the proportion of patients who have benefited by applying the MID to individual studies, estimating the proportions who benefit in each study, and then aggregate the results in order to provide a summary estimate of the proportion of patients who benefit from treatment across all studies. Details of the methods by which we will conduct this analysis are presented immediately below in our discussion of situations in which investigators have used different instruments to measure the same construct.
For trials that use different continuous outcome measures that address the same underlying construct, one cannot calculate a weighted mean difference, and we will therefore calculate a measure of effect called the standardized mean difference (SMD) or ‘effect size’. This involves dividing the difference between the intervention and control means in each trial (that is, the mean difference) by the estimated between-person standard deviation (SD) for that trial. The SMD expresses the intervention effect in SD units rather than the original units of measurement, with the value of a SMD depending on both the size of the effect (the difference between means) and the SD of the outcomes (the inherent variability among participants).
This common approach to pooling continuous outcome data is often problematic. If the heterogeneity of patients is different across studies, the SD will vary across studies. Therefore, given the same true difference in outcome between intervention and control groups, trials with more heterogeneous patients will show apparently - but spuriously - smaller effects than trials enrolling less heterogeneous patients. Furthermore, interpretation of the magnitude of effect when represented as SD units is not intuitive.
In order to address these issues, we will contextualize the SMD value through MID units, which are not vulnerable to the distortions that varying heterogeneity of populations can create and are more interpretable to both clinicians and patients[
58,
75]. For outcome measures that have an established anchor-based MID we will use this measure to convert the summary effect into OR. We will complement this presentation by either converting the summary effect into natural units of a widely accepted instrument used to measure changes in the domain of interest or, if such an instrument is not available, we will substitute the MID for the SD (denominator) in the SMD equation, which will result in more readily interpretable MID units instead of SD units[
58]. Finally, we will, as for SMD, provide a summary estimate of the proportion of patients who benefit from treatment across all studies
We illustrate this approach with the following example. We will first describe how we will summarize the outcome in MID units. Assume that a trial reports a mean difference (MD) on a continuous outcome measure “X”, and assume that an anchor-based MID for instrument X, MID
X, has been established. The estimated MD is a random variable. If we standardize this random variable by dividing it by the MID
X, we get a new random variable, MD/MID
X. We know from basic probability theory that because MID
X is a constant, the variance of MD/MID
X is given by:
That is, the variance of the mean difference divided by the square of the MID. Further, the standard error of MD/MID
X is given by:
Consider a meta-analysis that included k trials. The first j trials use disease-specific instrument A, and the last k-j trials use disease-specific instrument B. Let MD
i denote the mean difference observed in trial i, let MID
A denote the MID established for instrument A, and let MID
B denote the MID established for instrument B. Further, let m
i denote the MID standardized effect for trial i. To pool results across trials using MIDs we must first estimate the m
i and its associated variance for all trials. For i = 1, …, j we have:
and for i = j + 1, …, k we have:
By defining the trial weights as w
i = Var(m
i)
-1, we can use the fixed-effect model inverse variance method to pool the MID-standardized mean differences using the formula:
Where
denotes the pooled MID-standardized mean difference. The standard error of
can be calculated using the formula:
The associated confidence intervals can subsequently be derived. MID-standardized mean differences can also be combined in a random-effects model using weights wi = (Var(mi) + τ2)-1, where τ2 denotes the between-trial variance.
This presentation does not address the risk that clinicians may interpret all mean effects below the MID as unimportant, and presume important benefit for all patients when mean effects exceeds the MID. We will address this issue by assuming normal distributions of data and then calculating the proportions of participants in the intervention and control groups in each study that demonstrated an improvement greater than the MID[
76]. The results are then pooled across studies. If we only have post-test data (rather than magnitude of change), we will apply this approach if evidence exists regarding meaningful thresholds. For instance, if one knows that people with scores of less than 8 on the Hamilton rating scale for depression (HAM-D) are considered to be not depressed, one could examine the proportion of individuals below that threshold.
If such meaningful thresholds do not exist, we will use post-test data and assume that the minimally important change within an individual corresponds, on average, to the minimally important difference between individuals. Making this assumption, one can calculate the difference in the proportion who benefit in intervention and control. To do this, we will take the mean value in the control group plus one MID unit, and calculate the proportion of patients in each group above that threshold.
If an anchor-based MID has not been established for all instruments, we will assume a meta-analysis control group probability (
p
C
) and use the SMD to calculate the OR. Specifically, we will construct a conceptual meta-analysis control group with mean
μ
C
, standard deviation
σ
C
, and group size
n
C
, and a conceptual meta-analysis intervention group with mean
μ
E
, standard deviation
σ
E
, and group size
n
E
such that the SMD can be represented as
SMD = μ
E
-
μ
C
and
σ
E
= σ
C
= 1. We will set
μ
C
= 0 and
σ
C
= 1, and our threshold (
T) will be equal to
Φ
-1
(p
C
), where
Φ
-1
is the inverse standard normal cumulative distribution function. We will then use the derived threshold to calculate the conceptual intervention group probability (
p
E
). The intervention group mean response is assumed to follow a normal distribution with mean SMD and a SD of 1. Thus, the intervention group probability is
p
E
= 1 −
Φ(
T −
SMD). Having estimated the conceptual meta-analysis control and intervention group probabilities from the pooled SMD, we will calculate the OR as follows:
To calculate the 95% CI, we will use the above formulas substituting the upper and lower bounds of the SMD confidence interval. We will complement this presentation by converting the SMD into natural units of a widely accepted instrument used to measure changes in the domain of interest or, if such an instrument is not available, we will calculate the ratio of means[
58].
Knowledge translation
We plan to create a stakeholder advisory committee with representation from ambulatory health care providers from across Ontario, Canada as well as from key organizations. We will ensure that we have geographically diverse representation including primary care providers who practice in rural areas of the province. Members of our stakeholder committee will be invited to attend our planning meeting and share their input/advice with members of the review team.
Our team also will engage in an end-of-study knowledge translation workshop. The purpose of this activity will be to share our findings with key relevant stakeholders (researchers, clinicians and decision-makers) in order to identify future opportunities for dissemination, beyond traditional peer-reviewed publications, with our stakeholders, discuss how to maximize uptake of our findings in patient education and clinical practice, and determine future research directions. The overall goal of the workshop is to develop an agenda that will establish directions to develop and implement our research findings into practice.
The following strategies will be used to promote awareness of the stakeholder meeting findings according to the Ottawa Model of Research Use in which information is tailored to specific audiences: (1) distribution of findings to all involved participants for further input, sharing within their organization, and for sharing with their own stakeholders via newsletter, web site, or other methods; (2) presentation at relevant peer-reviewed and community conferences; and (3) publication in an open-source peer-reviewed journal. We anticipate that this meeting will identify new areas of inquiry for research and practice, such as the development of new educational tools for patients and clinicians. We also anticipate that new collaborations and networks will be created that will support the identified work going forward. Any groups identified through the meeting will be included as part of the report back to the stakeholders in order to broadly disseminate the findings.