Background
Objectives
Methods/Design
Definitions
-
Missing studies (e.g., unpublished studies);
-
Entire unreported outcomes (e.g., outcomes planned in trial protocols but not included in trial reports).
Categories of participants with potential MPD
-
“Mistakenly randomized and inappropriately excluded,” and “did not receive any treatment” participant categories, which are defined prior to the initiation of the trial intervention, most likely have MPD;
-
“Withdrew consent,” “outcome not assessable,” “moved out of country,” and “lost to follow-up” (for reasons not considered in our other categories) participant categories, which are defined after the initiation of the trial intervention, most likely have MPD;
-
“Dead,” “experienced adverse events,” “non-compliant,” “discontinued prematurely,” and “cross-over” most likely do not have MPD.
Eligibility criteria
Search strategy
Random sampling of citations
Selection process
Data abstraction
-
Type of systematic review (i.e., Cochrane vs. non-Cochrane);
-
Use or non-use of the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach for assessing the confidence in effect estimates by outcome;[9]
-
Source of funding (e.g., private for profit, private not for profit, governmental, not funded, not reported).
-
Type of intervention and control (e.g., pharmacological, surgical);
-
Outcome ranking on patient-importance hierarchy (i.e., I, II, III);
-
Duration of follow-up for outcome measurement;
-
Type of outcome (e.g., efficacy, safety).
-
Documented the categories of participants that could potentially be counted as having MPD (see previous section “Categories of participants with potential MPD”), and when documented, the number within each category;
-
Reported number of MPD and at what level (e.g., at the study arm level, study level, across studies);
-
Explicitly stated using the following for their meta-analysis: intention-to-treat, modified intention-to-treat, per protocol, or as treated;
-
Explicitly stated analyzing participants in the group to which they were randomized;
-
Explicitly stated the analytical method for dealing with MPD in the eligible meta-analysis (i.e., to account for MPD when generating the best effect estimate): complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD. Tables 1 and2 present the numerical details of the different assumptions that could be used to deal with MPD;
Intervention arm | Control arm | |||||
---|---|---|---|---|---|---|
Number of participants randomized | Number of participant with missing data | Number of observed events | Number of participants randomized | Number of participant with missing data | Number of observed events | |
Number of trial participants |
a
|
b
|
c
|
e
|
f
|
g
|
Analytic method | Intervention arm | Control arm | ||
---|---|---|---|---|
Numerator | Denominator | Numerator | Denominator | |
Complete (available) case analysis |
c
|
a - b
|
g
|
e - f
|
None has event |
c
|
a
|
g
|
e
|
All had event |
b + c
|
a
|
f + g
|
e
|
Best case scenario |
c
|
a
|
f + g
|
e
|
Worst case scenario |
b + c
|
a
|
g
|
e
|
Using the concept of RILTFU/FU
| [b x y x c/(a - b)] + c
|
a
| [f x z x g/(e - f)] + g
|
e
|
Incidence for missing participants same as observed in same arma
| [b x c/(a - b)] + c
|
a
| [f x g/(e - f)] + g
|
e
|
Incidence for missing participants in both arms same as observed in control arm | [b x g/(e - f)] + c
|
a
| [f x g/(e - f) ] + g
|
e
|
-
Explicitly stated any additional meta-analyses for dealing with MPD for the outcome of interest (“reported analytical method”) (i.e., to judge the risk of bias associated with MPD): complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD (see Tables 1 and2);
-
Explicitly stated using assumptions for MPD accounted for uncertainty associated with imputing events;
-
Justification for the analytical method used to deal with MPD in the eligible meta-analysis;
-
Tool(s) used to judge risk of bias associated with MPD at the study level (e.g., Cochrane RoB tool), if any;
-
Method(s) used to assess risk of bias associated with MPD at the meta-analysis level (e.g., sensitivity analysis, subgroup analysis), if any.
-
Number of trials included;
-
Numerator and denominator used in the meta-analysis for each arm for each trial;
-
Number of MPD reported (per arm or combined);
-
Pooled relative effect measure (RR or OR) and its associated CI, p value, and measure of heterogeneity (I 2);
-
Analysis model used (i.e., random effect or fixed effect);
-
Statistical method used (e.g., Mantel-Haenszel or Peto);
-
Results of sensitivity analyses applied to account for MPD.
-
Type of reference (e.g., abstract, full-text article);
-
Type of trial (e.g., randomized controlled trial, controlled clinical trial);
-
Duration of follow-up for outcome measurement;
-
Source of funding (e.g., private for profit, private not for profit, governmental, not funded, not reported);
-
Type of outcome (e.g., efficacy, safety).
-
Presence of MPD;
-
Whether the trialists report the MPD for this specific outcome, as opposed to reporting premature end of follow-up for trial participants in general;
-
Whether the trialists followed-up categories of participants that could be potentially counted as having MPD (see previous section “Categories of participants with potential MPD”);
-
Whether the trialists included in the analysis the categories of participants that could be potentially counted as having MPD in the analysis;
-
Level of reporting number of participants with MPD (e.g., per arm, both arms combined);
-
Whether the trialists described the pattern of missingness;
-
Whether the trialists compared the baseline characteristics of participants with and without MPD (e.g., MPD group vs. non-MPD group, MPD first arm vs. MPD second arm)
-
Number of participants in each of the categories of participants that, following our rules stated previously, would be counted as having MPD, per arm;
-
Whether the trialists report using the following for their analysis: intention-to-treat, modified intention-to-treat, per protocol, or as treated;
-
Whether the trialists report analyzing participants in the group to which they were randomized;
-
Whether the trialists report the analytical method for dealing with MPD in the main analysis, i.e., to account for MPD when generating the effect estimate (e.g., complete case analysis, making assumptions for MPD);
-
Whether methods using assumptions for MPD accounted for uncertainty associated with imputing events;
-
Whether the trialists provided any justification for the method used to deal with MPD in the main analysis;
-
Method(s) used to assess risk of bias associated with MPD in the analysis (e.g., sensitivity analysis, different methods for different subgroups of participants with MPD), if any;
-
Numerical information for the comparison and outcome of interest: number randomized per arm, number of observed events per arm (see Table 1);
-
Relative effect measure (e.g., RR, OR) and its associated CI, and p value;
-
Analytical method “actually” used by the systematic review authors based on the numerical information provided by the trialists;
-
Whether the trialists report any additional analytical methods for dealing with MPD (“reported analytical method”), i.e., to judge the risk of bias associated with MPD (e.g., complete case analysis, making assumptions for MPD);
-
Whether the trialists incorporated implications of MPD in results or discussion sections.
Sample size
Data analysis
Agreement
Characteristics of the included systematic reviews
Categories of trial participants reviewers considered as having MPD
-
The percentage of systematic reviews documenting that category;
-
The percentage of trials documenting that category;
-
The distribution across trials of the proportion of participants belonging to that category.
Reporting of MPD by trialists
-
The percentage of trials following-up categories of participants that could be potentially counted as having MPD;
-
The percentage of trials documenting the number of participants in each arm with MPD; if applicable;
-
The percentage of trials documenting different numbers of participants with MPD across different outcomes, if applicable;
-
The percentage of trials documenting the pattern of missingness;
-
The percentage of trials documenting the comparison of the baseline characteristics of participants with and without missing participant data;
-
The percentage of trials documenting the methods used to assess risk of bias associated with MPD in the analysis.
Reported vs. actual method of dealing with of MPD in meta-analysis
-
The percentage of systematic reviews using each of the terms “intention-to-treat,” “modified intention-to-treat,” “per protocol,” and “as treated;”
-
The percentage of systematic reviews reporting a plan to analyze participants in the group to which they were randomized;
-
The percentage of systematic reviews using different analytical methods for dealing with MPD in the main analysis, i.e., when generating the best effect estimate;
-
The percentage of systematic reviews using imputed data for MPD and accounting for uncertainty associated with imputing events;
-
The percentage of systematic reviews justifying the analytical method used to deal with MPD in the eligible meta-analysis;
-
The percentage of systematic reviews using different analytical methods (e.g., sensitivity analysis, subgroup analysis) to judge risk of bias associated with MPD;
-
The percentage of systematic reviews using different tools (e.g., Cochrane RoB tool) to measure risk of bias associated with MPD in individual trials, and for the body of evidence, across trials, supporting the outcome of interest.
-
Whether the trialists report using the following for their analysis: intention-to-treat, modified intention-to-treat, per protocol, as treated;
-
Whether the trialists report analyzing participants in the group to which they were randomized.
Impact of different methods of dealing with MPD on pooled effect estimates
-
None of the participants with MPD had the event;
-
All participants with MPD had the event;
-
None of the participants with MPD in the treatment group had the event and all participants with MPD in the control group did (best case scenario);
-
All participants with MPD in the treatment group had the event and none of participants with MPD in the control group did (worst case scenario).