Introduction
Methods
Inclusion and exclusion criteria
Search strategy
Data extraction and in-depth evaluation of recent IMPF articles
Rationale and initiation:
| We recorded the rationale for the IMPF, and whether there was mention of a project protocol and ethics approval. |
Process of obtaining IPD:
| We recorded how researchers identified relevant primary studies (e.g. systematic review, coalition of research groups); how they decided which studies to seek IPD from; the process of obtaining IPD; and problems encountered. |
Details of IPD obtained:
| We recorded the proportion of studies providing IPD; the total number of participants in the IPD; whether the number of participants and events were reported for each IPD study; whether there was any missing data problems; and whether there was variability in how prognostic factors were measured. |
Type and quality of IPD studies:
| We recorded the design (e.g. cohort, randomised trials) of studies providing IPD; whether they were published or unpublished; and whether they were assessed for their quality and, if so, how. |
Statistical methods used:
| We recorded whether a statistical methods section was provided; the statistical models used in the meta-analysis (e.g. Cox regression, logistic regression); and how some specific statistical issues were addressed (such as clustering of participants within studies; between-study heterogeneity in prognostic factor effects; and the analysis of continuous prognostic factors). |
Assessment of publication bias and availability bias:
| We recorded if and how researchers examined the potential impact of publication bias (studies unpublished due to non-significant prognostic results) or availability bias (studies providing IPD are a biased portion of the studies from which IPD was desired) in their meta-analysis. |
Adherence to reporting guidelines:
| |
Limitations and challenges of an IMPF:
| We catalogued all the problems that hindered the IMPF approach as reported by the researchers. |
Results
Search and classification results
Overview of the 48 IMPF articles
In-depth evaluation of 20 recent IMPF articles
Rationale and initiation
Process of obtaining IPD
Details of IPD obtained
Type and quality of IPD studies
Statistical methods used
Statistical models for the patient-level data
Meta-analysis framework
Assessing and accounting for heterogeneity in prognostic factor effects
Analysis of continuous factors
Assessment of independent prognostic value
Assessment of interactions between prognostic factors
Assessment of potential biases
Publication bias
Availability bias
Adherence to reporting guidelines
Limitations and challenges of an IMPF
Identifying all relevant studies
| |
· | Unavailability of IPD in some studies |
· | Time-consuming and costly nature of obtaining, cleaning and analysing the IPD. |
Issues within individual studies
| |
· | Dealing with skewed continuous variables and possible outliers. |
· | Inability of IPD to overcome deficiencies of original studies, such as being retrospective rather than prospective, being too small for a multivariable analysis, missing important confounders, missing participant data or being of low methodological quality, etc. |
· | How to assess the quality of studies identified |
· | Re-analysing individual study IPD before considering meta-analysis. For a summary of important issues for the analysis of single prognostic factor studies see Holländer and Sauerbrei [9]. The re-analysis of a single study as the preliminary or first step toward a meta-analysis is influenced by and has consequences for the meta-analysis strategy (15). |
Heterogeneity between studies
| |
· | Different definitions of disease or outcome; e.g. Noordzij et al.[44] note different definitions of hypocalcemia across studies, whilst the MeRGE [40] collaborators note the definition of acute myocardial infarction changed over time. |
· | Different participant inclusion and exclusion criteria |
· | Different methods of measuring the same prognostic factor, for example see difficulties described by Look et al [2]. |
· | For survival data different lengths of follow-up |
· | Factors measured at different points in time or at different stages of disease across studies; e.g. the MeRGE [40] collaborators note that the timing of echocardiography was variable in their included studies, although within 2 weeks of the index acute myocardial infarction |
· | Different (or out-dated) treatments strategies, especially when a mixture of older and newer studies are combined; e.g. Yap et al. [36] state that a large proportion of the patients in their included trials did not receive common post-myocardial infarction therapy such as β-blockers and ACE inhibitor. |
· | Insufficent information about treatment for some of the studies. |
Statistical issues for meta-analysis
| |
· | Missing data, including: missing factor values and outcome data for some participants within a study, and unavailable factors in some studies |
· | Inability to adjust prognostic effects for a consistent set of adjustment factors in each study |
· | Different measurement techniques between studies may be acceptable for adjustment variables, but are critical for the variable of main interest |
· | Insufficient information to separate patient outcomes more discretely, e.g. Thakkinstian et al. [37] could not separate chronic allograft nephropathy from graft rejection or acute rejection from chronic rejection |
· | Imposed choice of cut-off levels when individual studies categorise their continuous variables and/or categorise their continuous outcomes in their provided IPD |
· | Difficulty in using a continuous scale for continuous factors in meta-analysis when some studies give IPD values on a continuous cale and others do not (e.g. see Rovers et al. [43]) |
· | Considering whether it is sensible and/or possible to investigate differential prognostic effects in subgroups |
· | Potential for study-level confounding when assessing whether study covariates (e.g. year of publication) modify the prognostic effect. |
· | Difficulty of interpreting summary meta-analysis results in the presence of heterogeneity (and heterogeneous populations) across studies. |
Assessment of potential biases
| |
· | Potential for publication bias and availability bias |
· | How to assess the robustness of IPD meta-analysis results to the inclusion/exclusion of studies only providing summary data; and how to combine IPD studies with summary data studies |
Discussion
Rationale & Initiation
| |
· | Produce a protocol for the IMPF project prior to its initiation (detailing all aspects of rationale, conduct and statistical analysis) and reference this upon publication of the IMPF |
· | Consider whether ethics approval is necessary for the IMPF project, and report this upon publication |
Process of obtaining IPD
| |
· | Report how primary study authors were approached to obtain their IPD |
· | Report the strategy used for searching the literature for relevant studies (if relevant), including keywords used and databases searched. |
· | Provide a flowchart showing the search strategy, classification of identified articles, and retrieval of IPD from relevant studies (where relevant) |
· | Consider how to improve retrieval of IPD from unpublished studies |
Details of IPD obtained
| |
· | Report number of participants and events for each included study |
· | Report a summary of the missing data for each study |
· | Report the reasons why IPD was unavailable for some studies (if relevant), and if possible, report the number of participants, number of events and summary prognostic factor results in such studies |
Type and quality of IPD obtained
| |
· | Consider and report the quality of studies for which IPD were obtained; in particular, are they all of comparable quality? |
Statistical methods used
| |
· | Check and report the assumptions of the statistical models used; in particular, do model assumptions appear valid in each study separately? |
· | Where possible, analyse continuous factors on their continuous scale and consider non-linear trends. Univariate analyses are a good starting point, but a multivariable analysis adjusting for ‘standard’ factors is required to assess the added prognostic value of a factor over ‘established’ factors. |
· | In multivariable analyses consider carefully which variables can and should be used for adjustment. Sensitivity analyses should be conducted. In a similar way consider how treatment differences can be handled in the analysis. |
· | In multivariable analyses, define the criteria used to decide whether a factor has independent prognostic value over other factors; also potentially consider whether the interaction between two (or more) prognostic factors is important |
· | Consider a re-analysis of the IPD in each study as a first or preliminary step toward meta-analysis, to better appreciate the issues within each study first. |
· | In the meta-analysis, account for clustering of participants within studies (and do not merge IPD and analyse as if IPD all came from a single study) and report how this was done. |
· | Measure and, if necessary, account for between study heterogeneity in the prognostic factor effect(s) of interest when undertaking meta-analysis |
· | Where sufficient studies are available (e.g. 10 per covariate of interest) and heterogeneity of estimated effects of interest exists, examine the potential causes of such heterogeneity. |
· | Consider a sensitivity analysis to assess whether meta-analysis conclusions change when restricting to IPD from the higher quality studies (if relevant) |
Assessment of publication bias and availability bias
| |
· | Consider the potential impact of publication bias and availability bias on IPD meta-analysis results; in particular, are studies providing IPD comparable to those studies not providing IPD (if relevant)? |
Reporting guidelines
| |
· | Utilise reporting guidelines for meta-analysis, such as those for MOOSE [24] and IPD meta-analysis [16] |