Background
Objectives
Methods
Definitions and important concepts
Methodological quality and completeness of reporting
Eligibility criteria
Inclusion criteria
Exclusion criteria
Search methods
Screening
Data extraction
Analyses
Results
Report characteristics
Characteristic | Reports using PRISMA
N = 13 | Reports using QUOROM
N = 7 | Reports using OQAQ
N = 26 | Reports using AMSTAR
N = 27 | All reports
N = 56 | ||||||
---|---|---|---|---|---|---|---|---|---|---|---|
n
| % |
n
| % |
n
| % |
n
| % |
n
| % | ||
Year of publication of methodological report | 1996–2010 | 0 | 0 | 7 | 100 | 20 | 77 | 0 | 0 | 21 | 38 |
2010–2014 | 13 | 100 | 0 | 0 | 6 | 23 | 27 | 100 | 35 | 63 | |
Number of assessed SRs across reports | Median (IQR) | 88 (37, 134) | 61 (53, 107) | 59 (31, 109) | 46 (22, 106) | 57 (30, 109) | |||||
Range | 10–487 | 10–161 | 10–200 | 10–369 | 10–487 | ||||||
Were SRs of particular medical field? | No | 1 | 8 | 2 | 29 | 2 | 8 | 1 | 4 | 5 | 9 |
Yes | 12 | 92 | 5 | 71 | 24 | 92 | 26 | 96 | 51 | 91 | |
Intent of assessment | MQ tool for MQ assessment | – | – | – | – | 16 | 62 | 18 | 67 | 34 | 61 |
RQ tool for RQ assessment | 2 | 15 | 2a
| 29 | – | – | – | – | 4 | 7 | |
Both MQ and RQ (and appropriate use of tool, accordingly) | 10 | 77 | 5 | 71 | 7 | 27 | 8 | 30 | 15 | 27 | |
Used MQ tool for RQ assessment | – | – | – | – | 1a
| 4 | – | – | 1 | 2 | |
Used MQ tool for both MQ and RQ assessment | – | – | – | – | 1 | 4 | – | – | 1 | 2 | |
Used MQ tools plus other criteria; both MQ and RQ assessedb
| – | – | – | – | 1c
| 4 | 1c
| 4 | 1 | 2 | |
Used RQ tool for both MQ and RQ assessment | 1 | 8 | – | – | – | – | – | – | 1 | 2 | |
Cohort of Cochrane SRs | Cochrane only | 0 | 0 | 3 | 43 | 4 | 15 | 0 | 0 | 4 | 7 |
Sample of reviews | 6 | 46 | 3 | 43 | 11 | 42 | 13 | 48 | 28 | 50 | |
Specific journal sample or other | 7 | 54 | 1 | 14 | 11 | 42 | 14 | 52 | 24 | 43 | |
Number of databases searched | 1 | 2 | 15 | 4 | 57 | 4 | 15 | 4 | 15 | 10 | 18 |
2 | 0 | 0 | 1 | 14 | 2 | 8 | 3 | 11 | 5 | 9 | |
3 | 1 | 8 | 1 | 14 | 6 | 23 | 1 | 4 | 7 | 13 | |
4 | 4 | 31 | 0 | 0 | 5 | 19 | 4 | 15 | 10 | 18 | |
5 | 1 | 8 | 1 | 14 | 5 | 19 | 4 | 15 | 8 | 14 | |
6 | 1 | 8 | 0 | 0 | 1 | 4 | 2 | 7 | 3 | 5 | |
7 | 1 | 8 | 0 | 0 | 1 | 4 | 2 | 7 | 3 | 5 | |
8+ | 0 | 0 | 0 | 0 | 2 | 8 | 1 | 15 | 3 | 5 | |
Not reported | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 7 | 2 | 4 | |
Not applicable (select journals) | 3 | 23 | 0 | 0 | 0 | 0 | 4 | 15 | 5 | 9 | |
Reports restricted SRs by language | No restrictions | 2 | 15 | 2 | 29 | 12 | 46 | 4 | 15 | 15 | 27 |
Not reported | 7 | 54 | 4 | 57 | 10 | 39 | 13 | 48 | 22 | 39 | |
Restricted to English | 1 | 8 | 1 | 14 | 6 | 23 | 7 | 26 | 13 | 23 | |
Restricted to English and other specified languages | 3 | 23 | 0 | 0 | 0 | 0 | 3 | 11 | 6 | 11 | |
SR defined for inclusion criteria | Not reported | 2 | 15 | 1 | 14 | 7 | 27 | 6 | 22 | 12 | 21 |
Yes, but no reference given | 4 | 31 | 1 | 14 | 5 | 19 | 5 | 19 | 10 | 18 | |
“Systematic review” reported as a search term | 5 | 39 | 4 | 57 | 13 | 50 | 9 | 33 | 24 | 43 | |
Cochrane Collaboration and PRISMA Statement | 2 | 15 | 1 | 14 | 2 | 8 | 5 | 19 | 7 | 13 | |
Other reference | 0 | 0 | 0 | 0 | 1 | 4 | 2 | 7 | 3 | 5 | |
Was a study protocol reported as available for this report? | No or not reported | 11 | 85 | 6 | 86 | 24 | 92 | 24 | 89 | 49 | 88 |
Yes, link reported | 2 | 15 | 0 | 0 | 1 | 4 | 1 | 4 | 2 | 4 | |
Yes, upon request | 0 | 0 | 1 | 14 | 3 | 12 | 2 | 7 | 5 | 9 | |
Report source of funding | Industry Funded | 0 | 0 | 0 | 0 | 2 | 8 | 1 | 4 | 1 | 2 |
Non-profit Funding | 7 | 54 | 3 | 43 | 13 | 50 | 10 | 37 | 26 | 46 | |
Reported no funding | 1 | 8 | 1 | 14 | 5 | 19 | 6 | 22 | 8 | 14 | |
Not reported | 5 | 39 | 3 | 43 | 10 | 39 | 10 | 37 | 21 | 38 |
Adherence to MQ and RQ items in methodological reports
Item assessed | Item description | No. of reports reporting adherence by item | Adhering SRs | Total SRs | % |
---|---|---|---|---|---|
1. Title | Identify the report as a systematic review, meta-analysis, or both | 13 | 1480 | 1741 | 85 |
2. Abstract: structured summary | Provide a structured summary including the following as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number | 13 | 885 | 1741 | 51 |
3. Introduction: rationale | Describe the rationale for the review in the context of what is already known | 13 | 1532 | 1741 | 88 |
4. Objectives | Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS) | 13 | 1039 | 1741 | 60 |
5. Methods: protocol and registration | Indicate if a review protocol exists, if and where it can be accessed (e.g., web address), and, if available, provide registration information including registration number | 13 | 102 | 1741 | 6 |
6. Eligibility criteria | Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale | 13 | 1342 | 1741 | 77 |
7. Information sources | Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched | 13 | 1530 | 1741 | 88 |
8. Search | Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated | 13 | 923 | 1741 | 53 |
9. Study selection | State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis) | 13 | 1048 | 1741 | 60 |
10. Data collection process | Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators | 13 | 1059 | 1741 | 61 |
11. Data items | List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made | 13 | 865 | 1741 | 50 |
12. Risk of bias in individual studies | Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in any data synthesis | 13 | 1251 | 1741 | 72 |
13. Summary measures | State the principal summary measures (e.g., risk ratio, difference in means) | 13 | 1353 | 1741 | 78 |
14. Synthesis of results | Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I
2) for each meta-analysis | 13 | 1129 | 1736 | 65 |
15. Risk of bias across studies | Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies) | 13 | 657 | 1741 | 38 |
16. Additional analyses | Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified | 13 | 879 | 1738 | 51 |
17. Results: study selection | Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram | 13 | 1094 | 1740 | 63 |
18. Study characteristics | For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations | 13 | 1324 | 1741 | 76 |
19. Risk of bias within studies | Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12) | 13 | 1199 | 1738 | 69 |
20. Results of individual studies | For all outcomes considered (benefits or harms) present for each study: (a) simple summary data for each intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot | 13 | 1399 | 1737 | 81 |
21. Synthesis of results | Present results of each meta-analysis done, including confidence intervals and measures of consistency | 13 | 1150 | 1687 | 68 |
22. Risk of bias across studies | Present results of any assessment of risk of bias across studies (see item 15) | 13 | 527 | 1736 | 30 |
23. Additional analysis | Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see item 16]) | 13 | 631 | 1658 | 38 |
24. Discussion: summary of evidence | Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers) | 13 | 1085 | 1741 | 62 |
25. Limitations | Discuss limitations at study and outcome level (e.g., risk of bias) and at review-level (e.g., incomplete retrieval of identified research, reporting bias) | 13 | 1358 | 1741 | 78 |
26. Conclusions | Provide a general interpretation of the results in the context of other evidence and implications for future research | 13 | 1480 | 1741 | 85 |
27. Funding | Describe sources of funding for the systematic review and other support (e.g., supply of data) and role of funders for the systematic review | 13 | 647 | 1741 | 37 |
Item assessed | Item description | No. of reports reporting adherence by item | Adhering SRs | Total SRs | % |
---|---|---|---|---|---|
Title | Identify the report as a systematic review | 6 | 133 | 449 | 30 |
Abstract | Use a structured format | 6 | 402 | 449 | 90 |
Describe the clinical question explicitly | 6 | 341 | 449 | 76 | |
Describe the databases (i.e., list) and other information sources | 6 | 335 | 449 | 75 | |
Describe the selection criteria (i.e., population, intervention, outcome, and study design), methods for validity assessment, data abstraction, and study characteristics, and quantitative data synthesis in sufficient detail to permit replication | 5 | 177 | 388 | 46 | |
Describe characteristics of the RCTs included and excluded; qualitative and quantitative findings (i.e., point estimates and confidence intervals); and subgroup analyses | 5 | 180 | 388 | 46 | |
Describe the main results | 6 | 425 | 449 | 95 | |
Introduction: rationale | Describe the explicit clinical problem, biological rationale for the intervention, and rationale for review | 6 | 382 | 449 | 85 |
Search | Describe the information sources, in detail (e.g., databases, registers, personal files, expert informants, agencies, hand-searching), and any restrictions (years considered, publication status, language of publication) | 5 | 274 | 388 | 71 |
Study selection | Describe the inclusion and exclusion criteria (defining population, intervention, principal outcomes, and study design) | 6 | 417 | 449 | 93 |
Data collection process | Data extraction: describe the process or processes used (e.g., completed independently, in duplicate) | 6 | 363 | 449 | 81 |
Data items | Describe the type of study design, participants’ characteristics, details of intervention, outcome definitions, and how clinical heterogeneity was assessed | 6 | 316 | 449 | 70 |
Risk of bias in individual studies | Validity assessment: describe the criteria and process used (e.g., masked conditions, quality assessment, and their findings) | 6 | 240 | 449 | 54 |
Synthesis of results | Describe the principal measures of effect (e.g., relative risk), method of combining results (statistical testing and confidence intervals), handling of missing data; how statistical heterogeneity was assessed; a rationale for any a priori sensitivity and subgroup analyses; and any assessment of publication bias | 5 | 219 | 388 | 56 |
Results: study selection | Provide a meta-analysis profile summarizing trial flow | 6 | 40 | 449 | 9 |
Study characteristics | Present descriptive data for each trial (e.g., age, sample size, intervention, dose, duration, follow-up period) | 6 | 384 | 449 | 86 |
Results of individual studies | Report agreement on the selection and validity assessment; present simple summary results (for each treatment group in each trial, for each primary outcome); present data needed to calculate effect sizes and confidence intervals in intention-to-treat analyses (e.g., 2 × 2 tables of counts, means and SDs, proportions) | 5 | 213 | 388 | 55 |
Discussion: summary of evidence | Summarize key findings; discuss clinical inferences based on internal and external validity; interpret the results in light of the totality of available evidence; describe potential biases in the review process (e.g., publication bias); and suggest a future research agenda | 5 | 265 | 388 | 68 |
Item assessed | Item Description | No. of reports reporting adherence by item | Adhering SRs | Total SRs | % |
---|---|---|---|---|---|
1. Methods: Protocol and registration | Was an 'a priori' design provided? | 23 | 820 | 1794 | 46 |
2. Information sources | Was the status of publication (i.e. grey literature) used as an inclusion criterion? | 23 | 1013 | 1794 | 57 |
3. Search | Was a comprehensive literature search performed? | 23 | 1149 | 1794 | 64 |
4. Data collection process | Was there duplicate study selection and data extraction? | 23 | 534 | 1794 | 30 |
5. Results: Study selection | Was a list of studies (included and excluded) provided? | 22 | 537 | 1779 | 30 |
6. Study characteristics | Were the characteristics of the included studies provided? | 23 | 1439 | 1794 | 80 |
7. Risk of bias within studies | Was the scientific quality of the included studies assessed and documented? | 23 | 1200 | 1794 | 67 |
8. Synthesis of results | Were the methods used to combine the findings of studies appropriate? | 23 | 1169 | 1794 | 65 |
9. Risk of bias across studies | Was the likelihood of publication bias assessed? | 23 | 995 | 1794 | 56 |
10. Limitations | Was the scientific quality of the included studies used appropriately in formulating conclusions? | 23 | 590 | 1794 | 33 |
11. Funding | Was the conflict of interest stated? | 22 | 685 | 1779 | 39 |
Item assessed | Item description | No. of reports reporting adherence by item | Adhering SRs | Total SRs | % |
---|---|---|---|---|---|
1. Information sources | Were the search methods used to find evidence reported? | 22 | 1027 | 1387 | 74 |
2. Search | Was the search strategy for evidence reasonably comprehensive? | 22 | 754 | 1370 | 55 |
3. Study selection | Were the criteria used for deciding which studies to include in the overview reported? | 22 | 1112 | 1387 | 80 |
4. Risk of bias in individual studies | Were criteria used for assessing validity of the included studies reported? | 22 | 499 | 1367 | 37 |
5. Synthesis of results | Were findings of the relevant studies combined appropriately relative to the primary question addressed? | 22 | 830 | 1387 | 60 |
6. Results: study selection | Was bias in the selection of studies avoided? | 22 | 740 | 1351 | 55 |
7. Synthesis of results | Were methods used to combine the findings of relevant studies (to reach a conclusion) reported? | 22 | 1005 | 1387 | 73 |
8. Limitations | Was the validity of all studies referred to in the text assessed using appropriate criteria (either in selecting studies for inclusion or in analyzing studies that are cited)? | 22 | 898 | 1363 | 66 |
9. Conclusions | Were the conclusions made by the author (s) supported by the data and/or analysis reported in the overview? | 22 | 1076 | 1387 | 78 |