Background
Description of the assessment tools
Assessment of multiple systematic reviews (AMSTAR-2)
ROBIS tool
Previous research
Aim
Objectives
Methods
Assessment of methodological quality/bias of the included reviews
Data-analysis
Results
Criteria | AMSTAR-2 | ROBIS |
---|---|---|
Eligibility criteria | 1. Did the research questions and inclusion criteria for the review include the components of PICO? 2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? 3. Did the review authors explain their selection of the study designs for inclusion in the review? | 1.1 Did the review adhere to pre-defined objectives and eligibility criteria? 1.2 Were the eligibility criteria appropriate for the review question? 1.3 Were eligibility criteria unambiguous? 1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate (e.g., date, sample size, study quality, outcomes measured)? 1.5 Were any restrictions in eligibility criteria based on sources of information appropriate (e.g., publication status or format, language, availability of data)? |
Study selection and Data extraction | 5. Did the review authors perform study selection in duplicate? 6. Did the review authors perform data extraction in duplicate? | 2.5 Were efforts made to minimise error in selection of studies? 3.1 Were efforts made to minimise error in data collection? 3.3 Were all relevant study results collected for use in the synthesis? |
Literature search | 4. Did the review authors use a comprehensive literature search strategy? | 2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports? 2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? 2.4 Were restrictions based on date, publication format, or language appropriate? |
Grey literature | NA | 2.2 Were methods additional to database searching used to identify relevant reports? |
List of studies | 7. Did the review authors provide a list of excluded studies and justify the exclusions? | N/A |
Characteristics of studies | 8. Did the review authors describe the included studies in adequate detail? | 3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results? |
Quality assessment | 9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? | 3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria? 3.5 Were efforts made to minimise error in risk of bias assessment? |
Synthesis of the findings | N/A N/A 11. If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results? 12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? | 4.1 Did the synthesis include all studies that it should? 4.2 Were all pre-defined analyses reported or departures explained? 4.3 Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies? 4.6 Were biases in primary studies minimal or addressed in the synthesis? |
Heterogeneity | 14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? 15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? | 4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis? 4.5 Were the findings robust, e.g., as demonstrated through funnel plot or sensitivity analyses? |
Interpretation | 13. Did the review authors account for RoB in individual studies when interpreting/ discussing the results of the review? | A. Did the interpretation of findings address all of the concerns identified in Domains 1 to 4? B. Was the relevance of identified studies to the review’s research question appropriately considered? C. Did the reviewers avoid emphasising results on the basis of their statistical significance? |
Conflict of interest | 10. Did the review authors report on the sources of funding for the studies included in the review? 16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? | N/A N/A |
Section 2: Comparison of the inter-rater reliability of the tools
AMSTAR-2
Author (date), CAM | 1. Were PICO components listed? | 2. Protocol reported? Any deviations justified? | 3. Study design justified? | 4. Comprehensive literature search? | 5. Was study selection performed in duplicate? | 6. Was data extraction performed in duplicate? | 7. List of excluded studies? Were these justified? | 8. Characteristics of studies provided in detail? |
---|---|---|---|---|---|---|---|---|
Fibromyalgia | ||||||||
Multiple cam therapies | ||||||||
Holdcraft 2003 [14] | No | No | No | No | No | No | No | No |
Baronowsky 2009 [15] | No | No | Yes | No | No | No | No | No |
Yes | No | No | No | Yes | No | No | No | |
De Silva 2010 [18] | No | No | No | No | Yes | Yes | No | No |
Homoeopathy | ||||||||
Perry 2010 [19] | Yes | No | No | PY | Yes | Yes | No | PY |
Boehm 2014 [20] | No | No | No | PY | Yes | Yes | No | Yes |
Chiropractic treatment | ||||||||
Ernst 2009 [21] | Yes | No | No | No | No | Yes | No | No |
Acupuncture | ||||||||
Mayhew and Ernst 2007 [22] | No | No | No | PY | No | Yes | No | PY |
Daya 2007 [23] | No | No | No | PY | No | No | No | PY |
Langhorst 2010 [24] | Yes | No | No | PY | Yes | Yes | No | Yes |
Martin-Sanchez 2009 [25] | Yes | No | No | PY | No | No | No | No |
Cao 2013 [26] | Yes | No | No | PY | yes | Yes | No | Yes |
Deare 2013 [13] | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes |
Yang 2014 [27] | Yes | No | No | PY | Yes | Yes | No | No |
Herbal medicines | ||||||||
de Souza Nascimento 2013 [28] | Yes | No | yes | No | No | Yes | No | PY |
Colic | ||||||||
Multiple cam therapies | ||||||||
Perry 2011 [29] | Yes | PY | No | PY | PY | Yes | No | Yes |
Bruyas-Bertholon 2012 [30] | No | No | No | No | No | No | No | PY |
Harb 2016 [31] | Yes | No | No | No | Yes | Yes | No | PY |
Gutierrez-Castrellon 2017 [32] | Yes | No | No | No | No | No | No | No |
Manipulation therapies | ||||||||
Dobson 2012 [33] | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes |
Gleberzon 2012 [34] | No | No | No | PY | No | Yes | No | PY |
Carnes 2017 [35] | No | PY | No | PY | Yes | Yes | No | PY |
Acupuncture | ||||||||
Skejeie 2018 [36] | Yes | PY | No | Yes | Yes | Yes | No | Yes |
Herbal medicines | ||||||||
Anheyer 2017 [37] | No | No | No | No | No | Yes | No | Yes |
Probiotics | ||||||||
Sung 2013 [38] | Yes | No | No | No | Yes | Yes | Yes | PY |
Anabrees 2013 [39] | Yes | PY | No | PY | No | Yes | No | Yes |
Urbanska 2014 [40] | Yes | No | No | PY | No | No | No | PY |
Xu 2015 [41] | No | No | No | PY | Yes | Yes | No | Yes |
Schreck Bird 2017 [42] | Yes | No | No | No | Yes | No | No | Yes |
Dryl 2018 [43] | Yes | No | No | No | No | No | No | PY |
Sung 2018 [44] | Yes | PY | No | PYb | No | No | No | No |
9. Risk of bias assessed? | 10. Sources of funding of included studies? | 11. Methods used to combine the findings of studies appropriate? Test on heterogeneity? | 12. If meta-analysis performed was RoB accounted for? | 13. Was RoB discussed in individual studies? | 14. Was there discussion of any heterogeneity observed in the results? | 15. If a quantitative synthesis, was publication bias investigated and discussed in relation to the results? | 16. Reviewers’ conflict of interests stated? | Confidence in the review |
---|---|---|---|---|---|---|---|---|
Fibromyalgia | ||||||||
Multiple cam therapies | ||||||||
PY | No | No MA | No MA | Yes | No MA | No | CL | |
No | No | No MA | No MA | Yes | No | No MA | No | CL |
Yes | No | No | No | No | No | No | Yes | CL |
No | No | No MA | No MA | Yes | Yes | No MA | Yes | CL |
Homoeopathy | ||||||||
Yes | No | No MA | No MA | Yes | Yes | No MA | Yes | CL |
Yes | No | Yes | No | No | Yes | No | Yes | CL |
Chiropractic treatment | ||||||||
No | No | No MA | No MA | Yes | No | No MA | Yes | CL |
Acupuncture | ||||||||
No | NO | No MA | No MA | Yes | Yes | No MA | Yes | CL |
PY | No | No MA | No MA | No | No | No MA | No | CL |
Yes | No | Yes | Yes | Yes | Yes | No | No | CL |
No | No | No | No | No | Yes | No | Yes | CL |
Yes | No | Yes | No | Yes | Yes | Yes | Yes | CL |
Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Low |
Yes | No | No | No | No | Noa | Yes | No | CL |
Herbal medicines | ||||||||
Yes | No | No MA | No MA | No | No | No MA | Yes | CL |
Colic | ||||||||
Multiple cam therapies | ||||||||
Yes | No | No MA | No MA | Yes | No | No MA | Yes | Low |
PY | No | No MA | No MA | No | No | No MA | No | CL |
Yes | No | Yes | Yes | Yes | No | Yes | Yes | CL |
No | No | No | No | Yes | No | Yes | No | CL |
Manipulation therapies | ||||||||
Yes | No | Yes | Yes | Yes | Noa | Yes | Yes | High |
No | No | No MA | No MA | Yes | No | No MA | No | CL |
Yes | No | No | No | No | No | No | No | CL |
Acupuncture | ||||||||
Yes | No | Yes | Yes | Yes | Yes | No | Nod | Low |
Herbal medicines | ||||||||
Yes | No | No MA | No MA | No | No | No MA | Yes | CL |
Probiotics | ||||||||
Yes | No | Yes | No | Yes | No | No | Yes | CL |
Yes | No | Yes | No | Yes | Yes | No | Yes | Low |
Yes | Yes | No | No | No | No | No | No | CL |
Yes | No | Yes | No | No | No | No | Yes | CL |
Yes | No | No | No | No | No | No | No | CL |
Yes | No | No | No | No | No | No | No | CL |
Yes | No | Yes | Yes | Yes | Yes | Yes | Noc | Low |
Question | Number of studies | Gwet’s AC1/Gwet’s AC2 | 95% CI |
---|---|---|---|
1 | 31 | 0.69 | 0.48, 0.91 |
2 | 31 | 0.93 | 0.85, 1.00 |
3 | 31 | 0.55 | 0.30, 0.80 |
4 | 31 | 0.66 | 0.51, 0.81 |
5 | 31 | 0.70 | 0.47, 0.94 |
6 | 31 | 0.60 | 0.35, 0.86 |
7 | 31 | 0.97 | 0.94, 1.00 |
8 | 31 | 0.39 | 0.21, 0.56 |
9 | 31 | 0.65 | 0.46, 0.84 |
10 | 31 | 0.84 | 0.67, 1.00 |
11 | 19 | 0.54 | 0.19, 0.89 |
12 | 19 | 0.40 | 0.05, 0.75 |
13 | 31 | 0.52 | 0.27, 0.78 |
14 | 31 | 0.19 | -0.08, 0.47 |
15 | 19 | 0.61 | 0.28, 0.94 |
16 | 31 | 0.34 | 0.06, 0.63 |
ROBIS
Fibromyalgia review | Phase 2 | Phase 3 | |||
1. Study eligibility criteria | 2. Identification and selection of studies | 3. Data collection and study appraisal | 4. Synthesis and findings | 5. Risk of bias in the review | |
Homoeopathy | |||||
1. Perry | Low | Low | Low | Unclear | Low |
2. Boehm | High | Low | Low | High | High |
Acupuncture | |||||
3. Mayhew | Low | High | High | Low | Low |
4. Daya | Low | High | High | Low | Low |
5. Langhorst | Low | High | High | Low | Low |
6. Martin-Sanchez | Low | High | High | High | High |
7. Cao | Low | High | Low | Low | Low |
8. Deare | Low | Low | Low | Low | Low |
9. Yang | Low | Low | High | High | High |
Chiropractic | |||||
10. Ernst | High | Unclear | High | Unclear | Unclear |
Herbal Medicine | |||||
11. Nascimento | Low | Low | Low | High | Low |
Multiple CAM reviews | |||||
12. Holdcraft | Low | Low | Low | High | Low |
13. Baronowsky | Low | Low | Unclear | High | Low |
14. Terhorst | Low | High | Low | High | High |
15. De Silva | High | High | High | Unclear | Low |
Colic review | Phase 2 | Phase 3 | |||
1. Study eligibility criteria | 2. Identification and selection of studies | 3. Data collection and study appraisal | 4. Synthesis and findings | 5. Risk of bias in the review | |
Multiple CAM therapies | |||||
1. Perry | Low | Unclear | Low | Low | Low |
2. Bruyas-Bertholon | High | High | Unclear | High | High |
3. Harb | High | High | Low | High | High |
4. Gutierrez-Castrellon | Unclear | High | High | High | High |
Manipulation therapies | |||||
5. Dobson | Low | Low | Low | Low | Low |
6. Gleberzon | High | High | Unclear | Unclear | High |
7. Carne | Low | Low | Low | High | Unclear |
Acupuncture | |||||
8. Skejeie | Low | Low | Low | Low | Unclear |
Herbal medicine | |||||
9. Anheyer | Unclear | High | Low | High | High |
Probiotics | |||||
10. Sung 2013 | Unclear | Low | Low | High | Unclear |
11. Anabrees | Low | Low | Low | High | Low |
12. Urbansk | Low | High | High | High | High |
13. Xu | Unclear | Low | Low | Unclear | Low |
14. Shreck Bird | High | High | Low | High | High |
15. Dryl | High | High | Unclear | High | High |
16. Sung 2018 | High | Unclear | Unclear | Unclear | Unclear |
ROBIS question | No. of studies | Gwet’s AC1/Gwet’s AC2 | 95% CI |
---|---|---|---|
Domain 1: study eligibility criteria | |||
1.1 | 30 | 0.62 | 0.38, 0.85 |
1.2 | 31 | 0.70 | 0.56, 0.84 |
1.3 | 31 | 0.69 | 0.56, 0.82 |
1.4 | 31 | 0.61 | 0.48, 0.74 |
1.5 | 31 | 0.56 | 0.37, 0.74 |
Domain 1 Concerns regarding specification of study eligibility criteria | 31 | 0.45 | 0.22, 0.67 |
Domain 2: identification and selection of studies | |||
2.1 | 31 | 0.53 | 0.41, 0.65 |
2.2 | 30 | 0.53 | 0.35, 0.71 |
2.3 | 31 | 0.62 | 0.47, 0.77 |
2.4 | 31 | 0.41 | 0.20, 0.62 |
2.5 | 29 | 0.59 | 0.30, 0.88 |
Domain 2 Concerns regarding methods used to identify and/or select studies | 31 | 0.36 | 0.17, 0.55 |
Domain 3: data collection and study appraisal | |||
3.1 | 29 | 0.88 | 0.68, 1.00 |
3.2 | 31 | 0.66 | 0.51, 0.82 |
3.3 | 31 | 0.65 | 0.51, 0.78 |
3.4 | 31 | 0.77 | 0.61, 0.93 |
3.5 | 30 | 0.73 | 0.48, 0.98 |
Domain 3 Concerns regarding methods used to collect data and appraise studies | 31 | 0.55 | 0.35, 0.76 |
Domain 4: synthesis and findings | |||
4.1 | 31 | 0.60 | 0.46, 0.74 |
4.2 | 29 | 0.48 | 0.28, 0.68 |
4.3 | 31 | 0.77 | 0.66, 0.88 |
4.4 | 31 | 0.18 | − 0.02, 0.37 |
4.5 | 30 | 0.22 | 0.02, 0.43 |
4.6 | 31 | 0.39 | 0.17, 0.62 |
Domain 4 Concerns regarding the synthesis and findings | 31 | 0.17 | − 0.03, 0.37 |
Risk of bias in the review | |||
A | 31 | 0.28 | 0.09, 0.47 |
B | 31 | 0.64 | 0.54, 0.75 |
C | 31 | 0.45 | 0.31, 0.60 |
ROB | 31 | 0.45 | 0.24, 0.66 |
Section 3: Usability of the tools
Rater 1 | Rater 2 | Rater 3 | ||||
---|---|---|---|---|---|---|
n | Mean (SD) | n | Mean (SD) | n | Mean (SD) | |
AMSTAR-2 | 14 | 13.0 (5.2) | 15 | 18.7 (6.6) | 16 | 11.1 (4.2) |
ROBIS | 9 | 14.1 (6.5) | 10 | 15.7 (5.3) | 15 | 43.3 (23.3) |