Background
Methods
Locate existing SR(s) |
---|
A defined and reproducible approach to efficiently identify existing systematic reviews for possible use in conducting a newly proposed systematic review, including updates. |
Assess relevance
|
Methods by which existing systematic reviews identified in Step 1 can be evaluated as to whether they are similar enough to the newly proposed review to obviate the need for conducting one or several steps in undertaking the newly proposed review. Relevance evaluation considers how well the existing reviews’ research questions and inclusion/exclusion criteria for population, interventions, comparators, outcomes, settings, and study designs match those of the new systematic review, and how recently existing reviews’ literature searches were concluded. Use ‘almost’ relevant SR(s) when selecting, developing, and/or refining questions and providing context for a newly proposed systematic review (that is, Contextual Use) and scan references to check new search results. |
Assess quality of existing SR(s)
|
Determine appropriate use and incorporate existing SR(s)
|
Methods by which to determine appropriate uses for relevant, high-quality existing systematic reviews in the proposed review. Incorporate existing SR(s) or use information from existing systematic reviews to supplement or supplant one or more activities that would be conducted from scratch for the newly proposed review. Use of existing systematic reviews may include: 1) using the existing systematic review(s)’ listing of included studies as a quality check for the literature search and screening strategy conducted for the new review (Scan References); 2) using the existing systematic review(s) to completely or partially provide the body of included studies for one or more key questions in the new review (Use Existing Search); 3) using the data abstraction, risk of bias assessments, and/or analyses from existing systematic reviews for one or more key questions in the new review (Use Data Abstraction/Syntheses); or 4) using the existing systematic review(s), including conclusions, to fully or partially answer one or more key questions in the new review (Use Complete Review). |
Report methods and results from using existing SR(s)
|
Standards for reporting the rationale for incorporating existing systematic reviews and the methods by which specific existing systematic reviews were located, assessed, and incorporated into the current systematic review. Standards for reporting results in the current systematic review that rely on evidence and/or analysis from existing systematic reviews and caveats or limitations associated with that approach; guidance about discussing how findings of the current systematic review compare and contrast with those of existing relevant systematic reviews. |
Guidance summary
Evidence-based practice center discussions
Assessment of areas of need
Results
Guidance summary
Guidance areas | |||||||||
---|---|---|---|---|---|---|---|---|---|
Locating and defining appropriate use | Assessing relevance | Assessing review quality | Determining use: Scanning references | Determining use: Search strategy | Determining use: Risk of bias assessment | Determining use: Data abstraction | Determining use: Synthesis | Methods/Results reporting | |
AHRQ Evidence-based Practice Center Program (EPC program)
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Methods Guide for Effectiveness and Comparative Effectiveness Reviews[2] | |||||||||
Reporting the Findings of Updated Systematic Reviews of Comparative Effectiveness[6] | ✓ | ||||||||
Canadian Agency for Drugs and Technology in Health (CADTH)
| |||||||||
Guidelines for Authors of CADTH Health Technology Assessment Reports[7] | |||||||||
Cochrane Collaboration
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Cochrane Handbook for Systematic Reviews of Interventions[8] | |||||||||
Danish Centre for Health Technology Assessment (DACEHTA)
| ✓ | ✓ | ✓ | ||||||
Health Technology Assessment Handbook[9] | |||||||||
European Collaboration for Health Technology Assessment (ECHTA)
| ✓ | ✓ | ✓ | ||||||
Best Practice in Undertaking and Reporting Health Technology Assessments[10] | |||||||||
Health Technology Assessment International
| |||||||||
Institute of Medicine
| |||||||||
Finding What Works in Health Care: Standards for Systematic Reviews[11] | |||||||||
Institute for Quality and Efficiency in Health Care (IQWiG)
| ✓ | ✓ | ✓ | ||||||
General Methods[12] | |||||||||
National Institute for Health and Care Excellence (NICE)
| ✓ | ✓ | ✓ | ✓ | |||||
The Guidelines Manual, Draft for Consultation[13] | |||||||||
York Centre for Reviews and Dissemination: Systematic Reviews (CRD)
| ✓ | ✓ | ✓ | ||||||
CRD’s Guidance for Undertaking Reviews in Healthcare[14] |
Evidence-based practice center discussions
-
EPCs most commonly used existing reviews as a source of relevant literature and as context for the introduction or discussion section of reviews.
-
Existing reviews were most useful when key questions and/or PICOTS-SD (population, intervention, comparator, outcome, time frame, setting, and study design) matched or when they addressed a specific subquestion of the new review.
-
Using existing reviews was often more resource intensive than completing a review from scratch.
-
EPCs expressed that they often did not trust aspects of reviews conducted by others.
-
When relevant and rigorous, incorporating prior reviews into the review being undertaken by the EPC was clearly valuable in at least two instances: 1) allowing larger scope of the review being undertaken without additional resources, or 2) providing summarized evidence when a new in depth review of primary literature would not be feasible (for example, existing reviews provide individual patient data analysis or include hundreds of trials, supplemented by author-provided data).
Discussion
Methodological areas: assessment of areas of need
Locating existing systematic reviews
Available guidance
AHRQ Evidence-based Practice Center Program (EPC program) | Cochrane collaboration | Danish Centre for Health Technology Assessment (DACEHTA) | ||
---|---|---|---|---|
Locating | Two strategies are recommended for identifying existing systematic reviews for a CER. The first strategy is to perform a targeted search of a higher yield database, which includes output from the Evidence-based Practice Center program, MEDLINE’s Top 120 Index Medicus Journals, Health Technology Assessments, Cochrane Database of Systematic Reviews and Database of Abstracts and Reviews of Effects. The second strategy is to identify systematic reviews during a broad de novo literature search. | Systematic reviews can be located through CDSR, DARE and HTA database. MEDLINE and EMBASE can also be used to search for systematic reviews. In MEDLINE, most review articles can be found under the publication Term ‘Meta-analysis’ and in EMBASE, the thesaurus term ‘Systematic Review’ can be used. Specific search strategies can be used to identify systematic reviews in MEDLINE and EMBASE. Additionally, systematic reviews can be identified through search services such as Turning Research into Practice (TRIP). | Secondary studies (for example, systematic reviews, HTA reports, and clinical guidelines) should be located to determine if key questions have already been answered. Secondary studies can be identified through several databases (for example, The HTA Database, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Guidelines International Network, National Guidelines Clearinghouse, Health Evidence Network, National Electronic Library for Health: Guidelines Finder, and Turning Research Into Practice). | |
In an Overview, primarily only Cochrane Intervention reviews should be included, but other reviews may be included occasionally | ||||
Assessing Relevance | An existing systematic review should be used with the intent to answer parts or all of specific key questions. PICOTS-SD must be considered for relevance of existing systematic reviews. Reviews that are partially relevant may be useful for background or checking references. An initial screening for relevance should be performed, considering the timeliness of the review’s literature search. It is recommended to bridge any search date that ended more than one year from the time the systematic review is identified. If a review is outdated but still desired to be used, an update of the search should be done. | In an Overview, included reviews should be assessed using specific criteria. Considerations include whether a review is up-to-date and if there are specific limitations for the objectives of the Overview. | All evidence should be assessed for relevance to the topic. Identified articles should be compared to the focused question to determine if the article may answer the focus question. The literature can be divided into two groups (secondary studies and primary studies). If a large amount of evidence has been identified, it can be subdivided into groups based on presumed quality. The hierarchy of evidence is: 1) meta-analyses and systematic reviews (among others Cochrane reviews); 2) randomized controlled trials (RCTs); 3) non-randomized controlled trials; 4) cohort studies; 5) case-control studies; 6) descriptive studies, limited series; and 7) position papers, non-systematic reviews, leading articles, expert opinions. | |
In the second stage of screening, the review’s PICOTS-SD elements should be compared to those in the new review protocol for relevance. If these elements are poorly reported, the review should not consider including the existing review. | ||||
Assessing Review Quality | Only reviews of high quality should be included in a CER. Two independent reviewers should assess for quality and methods for resolving discrepancies should be reported. A quality rating instrument should be used to addresses all aspects of the review that will be incorporated into the CER. Both the methods used to minimize bias and the reporting should be assessed. QUOROM (PRISMA) is a checklist that can be used to assess the reporting of systematic reviews. As a common starting point, the AMSTAR tool should be used to assess the quality of reviews. Reproducibility and application of inclusion and exclusion criteria should be confirmed. As some limitation to AMSTAR exists, it is recommended to describe implications of potential methodological flaws instead of relying on numerical scores. | Generally, selection criteria for a Cochrane Overview limits included reviews to Cochrane reviews. Non-Cochrane systematic reviews may be included if there are good quality reviews for which a Cochrane review is not available. | No guidance | |
Determining use: Scanning References | The list of included articles from an existing review can be used in a CER if methods for identifying articles are of adequate quality. | Existing systematic reviews can be used as sources of relevant studies. References lists of systematic reviews can be searched to identify relevant articles. This should be done as an adjunct to other search methods as bias may be present in what studies were included in existing reviews. | No guidance | |
Determining use: Use Search | Part or all of the search strategy may be used from an existing review if it is consistent with EPC program methods for finding evidence. A search strategy from an existing review can be used, followed by de novo analysis and synthesis of data. | Existing reviews may be a useful source information about search strategies | No guidance | |
Determining use: Risk of Bias Assessment | In order to use the risk of bias assessment from a systematic review, the methods used must be consistent with the EPC program methods guide. These methods include selection of design specific criteria for risk of bias assessment and use of appropriate tools. | In an Overview, an assessment of the quality of evidence should be done. If no assessment was done in an included systematic review, authors should perform the assessment. If a quality assessment was done in an included systematic review, authors should assess the judgments and ensure consistency between included reviews. | A quality assessment tool should be used to uniformly assess the quality of identified articles. Check list tools developed by different national centers (for example, SIGN, NICE, GRADE and Centre for Evidence-based Medicine, Oxford) can be used to assess quality. | |
Determining use: Data Abstraction | The data extraction tables may be used from an existing review, if they are deemed to be of adequate quality. However, if results of individual trials are not reported, the use of summary findings from an existing review may compromise transparency for a CER and this is not recommended. | In an update, data from new studies should be abstracted and included, if applicable. | No guidance | |
In an Overview, if necessary, authors may seek additional data or information from the authors of primary studies from included systematic reviews. | ||||
Use Synthesis | If an existing review is very similar to a CER in research questions and is high quality, the entirety or portions of the existing review may be incorporated. Summarized evidence for specific populations or interventions may be included in a CER. If summarized evidence is to be included, the existing review must have methods consistent with EPC program methods for finding evidence, assessing quality, grading the strength of evidence and other principles including conflicts of interest. Summarized evidence can also be incorporated with a de novo sensitivity analysis. | In an update, data collected from new studies should be included and a new meta-analysis should be done. | No guidance | |
If multiple high quality reviews are found, a single review can be chosen, which is most relevant and least biased, or a meta-review can be performed. | In an Overview, authors should reply on previous analyses when possible. If there are differences between reviews (for example, different populations or subgroups are analyzed), data may need to be reanalyzed. | |||
If more than one high quality review is found with discordant findings, it may be an indication to start a de novo review on that key question. | ||||
Report Methods/Results | It is recommended to provide a summary table to show where existing review(s) were used to replace de novo processes. Summary tables of existing systematic reviews should be included to compare the reviews and should address any overlap (or lack thereof) in the primary research included in reviews. | In an update, revision to text of the existing review will depend on the influence of the new data and results. If there is no change in the results, little revision to the text is required. However, some updates may require a change to the conclusion of a review which will require much modification of the text. It should be noted in the Abstract and Background that this is an update. A ‘What’s new’ table should be completed and changes should be made to ensure no dates or other information is out of date. | No guidance | |
The discussion section should include a justification for using an existing systematic review and address any limitations. It is also important to compare findings from the CER with the findings from existing reviews. | ||||
In an update, it is important to show explicitly what has changed from the previous report. The desired depth of information varies between users of reports. Review updates may be effectively presented in an executive summary with tables and figures, identifying and modifications followed by a full report for users who require further depth of information. | ||||
European Collaboration Health Technology Assessment (ECHTA)
|
Institute for Quality and Efficiency in Health Care (IQWiG)
|
National Institute for Health and Clinical Excellence (NICE)
|
York Centre for Reviews and Dissemination: Systematic Reviews (CRD)
| |
Locating | In order to determine if key questions have already been answered, a search for previous HTA reports should be conducted. The search for previous reports should be systematic and well documented. | Different databases should be considered in locating systematic reviews that are used for primary literature. Databases that exclusively or mostly hold systematic reviews should be searched as well as biomedical databases (for example, MEDLINE and EMBASE), which also hold systematic reviews. | Core and subject databases, including MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Cochrane Central Register of Controlled Trials, and Health Technology Assessment (HTA) database, should be searched for every review. For questions on effectiveness, a search should be done for systematic reviews, followed by randomized controlled trials, then cohort or case–control studies. Search filters are available to assist in identifying studies, including a search filter for systematic reviews. | To check if key questions have been answered by existing or ongoing reviews, a search for systematic reviews should be conducted. The Database of Abstracts of Reviews of Effects (DARE), and the Cochrane Database of Systematic Reviews (CDSR) should be searched. Additionally, NICE and NIHR HTA program Websites can be searched along with the Campbell Library of Systematic Reviews and the Evidence for Policy and Practice Information Centre’s Database of Systematic and Non Systematic Reviews of Public Health Information (DoPHER). Guideline groups including NGC and SIGN can be searched for guidelines based on systematic review evidence. MEDLINE and other databases can be searched for previous reviews. |
Evidence scanning should be done continuously to identify systematic reviews that concern published or developing information products. Two people should regularly screen (CDSR, DARE, INAHTA, MORE and PubMed). Identified reviews that concern a product of the Institute can influence the updating process, including triggering an update or modifying the updating plan. | ||||
Assessing Relevance | All identified evidence should be assessed with pre-defined inclusion and exclusion criteria. Selection criteria should be developed from background information, research questions and the availability of evidence and should be defined prospectively to avoid bias in selection of evidence. Inclusion and exclusion criteria should cover patient characteristics, condition characteristics, technology aspects, methodological issues, outcomes measured, publication type. | No guidance | No guidance | No guidance |
Assessing Review Quality | No guidance | For systematic reviews to be used in a benefit assessment, they must be assessed for sufficient quality. They must ‘show only a minimum risk of bias; present the evidence base in a complete, transparent and reproducible manner; and thus allow clear conclusions to be drawn’. The searches conducted in the systematic reviews must not contradict the methodology of the Institute. Quality assessment should be done with Oxman and Guyatt’s quality index for systematic reviews or AMSTAR. Sponsors and authors’ conflicts of interests should be documented and discussed for systematic reviews. | Guidelines may contain reviews of evidence that are applicable to questions formulated by the guideline development group. These may be used as evidence if: ‘they are assessed using the appropriate methodology checklist from this manual and are judged to be of high quality, they are accompanied by an evidence statement and evidence table(s), the evidence is updated according to the methodology for the exceptional update of NICE clinical guidelines’. | Identified reviews should be assessed for quality. Quality reviews should have a well-defined question, comprehensive search, clear and appropriate selection or studies, unbiased processes for assessing study quality and extracting and synthesizing data. Checklists can be used to help in assessing quality of systematic reviews (for example, Oxman and Guyatt, Guidelines for reading literature reviews). |
If more than one systematic review of adequate quality is found to address a particular subject, additional quality assessment should be done. Items to compare include: content of the review, search strategy and date, sensitivity analysis, how bias is assessed and dealt with, and updating provisions. | ||||
Determining use: Scanning References | No guidance | No guidance | No guidance | Other sources of literature include references lists of systematic reviews. References lists of existing reviews can be scanned to identify additional studies. |
Determining use: Search | No guidance | No guidance | No guidance | No guidance |
Determining use: Risk of Bias Assessment | All evidence should be critically assessed for quality. Checklists can be used for appraisal of medical literature. All sources of information should be appraised for validity. No guidelines exist for assessing quality of sources of information other than medical literature, and this is a gap that future guidance needs to address. | No guidance | No guidance | No guidance |
Determining use: Data Abstraction | No guidance | No guidance | If using reviews of evidence published in other guidelines, the guideline development group should create new evidence summaries or statements. The original evidence tables should be referenced with a direct link to the source if possible or a reference to the published document. Verbatim quotes of recommendations from other guidelines should not be used, unless the recommendations come from NHS policy or legislation. | No guidance |
Determining use: Synthesis | No guidance | No guidance | No guidance | No guidance |
Report Methods/ Results | No guidance | Results of systematic reviews should be summarized in tables if possible. If discordant results on the same outcome are found, possible explanations should be given. If it appears that a new benefit assessment based on primary studies would produce different results, a new assessment should be done. | Original evidence tables from published guidelines should be referenced with a direct link to the source if possible or a reference to the published document. Verbatim quotes of recommendations from other guidelines should not be used, unless the recommendations come from NHS policy or legislation. | No guidance |
Evidence-based practice center discussions
Assessment
Assessing relevance
Available guidance
Evidence-based practice center discussions
Assessment
Assessing review quality
Available guidance
Evidence-based practice center discussions
Assessment
Determining appropriate use for relevant existing systematic reviews: scanning references
Available guidance
Evidence-based Practice Center Discussions
Assessment
Determining use: search strategy and results of existing searches
Available guidance
Evidence-based practice center discussions
Assessment
Determining use: data abstraction
Available guidance
Evidence-based practice center discussions
Assessment
Determining use: study-level risk of bias assessments
Available guidance
Evidence-based practice center discussions
Assessment
Determining use: synthesis, including strength of evidence assessment
Available guidance
Evidence-based practice center discussions
Assessment
Methods and results reporting
Available guidance
Evidence-based practice center discussions
Assessment
Conclusions
Methodological area | Guidance status | Recommendations/Further work needed |
---|---|---|
Locating existing reviews | Current guidance is time intensive | Narrow search to locate only highly relevant, well-done, very recent, existing reviews; consider narrowing further to specific sources like EPC or Cochrane. Empiric study of comprehensive versus limited consideration of specific sources may be warranted. Consider how much documentation of search strategies and yields is required for transparency. |
Assessing relevance | Guidance exists | Follow current guidance |
Assessing review quality | Current tools, such as AMSTAR, have limitations and none consider primary literature included in the reviews | Empiric evidence of quality rating approaches is needed. Consider which currently available (or soon to be available) tools best fit the EPC program’s needs. |
Determining use | ||
Scanning references | Guidance exists | Follow current guidance |
Search strategy/results of existing searches | Guidance exists | Empiric evidence for considering searches from >1 review and considering excluded studies is needed |
Data abstraction | Current guidance is limited | Guidance needed for specific scenarios and for confirming accuracy of abstracted data |
Study-level risk of bias assessments | Guidance available for primary studies | Guidance is needed for when to accept or repeat assessments from existing reviews |
Synthesis | Current guidance is limited | Guidance needed for specific scenarios and for assessing strength of evidence when integrating existing review |
Methods and results reporting | Guidance exists | Guidance needed on level of detail necessary for all aspects, and options with worked examples needed for evidence tables |