Background
Rapid review (RR) products are intended to synthesize available evidence and meet the time constraints of healthcare decision-makers [
1]. Systematic reviews (SRs) typically take many months, or even years, to produce [
2,
3], so they may not be completed in time to inform pressing decisions. As defined by the Cochrane Handbook [
2], necessary elements of a SR are as follows: clearly stated objectives, pre-defined eligibility criteria for studies, an explicit and reproducible methodology, a systematic search, an assessment of the validity of the findings of the included studies, and a systematic presentation of the characteristics and findings of the included studies [
2]. RRs are seen as an attractive alternative to SRs as they may use fewer resources and take less time. Health technology assessment (HTA) agencies have embraced RRs, and a 2012 survey reported that 70% of HTA agencies offer RR products alongside standard reviews [
4]. While the HTA community and producers such as the Canadian Agency for Drugs and Technologies in Health (CADTH) have conducted RRs for a long time, The Cochrane Collaboration and McMaster Health Forum have also recently initiated programmes to conduct rapid reviews (
https://www.cadth.ca/rapid-response-service;
http://innovations.cochrane.org/response;
http://mcmasterhealthforum.org/policymakers/rapid-response-program).
This analysis of review articles about RRs was conducted as part of a larger evaluation of RR products undertaken by the rapid reviews workgroup of the Agency for Healthcare Research and Quality’s Evidence-based Practice Centers (EPC) Program [
5]. The workgroup investigated existing RR products and their methods of production, guidance for RR producers, and any empirical evidence regarding the validity of RRs compared with standard SRs.
To yield evidence that will inform decisions by systematic reviewers to diversify their products by offering RRs, or to adopt efficiencies that may be demonstrated by RRs, the EPC adopted the following questions to guide their investigation:
1.
What are the definitions and characteristics of rapid review products produced by key organizations (for example, purpose, audience, timelines, personnel)?
2.
What methodological guidance exists for the conduct of rapid reviews? What trade-offs are incurred with different methodological approaches?
3.
What empiric evidence exists comparing the results of rapid reviews with systematic reviews?
The full report of the workgroup is available online [
5]. Part of the investigation involved conducting a literature search to address the questions above. As a result of that larger search, the workgroup identified publications that analysed a sample of RRs. This synthesis identifies and summarizes existing review articles about RRs and reports on what these review articles tell us about RRs. By surveying existing reviews, the workgroup aimed to identify commonalities and differences in RR methods and the benefits and drawbacks to undertaking RRs. Using reviews as a unit of analysis, this synthesis provided an overview of the current RR landscape and allowed for the identification of gaps in our understanding of RRs.
Methods
Librarians who were part of the workgroup conducted a systematic literature search of Ovid Medline, Ovid EBM Reviews, Cochrane Methodology Register, and the EPC Program’s Scientific Resource Center (SRC) Methods Library in October and November of 2013 (see
Appendix for the full search strategy). A Scopus citation reference search and a grey literature search were also performed. We searched on an inclusive range of terms (for example, rapid, mini, pragmatic, targeted, focused, and brief) to describe relevant products (for example, briefs, syntheses, reviews, and assessments) to obtain a broad collection of publications about RRs. An update search of Ovid Embase was also conducted in February 2015.
At the first level of screening of titles and abstracts, references were included if they discussed RR methods, described initiatives or programmes to produce RRs, or provided empiric evidence comparing RR methods to traditional systematic review methods. At the second level of screening, records were excluded if they did not describe rapid products within the healthcare field. An additional criterion was applied to exclude references that described mini-HTAs exclusively, as mini-HTAs are often checklists that guide decision-making and not a specific method to evaluate evidence. Titles and abstracts were screened by two investigators using ABSTRACKR software (
http://abstrackr.cebm.brown.edu), with disagreements resolved by a third investigator.
During the screening process, different groupings of articles emerged: papers describing RR methods; actual RR products; empiric data exploring differences in methods, results, and conclusions between RRs and standard SR methods; and review articles about RRs. This paper presents the analysis of the review articles about RRs.
Prior to full-text analysis at the second level of screening, review articles about RRs were excluded if they were not published in English and if the authors did not mention terms relating to short, focused reviews (rapid review, evidence inventory, evidence advisory, hotline response, and so on). One investigator completed second-level screening and data extraction of the review articles using 11 general questions to allow for comparison across reviews. The questions below were modified from Brassey [
6]:
1.
How many RRs were analysed by the review?
2.
What were the objectives of the review?
3.
How were RRs defined by the review?
4.
How long did it take to complete the RRs included in the review?
5.
What kinds of topics or questions were addressed by the RRs?
6.
What search methods were used by the RRs?
7.
What kinds of analyses (narrative, meta-analyses) were used in the RRs?
8.
Did the RRs included in the review analyse study quality or risk of bias?
9.
For what purposes were the RRs included in the review conducted?
10.
What conclusions did the RRs included in the review reach?
11.
Did the review reach a conclusion about how RRs differ from SRs or full HTAs?
A narrative synthesis of the review articles was conducted and focused on definitions of RRs, the questions addressed by RRs, the methods used to conduct the RRs, and the conclusions reached by RRs.
Discussion
This synthesis confirms what Ganann’s SR found previously: heterogeneous RRs are being produced in the absence of standards, particularly by HTA agencies [
15]. However, this synthesis of review articles about RRs allowed for the identification of common findings and conclusions that contributed to the recommendations included below.
Lacking a single definition, RRs are better understood as a spectrum of products: some use a different methodological approach compared to a standard SR, while others closely resemble a SR. Key questions pertinent to conducting reviews rapidly are the following: 1) what steps are eliminated or reduced in comparison to a standard SR and 2) what are the potential consequences of taking a different methodological approach? One review found a significant inverse association between the number of substandard or unclearly reported SR methods employed and the number of months between end search date and review publication [
16]. As the timelines shrank, the number of unclear or non-standard methods used increased. The methodological approach taken influences the time needed to complete the RR, and producers need to ascertain - in consultation with the individuals commissioning the RR - how urgently the results of the RR are needed and what level of methodological rigor is acceptable given the nature of the question.
There is a perceived trade-off between time and the comprehensiveness of the end product, but the implications of cutting methodological corners to produce a rapid result are unclear. Transparency in reporting should assist clients in reaching an appropriate level of confidence in RR conclusions, and more research is needed to understand how RRs are used by decision-makers. In terms of current timelines, a single week may be sufficient to produce a RR product to meet the needs of some clients, but for a RR that approaches the rigor of a full SR, clients should be prepared to wait 6 months or more for a completed product.
Question development requires particular attention by producers, but not all reviews found that RRs consistently used well-focused clinical questions. Clients and producers should agree on a clearly articulated question that can be answered by a RR product (for example, a single-intervention effectiveness question). Standard SRs or HTAs may be more appropriate than RRs for clients seeking answers to complex questions about multiple interventions and safety, social, policy, or ethical issues.
While universally accepted search methods for RRs were not described by the review articles, many found that RRs limited the number of databases included in the search and often reduced (or eliminated) hand-searching or grey literature searching. Database search results were constrained by language and date and often used study-type filters to focus on only the highest level of evidence available (that is, SRs, guidelines, or randomized controlled trials).
Producers should consider that a less comprehensive search may find studies that confirm what is already known from the literature and that limiting results to SRs or other pre-synthesized study types may be a suitable method to ensure a rapid product. However, client questions ought to dictate search restrictions. Narrow search parameters may not yield enough results for questions that have not been extensively investigated, so for questions about recent technological innovations or new interventions, more inclusive RR search strategies may be required. Conversely, exhaustive search methods may be entirely inappropriate for new or recent interventions when scant evidence exists and clients require a shortened timeframe for completion of the RR.
There was little consistency in the way quality of included studies was assessed in RRs. A majority of authors of included review articles concluded that QA was an essential element and that failing to use a QA tool could result in contradictory conclusions. In cases where the only available evidence is poor research, QA can impact how the evidence is interpreted and used to formulate conclusions when RR authors call attention to the weaknesses of the evidence. A lack of QA may lead to over-reliance on and misinterpretation of poor research, and producers should caution clients that summarizing or analysing data from studies without considering their methodological quality may misrepresent the evidence.
Review articles found that RRs commonly included a qualitative summary of included studies but seldom conducted meta-analyses. Tables were used to compare the findings and methods of studies and may prove useful to RR clients who prefer a snapshot view of the results. Any RR analysis should make transparent the limitations of the end product based on the methodological approaches taken, particularly if no QA was conducted. Conclusions may also be less generalizable than full SRs and only applicable to the healthcare organization that commissioned the RRs.
Future research is needed to determine how, or if, the results and conclusions of RRs differ from those reached by standard SRs or how RRs differ in quality given an evidence base of comparable size. In addition, concrete guidance for conducting and reporting RRs is needed from methods experts in conducting evidence syntheses. Heterogeneous methods of production suggest that multiple products fall under the umbrella term ‘rapid review.’ Clear definitions for the range of RR products will help inform decisions to undertake a RR. The Agency for Healthcare Research and Quality (AHRQ) EPC workgroup on RRs recently developed a classification system of RR types [
5] to assist producers and clients in selecting the best product for their information needs. Future research is also needed to confirm the implications for RRs of adapting their approach to produce a more rapid result.
Strengths of this analysis include the comprehensiveness of the search and the first-level screening by two independent reviewers. A limitation of this study was the use of a single reviewer for extracting and analysing review articles about RRs. Also, given current interest in rapid reviews, it was a limitation of the research that the search was not completely updated following analysis and manuscript preparation. Reviews published since November 2013, when the original search was conducted, have not been included in this analysis. Additional limitations include a potential overlap in the RRs described in each review article, the possibility that variability in RRs is a result of maturation of the review method, and the heterogeneity of review articles analysed.
Competing interests
The authors declare that they have no competing interests. As producers of evidence syntheses, however, the authors may benefit from the development of standards to inform assessment and production of RRs.
Authors’ contributions
RF conducted the full-text screening and analysis, the update search of Embase, and the cited reference search. RAP conducted the database literature searching and designed the search strategy. MF created the results table. LH, DMD, JG, MDM, KAR, and CU performed abstract screening. All authors were members of the working group and contributed to the design of the study and interpretation of results. All authors helped to draft the manuscript and read and approved the final manuscript.