A scoping review of rapid review methods

A formal definition for a rapid review does not exist. As such, we used the following working definition, ‘a rapid review is a type of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a short period of time’ [2].

A scoping review protocol was compiled using guidance from Arksey and O’Malley [8], and revised upon feedback received from the Canadian Institutes of Health Research peer review panel. It is available from the corresponding author upon request.

To identify potentially relevant studies for inclusion, the following electronic databases were searched: MEDLINE; EMBASE; and the Cochrane Library. Since two systematic reviews have already been published on rapid reviews [6, 7], we limited our search from 2008 until May 2013. An experienced librarian (LP) drafted the literature searches based on the previous reviews, which was refined through team discussion. The MEDLINE search strategy is presented in Additional file 1: Appendix 1 and the other searches are available from the corresponding author upon request.

Our literature search was supplemented by targeted internet searches for unpublished rapid review reports posted on the websites of producers of rapid reviews. For this search, we took a random 10 % sample of the unpublished rapid reviews available on the producers’ websites. Often only the title was available for the rapid reviews, so, we focused inclusion to the full rapid review, if available. The reference lists of relevant reviews were scanned [6, 7], as were the reference lists of all included rapid reviews.

Articles, papers, books, and reports were included if they evaluated, compared, used or described a rapid review according to the authors.

The screening criteria were established a priori (as outlined in our protocol) and calibrated amongst the team through a series of pilot tests. After >90 % agreement was observed, pairs of reviewers screened the literature search results independently, and discrepancies were resolved through discussion. All screening was performed using our online tool, synthesi.sr [9].

Data items included study characteristics (for example, first author, year of publication), terminology used to describe the rapid review, full citation of previous methods papers that were used to guide the rapid review design, timeframe (in months) for completing the rapid review, and operationalized steps of the rapid review, if reported. The rapid review type was categorized as an application (for example, a rapid review report), development (paper attempts to further refine the rapid review method), impact (examines the impact of rapid reviews) or comparison (compares the results of a rapid review to a systematic review). We abstracted the assessment of the rapid review approach, including accuracy of results, comprehensiveness, potential for risk of bias, timeliness, cost-effectiveness, and feasibility as reported by the publication authors. We also abstracted the skills or knowledge required to conduct the rapid review as reported by the authors.

To synthesize the descriptive results, we conducted qualitative analysis using NVivo 10 [10]. Content analysis was conducted by one team member (WZ) and verified by another team member (ACT) to synthesize common methodologies used across the included rapid reviews using a framework. The framework was developed by the review team and presented in Additional file 1: Appendix 2. The framework focused on the following steps for a rapid review: literature search (number of databases and grey literature); inclusion criteria (limited by date, language, and study design); screening (title/abstract and full-text); data abstraction; risk of bias/quality appraisal; and data synthesis. In order to depict the frequency of the terms used to describe the rapid reviews, a word cloud was created using Wordle, which is software that generates ‘word clouds’ from text that the user provides and places more emphasis on words that occur with greater frequency [11].

A total of 3,397 citations and 262 potentially relevant full-text papers were screened. Subsequently, 100 articles [2, 12‐110] plus one companion report [111] fulfilled the eligibility criteria and were included [31] (Fig. 1). Forty-seven of the included papers were unpublished rapid reviews posted on websites [13, 24, 29, 31‐36, 39, 45, 47, 50, 52‐57, 62, 63, 66, 68, 70, 73‐75, 77, 81‐83, 86‐94, 99, 100, 103, 104, 107, 109, 112].

The rapid reviews were published between 1997 and 2013, and 58 were conducted in Europe, while 20 were conducted in North America (Table 1, Additional file 1: Appendix 3). The type of articles included 84 application papers (two did not report any methods), seven development papers, six impact papers, and four comparison papers; one article [20] was categorized in two categories. Ten of the rapid reviews were reported in 5 pages or less, suggesting that they were brief reports or research letters. Most of the articles (73 %) did not report the duration of conduct for the rapid review. For the minority that reported this, the duration ranged from less than 1 month to 12 months, and 18 were between 1 and 6 months. For the application articles, 74 % examined interventions, 12 % charted the frequency of literature (for example, regarding outcomes or frameworks), 5 % examined associations between exposure and disease, 5 % assessed diagnosis or screening techniques, and 2 % examined the patient experience or barriers/facilitators.

Sixty-five articles assessed rapid review characteristics (Table 2) [2, 12, 14‐22, 24, 26‐30, 32, 37‐39, 41‐43, 45‐49, 51‐59, 61, 63, 64, 66, 69, 72‐76, 78‐80, 84, 86, 88‐94, 100, 103‐105, 110]. Sixty percent of the authors reported that the report was timely, 29 % believed that the method had potential risk of bias, 23 % deemed that the approach was accurate compared to a full systematic review, 8 % believed the approach was comprehensive, 5 % reported that the approach was cost-effective, and 6 % believed it was a feasible approach.

The most frequent term used to describe the rapid review approaches was ‘rapid review’, used in 34 of the included articles (Fig. 2). This was followed by ‘rapid evidence assessment’, which was used in 11 papers, ‘rapid systematic review’ in ten papers, and ‘health technology assessment’ or ‘rapid health technology assessment’ in six papers. All of the other terms occurred two times or less.

Twenty-six [2, 12, 13, 17, 20‐22, 27, 28, 30, 40, 42‐44, 48, 49, 61, 76, 78‐80, 84, 88, 103, 105, 110] articles provided citations of previous methods papers that were used to guide the rapid review method (Fig. 3, Additional file 1: Appendix 4). The citations were Ganann and colleagues [6] (cited in eight papers), Watt and colleagues [7, 111] (cited in seven papers), a Civil Service paper [113] (cited in four papers), Ehlers and colleagues [114] (cited in one paper), Armitage and colleagues [14] (cited in one paper), and Grant and colleagues [115] (cited in one paper).

Thirteen [16, 32, 39, 42, 46, 48, 49, 52, 79, 84, 88, 90, 94] of the included papers reported the skills and knowledge required to conduct the rapid reviews (Table 3). These were content experts in seven articles [16, 32, 42, 48, 49, 79, 90], information specialists in five articles [39, 49, 52, 84, 88], systematic review methodologists in four papers [16, 42, 48, 79], staff experienced in conducting reviews in four papers [46, 48, 49, 84], and knowledge users in three papers [32, 79, 94].

The 84 rapid review applications were categorized using our framework (Additional file 1: Appendix 2) and 50 unique methods were observed. Of these, only 16 occurred more than once; three approaches occurred five times [21, 36, 40, 44, 45, 47, 53, 54, 56, 57, 65, 75, 83, 91, 92], another two occurred four times [18, 37, 39, 64, 86, 93, 99, 107], three approaches were used three times [49, 51, 58, 61, 62, 69, 73, 76, 81], and eight approaches occurred two times [14, 16, 20, 25, 27, 30, 31, 66‐68, 70, 79, 82, 96, 100, 104]. The characteristics of the rapid review approaches that occurred more than four times were analyzed (Table 4). Rapid Approach 1 had the most details reported, with 5/5 papers mentioning that it was accurate and timely (but did not report the amount of time it took to conduct their rapid review), and had limited comprehensiveness.

Many of the steps used in the rapid reviews were not fully reported (Table 5, Additional file 1: Appendix 5). For example, 40 % (33/82) did not report whether reference lists were scanned and 67 % (55/82) did not report whether authors were contacted to obtain further material or information.

Streamlined methods that were used in the 82 rapid reviews included limiting the literature search to published literature (24 %) or one database (2 %), limiting inclusion criteria by date (68 %) or language (49 %), having one person screen and another verify or screen excluded studies (6 %), having one person abstract data and another verify (23 %), not conducting risk of bias/quality appraisal (7 %) or having only one reviewer conduct the quality appraisal (7 %), and presenting results as a narrative summary (78 %) (Fig. 4).

Four studies were comparisons, providing details on differences in results between rapid reviews and systematic reviews [20, 31, 34, 106]. Cameron and colleagues identified rapid reviews from health technology assessment (HTA) organization websites and then conducted a literature search to identify systematic reviews on the same topic [31]. Eight rapid review products were identified on four different topics. However, the authors did not appraise the methodological quality of the systematic reviews, so it is unclear whether shortcuts were also taken in the included systematic reviews. The authors noted that the conclusions did not differ substantially between the rapid and systematic reviews. Corabian and colleagues compared six rapid review products (called ‘technotes’) with their final peer-reviewed publications [34]. The authors found that the conclusions differed only in 1/6 cases. Van de Velde and colleagues compared the results from their rapid review to a systematic review that was conducted by another group and published on the same topic [106]. Despite having literature searches that were conducted for the same dates, conflicting results were observed; the rapid review concluded that potato peel was effective for burns, while the systematic review concluded that potato peel was not effective for treating burns. Finally, Best and colleagues noted that two of the rapid reviews they conducted were in agreement with systematic reviews published at a later point in time on the same topic [20].

Seven papers proposed methods to refine the rapid review approach [2, 12, 16, 20, 46, 79, 80]. Best and colleagues (1997) described their experience conducting 63 rapid reviews for decision-making beginning in 1991, through the Development and Evaluation Committee in the UK [20]. Abrami and colleagues (2010) described ways to produce brief reviews efficiently, and presented a checklist for the conduct and reporting of brief reviews [12]. Bambra and colleagues (2010) described their experience conducting nine rapid reviews for the Secretary of State for Health [16]. Jahangirian and colleagues (2011) described their experience conducting five rapid reviews for the Research into Global Healthcare Tools consortium and proposed a framework for the conduct of rapid reviews [46]. Khangura and colleagues (2012) described their approach to the conduct of 11 rapid reviews through the collaboration between the Ottawa Hospital Research Institute and the Champlain Local Health Integrated Network [2]. Thigpen and colleagues (2012) described their experience conducting rapid reviews using the 6-step Prevention Synthesis and Translation System process for the Division of Violence Prevention, National Center for Injury Prevention and Control at the Centers for Disease Control and Prevention [79]. Thomas and colleagues (2013) described their experience of conducting two rapid reviews for the UK Treasury to inform the 2006/07 Comprehensive Spending Review [80].

Guidance to streamline the rapid review process varied, yet some consistencies were observed (Table 6). For example, four papers suggested using integrated knowledge translation, in which researchers work closely with the knowledge users to complete the rapid review [2, 16, 19, 79]. Four papers suggested the use of a research question with a limited scope [12, 16, 80, 110]. Seven publications recommended streamlining the literature search [2, 12, 16, 46, 79, 80, 110] and three suggested restricting the eligibility criteria [2, 12, 80]. Two papers provided suggestions for efficiently appraising risk of bias [2, 80] and none suggested conducting a meta-analysis as part of the rapid review.

Six papers examined the impact of rapid reviews on decision-making [41‐43, 60, 85, 110]. Hailey and colleagues (2000) examined the impact of 20 rapid review products [43] and found that 14 had an influence on policy decision-making, four provided guidance, and two had no perceived impact. McGregor and Brophy (2005) evaluated the success of the conduct of 16 rapid reviews for a hospital rapid review service [60]. The results of all 16 products were directly implemented in the hospital, saving approximately $3 million per year. Hailey (2006) wrote a paper summarizing the impact of HTA in general, as well as related to rapid HTA. Overall, it was concluded that these reports can influence decision-making. Hailey (2009) conducted a survey of HTA organizations to examine the use of rapid reviews for decision-making [42]. Fifteen rapid review products were included; all influenced a decision, including using the rapid review for reference material (67 %) and directly using the rapid review’s conclusions for the decision (53 %). Zechmeister (2012) examined the impact of 58 rapid assessments and observed that 56 of these products were directly used for reimbursement decisions and two were used for disinvestment decisions [85]. Finally, Batten (2012) wrote an editorial discussing how rapid reviews can be used by school nurses [110].