A time-responsive tool for informing policy making: rapid realist review

Efforts to improve the connections between research evidence and policy decisions have resulted in a number of sophisticated synthesis tools [1]. Pawson has broadly classified evidence synthesis approaches as being meta-analytical (re-analysing data from numerous studies to arrive at broadly generalizable findings) or narrative (using text-based data extractions to compare study findings as a way for understanding why programs or interventions have certain effects), both of which are commonly employed for informing policy decisions [2]. While Pawson notes the relative advantages of meta-analytical and narrative approaches, each technique also has key limitations: meta-analyses fail to provide understanding of how programs work, while narrative reviews give little insight into how a program may operate in different settings or under different circumstances [3, 4]. Limited generalizability of study results for a given context has been addressed as a common concern in evidence-informed public health policy [5, 6]. Alternatively, we have realist reviews, which are an approach to synthesising evidence first promoted by Pawson [4].

Pawson notes that the underlying principles of realist approaches are the links between interventions (I), contexts (C), mechanisms (M), and outcomes (O) [4]. In contrast to specific activities included as part of an intervention or program, mechanisms (M) are thought to be the responses by people that are triggered by changes in context (C) [7]. Traditional realist syntheses use these C-M combinations to generate transferable ‘program theories’ that suggest that certain interventions are more or less likely to work in certain ways, for certain people, in certain situations [3, 4]. In other words, how does a change in context generate a particular mechanism that in turn produces specific outcomes?

In developing these program theories, realist reviews assume a broad inclusion of evidence, both quantitative and qualitative, which is used to develop an understanding of ‘what works, for whom, in what contexts, to what extent, and most importantly how and why [2, 3]?’ Therefore, instead of offering prescriptive advice on what the ‘best buys’ might be for policy makers, a realist synthesis attempts to provide a transferable theory that suggests that a certain program is more or less likely to work in these respects, for these subjects, in these kinds of situations [2, 3].

However, realist review methods are not without their difficulties. The iterative, flexible nature of realist reviews does not align well with protocol-driven, standardized processes common to established systematic review methods [8]. As results from a realist review are context dependent, the generalizability of findings to other settings will depend on the operation of similar mechanisms to generate outcomes of interest. Moreover, in completing a realist review, a high level of training and experience is required, which may not be found routinely in government or policy development agencies, academic institutions, or community-based organizations. Perhaps most relevant to our current discussion, realist reviews as outlined by Pawson et al.[3, 8] can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. In addition, due to their expansive and exploratory nature, realist reviews can often suffer from ‘scope creep’.

‘Rapid reviews’ have emerged in response to the incompatibility between information needs of policy makers and the time requirements to complete systematic reviews. Rapid reviews provide a way to generate similar types of knowledge synthesis as more comprehensive systematic reviews do, but in a much shorter time period. While some have questioned the validity of rapid reviews [9, 10], there remains a need to achieve a balance between comprehensiveness and timeliness for many policy-relevant decisions [11]. Therefore, understanding how this realist approach may be applied in cases where there is limited time and resources to generate a realist-based product that can incorporate research, theory, and practice knowledge and thus meet the demands of real-time policy developers/evaluators, is a valuable contribution to both evidence-informed policy-making and the realist review literature.

‘Rapid Realist Review’ methodology (RRR) has been developed by members of our group as a tool for applying a realist approach to a knowledge synthesis process and producing a product that is useful to policy makers in responding to time-sensitive and/or emerging issues where there is limited time and resources. Where relevant, we compare the RRR methodology to realist reviews generally, or ‘traditional’ realist reviews. While there is no set definition for what constitutes a realist review other than its use of realist logic and constructs, we present ‘traditional’ realist reviews as those that typically engage in a much longer exploration of the literature and ‘testing’ of theories, as well as those that present results within a framework of theory development. While others have begun to describe methodologies for short-term evidence synthesis projects [12], the RRR methodology described here is intended to incorporate the theory specification of a realist review [8] and the boundary clarification aim of a scoping review [13]. (See [11] for an example of an earlier review). This is also consistent with the recently published RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) publication standards for realist syntheses, which note that realist reviews need to be focused based on the time and resources provided as well as the questions that need to be answered [14].

Here, we describe our experience in developing RRR, the key ingredients in making it work, the policy questions to which it is most suited, the strengths and limitations of the method, and the future directions of how this approach may be optimized in practice. In doing so, we provide key examples of RRR projects completed by our group over the past six years.

We provide here brief summaries of key examples of reviews completed to date by this team. Table 1 includes the commissioning agencies, title, duration and purpose of each review. For this paper, we also considered key findings and recommendations resulting from each, as well as the strengths and challenges that arose while completing the review. We draw from these examples to highlight the conceptual strengths and limitations of rapid realist review methods and the circumstances under which they offer most value.

Table 1 provides details of the RRRs completed by this team to date. Agencies commissioning RRRs have included Federal Government Agencies (e.g., Public Health Agency of Canada, U.S. National Institutes of Health), Provincial Health Ministries (e.g. Saskatchewan Ministry of Health) and associated initiatives (e.g., Healthlink BC), regional health authorities (e.g., Vancouver Coastal Health Authority), and key research funding bodies (e.g., Canadian Institutes of Health Research). In a number of instances, reviews have been funded by research granting agencies for work to be completed in partnership with identified collaborators.

There is considerable variation in the topical focus of each RRR included in Table 1. Reviews often have been commissioned to synthesize information on complex, system oriented issues, such as large system transformation, improving organizational competencies for addressing health literacy needs, enhancing interorganizational partnerships, and improving evidence-informed health policy as well as bettering the people-centredness of health authorities. In addition, a number of reviews have focused on specific clinical specialties, domains or services, such as tele-health interventions, emergency department discharge practices, and optimal care models for chronic disease management programs.

As outlined in Table 1, RRRs may be completed for a variety of purposes. Two broad purpose domains appear to emerge, with RRRs used either to inform the development of practical solutions to problems faced in care delivery, or to develop a more conceptual understanding of how to organize ideas or frameworks for refining high level strategic thinking (Table 1). For the former, RRRs have been used to synthesize best practices for specific service provisions, such as evidence-based tele-health initiatives or an inventory of self-management interventions for chronic disease patients. More conceptual applications have produced ‘simple rules’ for guiding large system transformation, syntheses of frameworks for supporting evidence-informed policy-making and strategies for fostering organizational partnerships. Methodologically, RRRs have offered decision makers a tool for merging sources of evidence including published literature, grey literature and expert opinion, and as such, have provided knowledge users with the ability to make sense of how and why they might expect certain outcomes to occur, and the role of context in these outcomes. Further, they have helped knowledge users determine if sufficient substrate exists for other structured review exercises. In doing so, RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist.

From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy making (i.e., evolving patterns of decision-making that can be adapted based on new knowledge or contextual factors); knowledge quantification (i.e., the amount of literature available that identifies context, mechanisms and outcomes for a given topic); understanding tensions/paradoxes in the evidence base, as when similar changes in context trigger similar mechanisms and produce divergent outcomes, or when changes in context trigger different mechanisms to produce the same or different outcomes; and, reinforcing or refuting beliefs and decisions taken. See Table 2 for case studies that include detail on specific findings for two reviews. RRR provides a tool for developing rules for guiding decision-making, such as the advantages of investing in both top-down and distributed leadership, as well as on-going performance measurement in large system transformations (e.g., Knowledge And Action For System Transformation (KAST) review), or the need for innovative inter-organizational investments for improving the links between knowledge creation and adoption. While these guiding rules can inform the development of strategic directions, they can also be used as tools to confirm or dispel existing beliefs; for example, confirming the superiority of multi-faceted chronic disease prevention programs compared to non-multi-faceted approaches within a specific context for particular populations, or identifying that large scale changes in Canadian health systems at this moment in time require engaged physician and patient groups. Because they highlight I-C-M interactions, they also make apparent situations when similar types of interventions would not result in their intended outcomes due to contextual differences triggering alternate mechanisms. RRRs make explicit the amount of literature readily available for review, and in the case of a limited evidence base, can draw on the input of the expert panel in understanding the absence of literature and subsequently developing directions to take in sourcing other relevant literature for the review. In addition to understanding when little is known on a given topic, RRRs also highlight where tensions or uncertainties exist in an evidence base, such as what the precise characteristics of an effective network might be for a given purpose within a specific geographic, political and temporal context, or the differential effectiveness of tele-health methods for chronic disease self-management for populations within varying contexts.

The RRRs cited in this analysis have a number of strengths. Engaged members of both the expert panel and reference group ensure rigorous methods by providing an extended body of expertise and experience, and validating findings and program theories in an iterative way. In addition, the participation of the reference group increases the local utility of review findings. Moreover, the results speak to the value of diverse data collection methods, building on the literature through methods such as surveys and focus groups. In addition, reference groups tend to include (or provide links to) key decision makers in commissioning agencies, leading to closer connections between knowledge producers and users. As such, the RRRs completed to date have generally performed well as tools for responding to time-sensitive policy issues.

As with any review process, there are challenges to the RRR process. Ensuring that the scope of the review remains contained and the questions definable can be especially important and difficult in the RRR process. Like all reviews, difficulties may emerge when limited published or grey literature exists, where the quality of included studies is poor, or where the findings of studies are heavily contextualised to particular settings or times. Forming the membership of the reference group and ensuring sustained involvement are challenges for all reviews, particularly in rapidly changing political environments where membership of the reference group may change during the course of a given project.

The RRR method outlined here and a more traditional realist review share numerous similarities (see Table 3). Both are designed to be responsive to local policy needs. Both are grounded in realist logic and constructs, explicitly attempting to understand the interactions between contexts, mechanisms and outcomes. However, they also have some key differences, which allow policy makers to apply each type of methodology strategically to maximize its utility within a particular local constellation of history, goals, resources, politics and environment. In particular, the RRR methodology is explicitly designed to engage knowledge users and review stakeholders to narrowly define the research questions, and to streamline the review process, allowing the entire review process to be completed within three to six months (see [11] for an example). There are limitations associated with reducing the time available to complete such a review, as are discussed below. However, some policy decisions are required in a very short time frame (six months or less), thus making a methodology that can gather and synthesize the maximum amount of evidence in a compressed time period extremely useful.

The cost implications for the rapidity of the review process are difficult to quantify, since the bulk of our work is with RRR, thus prohibiting a good comparison within our own team for rapid vs. traditional realist reviews. We surmise that the skillset required for the review and synthesis increases the cost for personnel, although the specific relative costs will vary by project, content area, and individuals involved. While the overall length of time for RRRs will decrease costs overall, project management costs per month are likely to be greater for RRRs than for longer realist reviews, given the necessity to interact frequently with the reference group and expert panel, and the need to obtain timely responses from them.

As discussed, the RRR method outlined here involves both a local reference group and an expert panel. A traditional realist review likely engages a group equivalent to a local reference group, but to our knowledge no expert panel is engaged as a standard part of the realist approach to knowledge synthesis. Among realist reviews more generally, ‘advisory groups’ are quite common, and often serve the role of a merged ‘reference group’ and ‘expert panel’. We have found it useful to separate out the ‘expert panel’ into its own functioning unit. To make the engagement process more efficient, asking the reference group to focus on the utility of the findings, and their relevance to the work at hand within a local context, has worked well. The expert panel is similarly asked to focus its efforts on the validity of the findings and our interpretation of the evidence. The engagement of experts allows for at least two important contributions: firstly, it provides a validation process for findings that have been generated from an expedited review process. In the RRR method, the search strategy is intended to be robust, but not comprehensive. Due to the time constraints, it is fully acknowledged that not all references related to the research question, or even all that will produce theoretical saturation, will be identified and included in the review. However, by engaging the expert panel in the searching stage of the project, we can accelerate the search process, and at the same time be reasonably assured that any key theoretical contributions within critical articles, reports, or other resources (including those in the process of publication) will be identified and included. Secondly, review of preliminary findings by the expert panel validates results of the review, ensuring that the interpretation of the literature is consistent with the experiences of those currently engaged in work around the review topic. The intent of incorporating an expert panel is to integrate the lessons learned from the literature with those of identified content leaders who have experience working to integrate knowledge from research, practice and policy perspectives, and to build program theories based on that combined knowledge.

The RRR method also differs from a more traditional realist review in the role of theory, and specifically the presentation of theory in the final synthesis product. A ‘traditional’ realist review explicitly grounds the review process in theory; it produces a narrative summary of the interactions between context, mechanisms, and outcome configurations [8]. The reviewer then constructs one or more ‘middle range theories’ (i.e., ones that ‘involve abstraction… but [are] close enough to observed data to be incorporated in proportions that permit empirical testing’ [19]) to account for the configurations. The literature is then explored to ‘test’ these theories based on emerging findings from the review. In the RRR method, results are generated through a very similar process; however, theory-building is viewed as a supportive process to identifying families of interventions (I) and explaining why they would produce outcomes of interest (O) by generating specific changes in context (C) that would then trigger particular mechanisms, rather than the primary result. In some cases, however, the content area for the review is nascent enough that there is insufficient published literature to generate fully developed theoretical frameworks in which to ground the review. In those cases, the local reference and expert panels assist in producing and refining program theories that describe the current best thinking about how programs should work in different contexts. In either case, the focus of the review team is to produce generally accepted explanations among policy makers and content experts for what works within a particular set of parameters, rather than aiming to produce explanations that are potentially transferrable across contexts and populations. The distinction may appear to be semantic, but its impact is very real for both process and product of the review.

Rapid Realist Review offers a useful tool for addressing time-sensitive policy decisions. The RRR methodology provides a way of handling the tight time limits and rapid turnaround necessary that often characterize policy decisions. The risks to the rapid nature of the review methodology are a potential lack of comprehensiveness, or an increased risk of bias. These risks are compensated for by the involvement of content experts who can ensure that the results are complete, and that they resonate with practice knowledge. RRR also incorporates local knowledge users and links to commissioning agencies, thus enhancing both usability and utilization of results.

Rapid Realist Reviews are scalable with respect to involvement of external experts based on the time and resources available. The RRR methodology works well in contexts where there is limited evidence due to its incorporation of practice evidence via expert panels.

While the end products of RRRs focus less on theory development than on intervention-outcome links, they still rely on program theories to explain how and why certain families of interventions may produce outcomes of interest within a given context for specific populations. It can be more difficult to engage policy makers in theory development, but this does not lessen the utility of realist reviews for producing evidence-informed policy. Government agencies interested in using a RRR methodology will likely need to involve expert methodologists. However, this helps maximize the efficiency of key agency or government staff.

In conclusion, for policy makers faced with making difficult decisions in short time frames, related to system-level programs/initiatives/directions, for which there is sufficient (if limited) published/research and practice-based evidence available, RRR provides a practical, outcomes focused knowledge synthesis method.