Persistent misunderstandings about evidence-based (sorry: informed!) policy-making

From its gradual emergence as a potentially viable and fruitful idea, evidence-based policy-making has prompted some reservations. From technocratic suspicions [2], to evocations of restrictive ‘RCT-based policy-making’[3], the overall unreliability of scientific results and to the possibility of doing more harm than good when it comes to health policy interventions [4‐6], the idea of evidence-based policy-making seems to have been hashed out from just about every angle.

As in any area of practical rationality, it is rather unlikely that evidence can be the only decisional factor. Under practical constraints and limited rationality, realistic expectations can only go so far as to make valid and relevant evidence one of the factors weighing in a decision process and, at best, a weighty justification for a specific course of action. On the other hand, and this is possibly where evidence-based policy-making has a strong case, policy processes can be more or less informed by evidence. Alternatively, they can be completely uninformed by evidence, thus giving weight to other sources of justification (opinions, interests, gut feelings, etc.). Evidence-based policy-making, rather than trying to turn everything into an occasion to delve into scientific literatures, has a clear answer to these alternatives scenarios: one should try and find the most reliable, most objective, most relevant evidence available and make the most out of it within practical constraints.

Inasmuch as evidence-based policy-making requires the interplay of knowledge generation (by scientists and/or academics) and public decision-making processes, it is likely to be a complex enterprise. One obvious reason is that both processes are complex enough on their own and commonly take place independently of one another. As Black [7] rightfully warns us:

Researchers need a better understanding of the policy process, funding bodies must change their conception of how research influences policy, and policy makers should become more involved in the conceptualisation and conduct of research. Until then, researchers should be cautious about uncritically accepting the notion of evidence based policy

This appears entirely reasonable in that it does not discredit the principles themselves, but rather sheds light on their difficult implementation and provides realist expectations about the success rate of evidence-based policy-making practices (i.e. they do not always work and they are not ready-made solutions to every policy problem).

So, what does it really imply? Multiple things, mainly along two lines: 1) behavioral change from policy actors and the adaptation of the policy-making process, and 2) behavioral change from researchers in their way of communicating research evidence. On the one hand, proper infrastructures and resources need to be in place to support evidence-based principles [8, 9] and the decision-making process need some adjustments to allow a greater input from research evidence. On the other hand, further attention should be directed at the level of proof. In this respect, hierarchies of evidence typically display a structured view of the internal validity of research designs when pursuing causal inference. However, we believe that the relation between evidence and the policy process needs to be even more elaborated and structured so as to reflect the variety of research questions policy-makers are interested in.

No single design can answer everything, not even RCT’s [10]. Turning evidence gathered in one context into policy advice in another context is not straightforward, not even with well executed RCT’s [11]. The question of transferability of results is an important and difficult one, but tools exist to guide this process [12, 13]. In sum, even in the specific context of causal inference, RCT’s are not the only research design that can claim to validly inform decision, as they can themselves be controversial, contradictory or irreproducible. As Chalmers [5] puts it: “any engagement with empirical evidence needs to be serious, not polemical”. We believe the seriousness of the engagement should reflect a thorough discussion (well beyond the scope of this article) regarding the adequacy of a given research design to the questions on is interested in.

In the policy arena, policy-makers typically pursue a wide variety of policy objectives and are in need of multiple sources of information and evidence. In science, researchers are attentive to questions of efficacy and causal inference, but also other types of questions. Analogously, policy-makers are interested in answering different kinds of questions, where matters of causal inference are just one of them. If one is interested in evaluating the efficacy of a given potential intervention, no doubt that surveying RCT’s (of syntheses of them) are your first best bet. But if you are interested in something else, or should you find yourself facing a lack of such evidence, you have to turn to different research designs. This is where evidence-based policy-making advocates generally leave potential research users without much clue, perhaps giving the impression that RCT’s are the overall, wide-ranging gold standard. As argued by Davies & Powell [14] “the knowledge requirements for effective social policy and effective service organisation go far wider than just ‘what works”’. We couldn’t agree more, but a potential guide to the principled inclusion of other research designs in the policy process is still lacking.

As is generally the case in science, the research process usually starts by defining a problem and accordingly formulating specific questions. The next logical step is to identify and perhaps devise the proper research design to answer your question. We argue here that policy-makers should probably proceed in a similar fashion when looking for evidential support.

Policy-makers can look into a variety of types of questions in need of answers. The can be broadly categorized as questions related to 1) specific interventions, 2) non-interventional determinants or, 3) specific outcomes. For each of these lines of questioning, a wide array of research question can be elaborated. For example, if one is interested in a specific intervention, say a policy to increase access to health services, one can be interested in the effectiveness of the intervention or in the views of different stakeholders. In cases of questions pertaining to effectiveness, the ideal research design to provide such answer is undoubtedly an experimental design (such as an RCT) or if unavailable a quasi-experimental one. In the case of stakeholder’s views on the said intervention, experimental and quasi-experimental designs are not needed, but rather designs such as cross-sectional opinion surveys, qualitative studies using interviews or mixed-methods are appropriate. In other words, each type of question can be linked to an appropriate research design to be prioritized. Doing so offers a complement to traditional hierarchies of evidence and provides more detail and guidance for end user’s such as policy-makers.

Table 1 presents a non-exhaustive list of types of questions and of research designs that would ideally provide each type of question with the most reliable and valid answers. In a recent unpublished rapid review from one of the co-authors (MO) and colleagues conducted on behalf of a division of a Canadian ministry whose analysts wanted to learn about the phenomenon of skills mismatch, the list of 11 questions was presented to the client at the start of the project to assess their needs. The client selected seven questions that were deemed relevant to him. This example perfectly illustrates that policy-relevant questions extend way beyond matters about effectiveness and that all kinds of research studies might at one point be needed.

Looking at the table, one first needs to figure out what kind of question needs to be answered and can then look for methodologically grounded suggestions for research designs to look for. For instance, cross-sectional studies should not be your first go to place when interested in causal inference (assuming other types of evidence is available), but cross-sectional surveys might be your best bet to figure out what views stakeholder have of a given intervention. The key here is not to be rigid about which design should be implemented or consulted, but rather to take on board the fact that some designs are more appropriate than others (all things being equal) to answer certain types of question. This principle is quite simple, and to some extent well known by methodologists, but not necessarily cashed-out for policy purposes.

Here is a list of commonly encountered objections that we mostly agree with but still believe need to stop being put forth as justifications against evidence-based policy-making principles, or otherwise need being addressed in a more constructive manner:

Even though that last remark might instill some disappointment in the evidence-based enthusiast, it is still illustrative of the added value of evidence-based practices. It just goes to show that the actual process is perhaps what defines a policy procedure to be evidence-based, rather than its single output. Along with Chalmers [5] we agree that:

A lack of empirical evidence supporting opinions does not mean that all the opinions are wrong or that, for the time being, policy and practice should not be based on people’s best guesses. On matters of public importance, however, it should prompt efforts to obtain relevant empirical evidence through evaluative research, to help adjudicate among conflicting opinions.