The use of evidence in public governmental reports on health policy: an analysis of 17 Norwegian official reports (NOU)

Nearly 30 years ago, Lindblom and Cohen stated: '... in public policy making, many suppliers and users of social research are dissatisfied, the former because they are not listened to, the latter because they do not hear much they want to listen to' [1]. Twenty years later, the British government asked for policy that was 'shaped by the evidence rather than a response to short-term pressures' [2]. This quest for evidence by the British government exemplified a new trend in modern policy-making and, since the early 1990s, the success of evidence-based medicine has generated calls for, and discussions of, evidence-based policy-making [3‐11]. This trend has been encouraged by international institutions such as the EU and WHO [12‐14].

A call for evidence-based policy-making is also a call for the use of scientific methods in data collection and in the validation of information. Whenever policies are discussed, the question is often simply, "What will work?" Accordingly, when making priorities and formulating healthcare policies, a key factor will be evidence of their effects. A transparent and explicit approach is essential for reliable, valid and rational decision-making.

In interview studies, policy-makers state that three factors will facilitate their use of scientific evidence in policy decisions: high quality research; the inclusion of data on effectiveness; and summaries with clear recommendations [1, 15‐18]. If research studies are to have a greater impact in decision-making, this means it is important to assess the quality of such a vital element in official policy documents.

Previous studies of health policy documents commissioned in the UK and elsewhere and by the WHO have investigated the consistency of suggested policies with evidence gathered in the field [14, 19‐23]. Studies of documents commissioned in the UK have revealed a lack of consistency between recommendations and best evidence [20].

In 2002, the Norwegian government stated that it wished to adopt the evidence-based approach to policy-making[12]. In Norway, the government and the parliament frequently commission comprehensive reports, known as Norwegian official reports (NOUs), which deal with complex policy questions of national importance [24]. In such cases the government nominates a committee to write the report on the basis of a set of relevant questions (the mandate). NOUs are clearly the government policy documents that make most reference to research evidence. Similar reports exist in Sweden and in both countries these reports resemble a hybrid of British green papers and commissioned reports. There are many studies of health policy-makers' perceptions of their use of evidence, but only few studies of their actual use of evidence [15]. This paper presents an empirical study that examines how scientific evidence is used in policy documents by reviewing reports from government-appointed committees in Norway to assess their handling of questions of effect.

All NOUs published by the Norwegian Department of Health and Care Services during 1994-1998 were collected and those that reported evidence of health effects were included in this study. Of the 20 NOUs published during the study period, three were excluded either because the mandates did not address questions of effect or because the committees did not interpret the mandates as doing so, resulting in a total of 17 analysed NOUs. A full list of titles appears in Table 1.

The themes of the reports varied from broad policy issues, such as the setting of healthcare priorities, to narrower issues such as the health consequences of particular stimuli (e.g., electromagnetic fields). Some reports spanned a number of topics, such as those covering health and social services among the Sami population or public health services in local communities.

Systematic reviews of the outcomes of randomised controlled trials have probably been the key element in the development of evidence-based medicine. Systematic reviews assess search strategies and carry out critical appraisals. The ISQ is based upon the same principles as those underlying systematic reviews. Ministries, the government, and parliament rely on NOUs to provide the most comprehensive advice on subjects of national importance. Our analysis suggests that more than half of NOUs dealing with healthcare contain inadequate descriptions and assessments of the quality of the evidence of effect. We are not suggesting that NOUs provide the public with low-quality reports, merely that their scientific quality regarding questions of effect is low.

Our study suggests that committees prioritise policy relevance, as the NOUs included in the study primarily focused on policy relevance and policy implications. In both respects we found the reports' use of scientific evidence to be generally mediocre. We identified a lack of clarity regarding both the reports' primary intended audience and the identification and assessment of the main consequences of the study. There was insufficient treatment of validity and precision in relation to hard science questions. This may arise from a belief among the authors of reports that policy-makers are uninterested in the science underlying committee recommendations. Committees may also not consider their time and resources best spent teaching policy-makers about scientific standards.

It is impossible to be sure why committees give lower priority to hard science questions, since we have no data from interviews with committee members. We do know, however, from a systematic review of interview studies concerning health policy-makers' perceptions of their use of evidence, that policy-makers want summaries that make clear recommendations [15]. The committees may focus on relevance and consequences simply because they believe this is what policy-makers primarily want and find useful in decision-making. There is some support for this inference in our finding that the scientific quality of the reports is highest where evidence is used in the recommendations and in the summary. Our findings suggest that both policy-makers and experts tend to prioritise summaries and recommendations, rather than the hard science underlying them, since experts' participation on committees did not seem to influence the strategy used to gather research information.

The fact that seven reports obtained an average ISQ score of below 2.0 suggests that committees give insufficient attention to the assessment of scientific evidence. In other words, policy-makers must look elsewhere for transparent descriptions of search strategies for literature, references to evidence gathered in the field, and assessments of the evidence of effect. Our raw data indicates that questions of effect escaped the committees' attention in three of the seven reports, making their ISQ scores of 1 or 2 hardly surprising. We have no information about why some of the committees largely avoided questions of effect.

The mandates typically posed broad questions, rather than clear-cut questions of effect. Although NOUs were only included in our study if questions of effect were relevant to their mandates, these questions of effect were sometimes rather implicit and were always connected to broader issues and to a broader mandate. Since NOUs are intended to deal with questions of national importance, their mandates will naturally be broad. It is easy to imagine the broad mandate for the NOU report 'Care and knowledge: the Norwegian Cancer Programme', which deals with a variety of topics, such as: definitions of cancer; cancer as a national problem; the social consequences of cancer; care structures and processes for cancer sufferers; and the administrative and organisational consequences. Within the time span allotted for producing a report of this nature, it is very difficult for a committee to carry out a comprehensive review of questions of effect.

When reviewing cancer treatments, it is also of course difficult to decide where to focus. Even if there is a decision to focus on breast cancer (instead of, for example, lung cancer or childhood cancer), it will still be necessary to select from the many different questions of effect in the treatment of breast cancer. Even a narrow focus on 10 different questions of effect with a protocol in the Cochrane library will still exclude more than 50 other protocol questions in the same database [42]. Let us suppose that an NOU dealing with a Norwegian cancer programme dealt with 10 questions of effect in the treatment of breast cancer. Even if the report were written in such a way as to receive top ISQ scores, it would still tell us little about the Norwegian cancer programme. An NOU that answers a broad mandate by focusing narrowly on questions of effect is less useful for policy-makers.

Politicians, scientists, bureaucrats, experts and lay people often have (quite legitimately) differing interests. If public governmental reports refer to evidence in an explicit and transparent way, readers will largely be able to make reliable judgments by comparing the pros and cons apparent from the evidence. Explicit and transparent references to supporting evidence will assist in the identification of bias rooted in special interests in the recommendations of governmental reports.

This study shows that NOUs, in relation to their evaluation of health effects, lack transparency and, overall, show few signs of the use of systematic processes as far as scientific quality is concerned. A systematic, explicit and transparent approach - in fact, an approach that would generate a high ISQ score - may encourage the development of the evidence-based decision-making that Norway and other nations and international institutions desire and seek.

Some people might object that applying the ISQ criteria would overly narrow the focus of reports and cause the omission of other kinds of information more relevant for policy-makers: instead of promoting democratic debate, application of the ISQ criteria might lead to meritocracy. However, such objections would be convincing only if reports scoring highly on the ISQ criteria were inherently difficult for politicians, bureaucrats, experts and lay people to understand. Instead of rejecting out of hand the possibility of writing understandable reports of high scientific quality, committees should focus on writing reports that are accessible to academics and non-academics alike. Committees that include experts as well as non-experts should make particular efforts in this regard. One possible solution would be to prepare separate reports for different audiences. The process of doing so might help clarify the role of scientists in such committees and protect researchers from having their results abused [29, 43].

Research is a cumulative enterprise. Whether the ISQ is useful for policy-making, and whether a narrow focus on questions of effect helps policy-makers make wise decisions remains to be seen. The ISQ cannot and should not exclude politics from evidence-based policy-making. But if calls by governments for evidence-based policy-making are to be taken seriously, we need to develop and test evidence-based approaches such as those endorsed by the ISQ so that evidence-based policy may progress from aspiration to reality.