Background
Measuring the wider policy and practice impacts of health research is becoming an important field of study as researchers and funding bodies are increasingly required to demonstrate value for money, both in terms of generating new knowledge and contributing to health and economic outcomes [
1]. The research impact assessment literature calls for metrics as proxy measures for the impacts of research, and the number of research citations in public policy documents has been suggested as one such indicator [
2‐
4]. Presumably, if research is cited in a policy document, that research can be thought of as having an “
effect on, change or benefit to the policy” and therefore an impact beyond the academic setting [
5]. Reviewing sources cited in policy documents potentially offers a quantifiable, objective and simple method of measuring the impacts of research. The method has been proposed as less subjective and burdensome than relying on researchers themselves to recall and track the impacts of their research, particularly if the tracking of citation can be automated [
6].
Only a few studies have examined the feasibility of extracting references from health policy documents [
6‐
10]. These studies suggest that it is possible to use bibliometric techniques to identify the geographical origins of research, categorise the type of research cited (e.g. basic versus clinical research), and identify the funding sources of cited research. Quantifying citations of specific research outputs in a sample of policy documents has been incorporated into comprehensive research impact assessments [
11‐
14]. To date, research citation studies have examined a limited range of health policy document types and areas of research, including clinical guidelines, on a range of topics [
7‐
11,
13] and policy documents relevant to nursing home visiting research [
14], drug monitoring systems [
12], environmental health research [
6] and climate change research [
15].
Recent studies have investigated the feasibility of using automated approaches to policy document analysis, such as ‘text mining’ and reference list data extraction, to provide information about research impacts [
6,
15]. Such methods rely on policy documents to be electronically accessible and to cite research in a manner that allows data extraction. It is likely that some policy documents are more suited to this type of analysis than others. For example, clinical guidelines are often readily available online, informed by an explicit process of evidence appraisal, and cite evidence in the form of a bibliography or database of sources [
14]. Other types of policy documents, for example, government plans and strategies, may not share these characteristics. To interpret the findings of automated research citation assessment processes, it is important to understand their strengths and limitations with respect to different types of policy documents. This is an important area of research given that commercial applications of such technology, such as Altmetric, are already available and being utilised by researchers, universities and research funding bodies to track research impacts [
16].
Existing studies suggest it is especially important to understand the research and policy context of research citation in policy documents [
13,
15]. The citation of research in policy documents may be the result of different factors, including efforts by researchers to interact with policy-makers to disseminate their findings, or policy organisation-driven initiatives to include research in policy documents [
15]. One hopes that research cited in policy documents was read, understood and considered in relation to the content of the policy. However, it can be difficult to determine the actual role played by cited research [
14]. For example, research may be used in many ways by policy-makers, including instrumental uses, where research is explicitly or directly applied to address a policy problem; conceptual uses, where research has an influence on awareness, understanding or attitudes/perceptions amongst policy-makers generating ideas, arguments and criticisms which feed into policy debate; or symbolic uses, where research is used to justify a position or specific action already taken for other reasons or to attain specific goals based on a predetermined position [
17]. In addition, many factors other than research contribute to policy development, including political ideology, stakeholder interests, community expectations and other competing information [
14,
18].
In this paper, we investigate the feasibility of compiling and assessing research citations in a sample of public health policy documents and consider the utility of this approach as an indicator of research impact in the context of childhood obesity prevention policy in New South Wales (NSW), Australia, between 2000 and 2015. During this period, childhood obesity was a NSW Government public health priority (Box 1) and a wide variety of policy documents were produced. We examined the feasibility of retrospectively obtaining these policy documents, how these were accessed (publicly and by request), whether and how they cited research, the rates of citation, and the type of research cited. We also explored the document context of research citations to consider what this could reveal about how the research had been used.
Methods
In considering policy documents as a source of information about research impact in the case of childhood obesity policy in NSW, we sought to answer the following research questions:
1.
What types of public health policy documents were available, to what extent did these cite research, and how feasible was it to extract citation information from these documents?
2.
What were the rates of research citation in these documents, and what type of research was cited?
3.
Where and how was research cited, and what, in this case, did the citation of research indicate about research impacts?
Policy document search
We searched for childhood obesity policy documents produced between 2000 and 2015 by NSW government agencies. Policy documents were defined as government documents in the form of a report, discussion paper, draft or final policy, formal directive, programme plan, strategic plan, budget bid, service agreement, implementation plan, guideline, or protocol with a focus on health service or programme design, delivery, evaluation or resourcing [
19]. We excluded documents that did not mention obesity or were not relevant to children and young people (i.e. specifically targeted at another population group). We began with key policy documents known to the authors, and used a snowball technique to identify new documents mentioned in those already collated. This initial search was supplemented by further searches of current and archived NSW Government websites. Access to archived websites was provided by the NSW Health Library. Two current and two former public health employees of the NSW Ministry of Health reviewed the list of identified documents and made additional suggestions that led to further online searches. Documents that could not be located through internet searches, or via the NSW Health Library, were requested directly from the NSW Ministry of Health. Our search did not seek ministerial and cabinet briefing notes or internal meeting minutes as these documents were likely to be subject to Cabinet-in-confidence legislation.
Policy document coding and reference extraction
A structured coding instrument was used to collate descriptive information about each policy document, including policy document title, date, author, type of document, availability, length, inclusion of any references, inclusion of childhood obesity references, and referencing style (e.g. reference list, footnote, hyperlinks or combination of these). The text of each document was scanned for passages or sections relevant to childhood obesity. References referring to childhood obesity or linked to statements relevant to childhood obesity, cited in these passages/sections, were identified. Details for these references were located through Scopus and exported or manually entered into a reference database. Extracted reference details included the publication title, date, authors’ names and affiliations, abstract, keywords, and publication source. Information on funding sources was rarely provided and was therefore not included in our analysis.
Reference coding
A second coding instrument was developed to compile information about each reference, including type of publication/information source, geographical origin and type of research for research publications. We distinguished research from other referenced information sources (policies, media releases, information on websites, reference books) (see Table
1 for definitions). The research category was further subdivided into peer-reviewed research, non-peer-reviewed research and internal/external statistics. We further categorised peer-reviewed research according to whether the study was a single study, review/research synthesis or other. Finally, using an existing coding system, peer-reviewed research was categorised as descriptive research, intervention research, measurement research or other [
20]. In most cases, the previously extracted reference details (which often included an abstract) provided enough information for coding. Where this was not the case, the primary source document was obtained. We also recorded, for each reference, the number of policy documents in which the publication had been cited.
Table 1
Definitions used for coding extracted research references
Geographical origin of publication |
International: | Document not Australian in origin. Research not conducted in Australia (for single studies) or none of the authors have Australian affiliations (for research synthesis and other) |
Other Australian: | Document Australian in origin but not from a NSW source. Research conducted in Australia but not NSW (for singles studies) or any authors have Australian affiliations but not NSW affiliations (for research synthesis and other) |
NSW: | Document from NSW source. Research conducted in NSW (for single studies) or any authors have NSW affiliations (for research synthesis and other) |
Referenced information sources |
Research: | Publications reporting analyses of quantitative or qualitative data, development or application of theory [ 19], or discussions/commentaries about research and research methods. Including: |
| Internal/external statistics: Statistical data from internal (e.g. NSW Health Survey) or external (e.g. Australian Bureau of Statistics) sources |
| Non-peer-reviewed research: Research reported in technical monographs or in the grey literature |
| Peer-reviewed research: Research reported in peer-reviewed journals |
Policy documents: | Documents (report, discussion paper, draft or final policy, formal directive, programme plan, strategic plan, budget bid, service agreement, implementation plan, guideline or protocol with a focus on health service or programme design, delivery or resourcing [ 19]) authored by government agencies, non-government organisations or professional bodies where the purpose of the document was not to report research findings |
Other information sources: | Cited information other than policy documents or research, e.g. media releases, information on websites, reference books, expert testimony |
Peer-reviewed research |
Single study: | Publication reports the findings of a research study |
Research synthesis: | Meta-analyses, systematic reviews, literature reviews |
Other: | Commentaries, editorials, case reports, conference reports, conceptual models, health programme descriptions |
Descriptive research: | Describes the nature, scope and correlates of the problem |
Intervention research: | Evaluates the impact of an intervention, implementation under real world conditions, or the wide scale use of an intervention |
Measurement studies: | Development of measures or testing the reliability, acceptability or validity of measurement instruments |
Policy document content analysis
For each policy document, we noted the structure and purpose of the document, the type of research that was used (if any), how different types of research were used and in what component of the document, and whether the processes for accessing any cited research had been described. In addition, we reviewed each citation occasion to note the way in which the research was cited (used to support a policy argument, decision or statement in the text), whether the research was cited alone or with other references, and whether the importance of the cited publication was explicitly stated. Illustrative examples of common citation contexts were identified and compiled.
Discussion
Policy documents are a commonly used source of information in research impact assessments [
29,
30]. Most assessments start with a specific research project and trace that forward to identify its impact [
31]; for example, an Australian study utilised this forward tracing approach to assess the impacts of a population-wide series of obesity monitoring surveys [
32]. In the current study, we used a ‘backward tracing’ approach to examine 15 years of NSW childhood obesity policy documents as the starting point for analysis to determine what information these documents could provide about the utilisation and impact of research.
We found that it was possible to locate a range of policy documents covering our study period. However, over a quarter were not publicly available, only two-thirds contained references, just under half cited obesity research and over a third of the documents containing references used unorthodox referencing styles. There did not appear to be any patterns in terms of the types of documents more likely to cite research, and the number of research citations in each document was highly variable. It was possible to collate reference information, although this was difficult and time consuming where references were not contained in a single list.
In total, 263 unique peer-reviewed and 94 non-peer-reviewed publications were cited, of which only a fifth of peer-reviewed and just over a quarter of non-peer-reviewed publications were cited in more than one policy document. The cited peer-reviewed research included almost equal numbers of descriptive research and intervention research publications. Descriptive research was used to support a policy rationale, while intervention research was used to support policy options, chosen solutions or to substantiate evaluation findings. Both single research studies and reviews of the literature were cited. Overall, 40% of the cited peer-reviewed research and twice that proportion of non-peer reviewed research was Australian in origin, possibly reflecting the importance of locally relevant research to the policy process or knowledge of local research amongst local policy-makers [
8,
9]. Cited non-peer reviewed research commissioned by NSW policy agencies appeared to have been particularly influential in terms of informing the policy process. Research was not the only form of information cited in policy documents reflecting the many sources of information considered by policy-makers, including information from monitoring studies and population-based surveys, government reports, stakeholder consultation, other policy documents and expert testimony [
18,
33].
Our findings raised several issues related to the limitations of this methodology as a measure of research impact, when used in isolation. Firstly, citation of a publication did not necessarily indicate that the research had influenced the policy process [
14]. Sometimes, the role individual publications played in the policy process was explicitly stated, while in other cases it was only possible to say with certainty that the publication was cited to establish credibility for the argument being put forward in the policy document. Research was also sometimes used to support statements that were tangential and not relevant to the overall policy direction. This is not entirely dissimilar to the ways in which research may be used and cited in academic publications. In academic publications, some cited research will have provided a foundation for the research under study or contributed to the researcher’s knowledge of the topic, but some will have been cited for comparative purposes, or as contrasts. We would suggest that research citation in a policy document is an indication of the research publication’s accessibility to the document authors and its relevance to the policy topic, but not necessarily that the research had an impact on policy (defined as an “
effect on, change or benefit to the policy” [
5]). In addition, it was difficult to determine the nature of any effect. For example, distinguishing between whether the cited research had been utilised in a tactical or symbolic way to support a predetermined position, or whether it had directly contributed to the policy’s adoption and development (instrumental use) was very challenging. Certainly, determining whether the cited research had influenced the knowledge, ideas and awareness of issues amongst the policy-makers involved (conceptual use) was not possible utilising this method. Other evidence, such as the testimony of policy-makers, may be required to confirm whether the cited research had an influence on policy directions, how it was utilised and the extent of that influence.
A second issue is that the method identified research that was cited, but not necessarily all the research that was considered during policy development. Sometimes, research was mentioned without being cited, or accessed and considered through other documents (e.g. systematic reviews or other policy documents) or processes (e.g. consultations) but not cited directly. In addition, not all policy deliberations will be recorded in a formal way, so policy documents only provide a partial record of policy discussions, some types more so than others. For example, legislative documents may provide a more fulsome account of policy deliberations, including any information presented, as records of verbal proceedings and written submissions are kept [
34‐
36]. Citation rates are likely to provide an underestimation of research use by policy agencies and the method has the potential to miss research that was in fact impactful, and place undue importance on cited research. It may be that the method is more suited to policy documents that utilise a more consistent approach to the consideration, appraisal and citation of research. Certainly, studies examining research cited in clinical guidelines do not leave as many questions unanswered, as our study has, regarding the base of research considered and the process for appraising and including research that is cited [
7‐
9].
Finally, the high degree of variability in this sample of policy documents in terms accessibility and research citation patterns and styles has implications for automated citation extraction systems. Only some of the policy documents we examined would have been suitable for or yielded results from automated data extraction, thus producing rather hit and miss outcomes for identifying impactful research publications. Whether the variability we encountered would be found in other public health policy areas warrants further investigation. However, it is likely that there will be differences between policy areas which will influence citation rates and the utility of automated systems. This is an important issue if rates of citation of research in policy documents are ever to be compared across research disciplines (e.g. clinical compared to public health research).
Limitations of this study
We have focused on a single policy issue in one Australian jurisdiction. The types and requirements of documents obtained for this study may differ from policy documents available for other public health policy areas or other jurisdictions. The findings of this case study are not intended to be generalised, but rather indicative. Within this case study, we attempted to obtain as many policy documents as possible, but some policy documents may have been missed. Recall of policy documents was difficult for some policy-makers we asked due to the timeframe of the study. Prospective collation of documents as they are produced would eliminate some of these issues. In addition, only one author completed the coding of policy documents and references, which may have introduced some bias, although strict coding definitions were developed and implemented.
Conclusions
The analysis of citation of research in policy documents performed herein indicated that research had been accessed and considered, but further qualitative investigation is required to answer the more important questions of why and how the research was used. Rather than being used as a definitive indicator of impact, automated citation searches may be useful in terms of pointing researchers in the right direction, so that they can investigate what role their research played in a policy process. Nevertheless, the links between citation and impact may be more easily drawn in other policy areas (e.g. clinical guidelines), where research appraisal feeds more directly into policy recommendations.
While highlighting instances of use of specific research products, research citation across the public health policy documents included here varied greatly and is unlikely to be an accurate reflection of actual research use by the policy agencies involved. Research citation in policy documents in this instance said more about the policy document than research utilisation and this may also be the case for other public health policy areas.
Box 1: Overview of the NSW childhood obesity policy context
The NSW State Government was the first in Australia to identify and respond to the need for a coordinated approach to obesity prevention. In 2002, the Childhood Obesity Summit brought together key stakeholders to develop an across-government, inter-sectoral response to childhood obesity, culminating in the development and implementation of Prevention of Obesity in Children and Young People: NSW Government Action Plan 2003–2007 [
37]. This plan aimed to raise community awareness and developed the platform for a coordinated cross-agency commitment to the implementation of obesity prevention strategies. Subsequently, targets for reducing childhood obesity rates have been included in successive NSW Government State Plans, which set the strategic direction for all NSW government agencies and services, and integrated as priorities in NSW State Health Plans. An increase in national and state funding enabled the expansion of existing initiatives and the introduction of new programmes [
21,
38]. Tackling Childhood Obesity is currently (since September 2015) one of the 12 Premier’s Priorities for improving outcomes for the people of NSW [
39].
The NSW Ministry of Health has been the lead agency for delivery of obesity strategies in NSW, supported by governance structures to ensure coordination and accountability for actions across all NSW government agencies. Initiatives to address childhood obesity have been implemented utilising a settings-based approach with a focus on homes and communities, healthcare environment, schools, childcare and children’s services, and the urban environment. Programmes are run at State and local health district levels and include a mix of whole of population and targeted approaches to obesity prevention. Another identified priority area has been knowledge production and research infrastructure development. The NSW Government funds the Schools Physical and Activity and Nutrition Survey, to provide a mechanism for objectively monitoring overweight and obesity rates for children and young people in NSW, as well as the Physical Activity Nutrition and Obesity Research Group at the University of Sydney [
40].
These investments in NSW have run parallel to national obesity strategies including successive national obesity plans and a parliamentary inquiry into obesity in 2008. National activities have focussed on community education, including social marketing campaigns and national physical activity and nutrition guidelines for children and young people. The Australian Government has responsibility for food taxation and marketing legislation; however, no changes in this area have been made to date.