Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006

Public funds are expended on health research in large part to lead to improvements in policy [1‐3], practice, resource allocation, and ultimately, the health of the community [4, 5]. However, the transfer of new knowledge from research into practice continues to be far from optimal [2, 6, 7]. It is widely recognized that increasing the impact of research on policy and practice is likely to require many different strategies, including the development of research-policy partnerships, better summaries of evidence and more research-receptive policy and funding agencies [8, 9]. It is also increasingly acknowledged that studies designed to evaluate the impact of interventions to improve health (intervention research) can inform subsequent intervention-specific policy and practice [10]. However, only a relatively small proportion (between 10-23%) of primary research funded by public health agencies or published in the peer reviewed literature is intervention research [11, 12].

Little is known about the nature and mechanisms that underlie the influence of intervention research on health policy or practice. In fact, there are no agreed systematic approaches for measuring such impacts [13]. Traditional indices of research productivity relate to numbers of papers, impact factors of journals and citations. These metrics are widely used by research granting bodies, although they do not always relate well to the ultimate goals of applied health and medical research [14‐16]. The emerging literature on research impact [17‐19] highlights its complex, non-linear, unpredictable nature, and the propensity, to date, to count what can be easily measured, rather than measuring what “counts” in terms of significant, enduring changes [14].

A recent systematic review of approaches to assessing research impacts by Banzi et al. [13] identified 22 reports included in four systematic reviews and 14 primary studies. These publications described several theoretical frameworks and methodological approaches (for example, bibliometrics, econometrics, interviews, ad hoc case studies) to measuring research impacts, with the “payback model” as the most frequently used conceptual framework [19]. Based on this review of existing models, Banzi et al. differentiated five broad categories of research impacts: i) advancing knowledge; ii) capacity building; iii) informing decision-making; iv) health benefits; and, v) broad socio-economic benefits.

To date, most primary studies of research impacts (‘impacts research’) have been small scale case studies; and there has been no comprehensive assessment of impacts and their mediators across any single applied research funding scheme. The New South Wales (NSW) Health Promotion Demonstration Research Grants Scheme (HPDRGS) was designed by NSW Ministry of Health, Australia, in response to a paucity of evidence on large-scale intervention effectiveness across prevention policy priorities. The specific aims of the scheme are to fund applied research that builds the evidence-base for health promotion policy and practice; develop partnerships and build capacity for health promotion research between health districts, universities and organizations outside of the health sector. This paper reports on a exploratory case series analysis of all 15 projects funded under the HPDRGS during the period 2000 to 2006, to determine their subsequent policy and practice impacts (the ‘what’) and to explore the forces and factors influencing these impacts (the ‘how’ and ‘why’).

At the commencement of the study in January 2012, 15 projects funded during the period 2000 to 2006 had been completed for at least twenty-four months and most (n = 12) for longer than four years. This period was selected to balance the time required for evidence of impact to become manifest, against the potential accuracy of recall by respondents. A case study approach was used to explore if, and in what ways, research projects were used to influence policy and practice, and to identify the key factors (how and why) which influenced their use. Case study methods are appropriate for answering ‘how’ and ‘why’ questions when the phenomenon of interest (in this case, applied research) is embedded within a real-life context (policy and practice environment) [20]. Due to the diversity of projects under consideration, this case series included a number of different methods (Figure 1). The study was approved by the University of Sydney Human Research Ethics Committee and all participants gave written informed consent to take part in the study.

While a growing number of studies have examined impacts of research [13] and research funding [26], this is the first study to document the impacts of a policy-driven applied research funding scheme. This analysis of research impacts indicates that some, but not all, of the intervention research funded through the HPDRGS achieved a wide range of tangible impacts across most domains. It is clear that the three projects with the highest overall impact ratings in this study had substantial impacts on advancing knowledge and capacity building, as well as policy and practice. However, some projects with substantial research impacts (papers and citations) yielded only minimal policy and practice impacts. This reinforces that traditional indices of research impact and researchers’ track record on publications and grants are not always an accurate guide to the policy and practice impacts of their research.

This case study analysis demonstrates the positive impact that intervention research funding can have on a range of policy and practice decisions, with findings used as a policy advocacy tool (to attract attention and funding to an issue), for priority setting (identifying areas and target groups for intervention), and to support and justify existing programs/approaches or identify the need for alternatives. In a number of instances project findings informed the early stages of policy development, when there had previously been a lack of definitive evidence about effective intervention approaches. We also found that research findings were used to directly underpin key elements of existing policies for falls prevention in older people, and tobacco control. In addition, findings were used to improve understanding of issues associated with implementing and assessing new interventions such as travel guides and other promotion of active transport.In many instances the use of project findings by practitioners reflected a need to act on state-wide policy imperatives. The introduction of the NSW policies on smoke free hospitals and falls prevention saw many practitioners tasked with developing local responses using relevant HPDRGS project resources. This highlights the value of having research funding aligned with state-wide policy.

It is clear from this analysis that many factors influence public health policy and practice, with evidence from an effective intervention study in itself generally not enough to shift the current approaches. Consistent with previous research [5], we found that findings of a single study were usually considered alongside a broader body of evidence about effective intervention approaches, as well as a consideration of the local context and timing requirements. In a seeming paradox, some studies that had null or equivocal results still achieved moderate policy and practice impact. This suggests that adoption of project findings into policy and practice are influenced by factors other than evidence of effectiveness. Closer review of these projects revealed that the introduction of state-wide policies and programs meant that practitioners adopted the available project materials, meeting an immediate practice need, even though studies were demonstrably not effective.

Further, we found that a range of contextual factors were critical in facilitating the use of the projects findings, which is in agreement with previous studies [27‐31]. In particular, supportive policy contexts encouraged partnerships between researchers and end-users from the inception of projects, and where possible utilized existing structures (policy and practitioner networks, etc.) for communication. Tapping into existing policy and practitioner networks and processes appeared to enable researchers to build partnerships and trust with practice and policy ‘users’ and allowed better utilization of policy ‘windows of opportunity’. In some instances a confluence of events provided the right conditions for an intervention to be widely adopted into policy and practice. One such tragic event was the Virginia Tech massacre in the United States that highlighted the importance of mental health literacy and was thought to be a critical factor by the CI of this project in driving the early expansion of Mental Health First Aid in North America. All of the high impact projects were characterized by simple interventions that were well implemented, high quality research, champions to advocate and disseminate for adoption, as well as supportive contextual factors. The review of materials by the verification panel identified that an intervention’s capacity to be packaged and change ‘agents’ trained in its delivery was particularly important, as evidenced by the rapid expansion of Mental Health First Aid and Tai Chi across respective practice settings.

This study supports a growing body of evidence about the importance of embedding and linking research with broader strategic policy contexts. In a systematic review of 24 studies of the use of evidence by health policy makers, Innvaer and colleagues [28] found that personal contact, timeliness and relevance were the most commonly reported facilitators of research use. In our current study, impactful projects appeared to effectively engage key end-users groups, to ensure that projects were aligned to the interests and needs of such groups and to promote ownership of the findings and by doing so, increasing commitment to action. Most of the low impact studies had no such clear links with end-users or existing policy and practice networks.

Findings of this and other recent studies [31] also highlight the importance of the production of a range of dissemination products such as short reports, fact sheets and project resources, and their availability through websites, as well as publishing in open access journals to facilitate the use of the findings by end-users. There is increasing emphasis from funding agencies on making research evidence readily available [32, 33]. Yet, recent studies of public health research suggest that most dissemination activity rarely goes beyond publishing academic papers, appears to be undertaken in an ad hoc, unfunded fashion, and that access to dissemination advice and support for researchers from funding agencies and academic institutions is lacking [34, 35]. This study highlights the value of funding and systematically supporting a wide range of dissemination activities.

The excellent return on investment from the Rural Hearing Conservation Program Evaluation highlights what can be achieved with limited resources when research funding is well targeted. There appears to be merit in funding high quality evaluations of existing policies and programs. Increasingly, funding agencies require investigators to detail how their research impacts on policy and practice [36, 37], and a growing number of theoretical frameworks for assessing impact have been proposed [16, 38‐41]. This study demonstrates the utility of the scoring and panel verification methods used for identifying and measuring proximal research impacts (advancing knowledge, capacity building and policy and practice impacts). The findings of this and other recent studies [31] suggest however, that the longer-term impacts (societal, health and economic) of a single study can be difficult to discern and attribute. The CIs, end-users and verification panelist all reported difficulty in identifying and assessing these impacts for any single study. This is understandable, as such impacts almost always result from a complex interplay of contributing factors; and there remains a need for alternative ways of conceptualizing and measuring longer term research ‘impacts’.

This study has a number of strengths and limitations. The strengths of this study were that impacts were assessed using multiple methods, including bibliometric analysis, interviews with researchers and end-users, and documentary checks. These data were triangulated and distilled into case summaries, which were used in a rigorous verification process involving independent assessments of impacts and a group panel assessment. The documentary checks lend confidence that the perspectives of the chief investigators and end-users were credible, while the verification panel process provides a well established and tested methodology for reaching expert consensus, and minimizing subjectivity of assessments. The end-users were purposefully sampled on the basis of having knowledge and/or experience of how the project findings had been used, and while this ensured they contributed relevant information, there was potential for some degree of social response bias, as some end-users may have been inclined to report positive impacts or over-inflate those impacts. We attempted to reduce social response bias by having researchers not previously involved in funded HPDRGS projects conducting the interviews and undertaking the analysis. The recall of impacts was somewhat uncertain for some projects from the early funding rounds, as these were conducted between 10 to 12 years ago. For one of the projects the end-users could not be identified. It is possible therefore that the impacts may have been underestimated for some of these older projects.

This HPDRGS case series analysis provides new methods and insights into how intervention research projects influence policy and practice. Funded projects had variable impacts on policy and practice. Where impacts occurred they ranged from raising awareness of health interventions, identifying priority issues and target groups for interventions, underpinning new policies, and supporting and justifying existing policy and/or programs. The success of high impact projects was perceived in large part to be due to the nature and quality of the intervention itself (simple to understand, built in mechanisms for training and delivery), high quality research, champions who advocated for adoption, and active dissemination strategies. Our findings also highlight the need for strong partnerships between researchers and policy makers/practitioners to increase ownership over the findings and commitment to action.