Utilization of a population health survey in policy and practice: a case study

Public funds are expended through health research to lead to improvements in policy [1, 2], practice, resource allocation and, ultimately, the health of the community [3]. This can only occur if the evidence derived from the research is used to inform practice and policy decisions. There is growing interest by both funding bodies and researchers in measuring the impact of research. Over the past two decades many theoretical frameworks and approaches to measuring research impacts have been proposed [4], but there have been few empirical studies exploring how and why research is used [5, 6].

Population health monitoring surveys form one component in the public health research cycle, providing a key source of information about the prevalence and patterns of public health problems [7]. Such information may assist in guiding appropriate interventions, track changes over time, and support evaluation processes. That is, population monitoring surveys potentially provide valuable information to support policy and practice, although further research is generally required to test and disseminate effective interventions [8].

Despite the high investment in monitoring, little is known about if and how this type of population health data is used to inform policy and practice or the key factors influencing its use. We identified only one such study by de Goede and colleagues, exploring the use of epidemiological research in the development of local public health policy in the Netherlands [9, 10]. This study found that survey data was more often used in a conceptual way to improve understanding of the health problem or issue, rather than in a specific and direct way (instrumental use) or to justify a position or particular course of action (symbolic use) [10]. Research use was influenced by interaction between researchers and local health officials, the personal belief systems of the actors involved, and a range of contextual factors [9].

Further research is required to explore if, and in what ways, population health monitoring data is used in other contexts and settings. Understanding the factors influencing the use of this type of research is important in informing how population health monitoring systems can be planned and implemented to enhance the uptake of findings into the policy and program planning process.

Therefore, the aim of this paper is to provide a case study analysis of the utilization of the findings from an Australian series of population monitoring surveys of school-aged children’s weight and weight-related behaviors, the Schools’ Physical Activity and Nutrition Survey (SPANS). Three surveys have been conducted in 1997, 2004 and 2010 in the state of New South Wales (NSW), and provide a key source of data on NSW school children’s weight status, physical activity, sedentary behaviors and nutrition [11‐13]. Funded by government educational and health agencies since 1997, this monitoring program provides information on changes over time in key variables, and can potentially contribute to guiding and monitoring the progress of public health policy and programs that address children’s weight-related behaviors. However, the extent to which the surveys have actually been used for these and related purposes, has not yet been systematically investigated or documented.

The specific aims of this paper are to: i) describe if and how SPANS findings have been used at various time points to influence policy and practice; ii) explore the key facilitators and barriers to the use of the findings; and, iii) critically reflect on the methods applied to assess research use and make recommendations for future studies.

This study utilized a case study approach to explore if, and in what ways, a population health monitoring program was used to influence policy and practice, and to identify the key factors influencing its use. Case study methods are appropriate for answering ‘how’ and ‘why’ questions when the phenomenon of interest (SPANS surveys) is embedded within a real-life context (policy and practice environment) [14]. The study was approved by the University of Sydney Human Research Ethics Committee and all participants gave written informed consent to take part in the study. An overview of the study methods and key steps in the research process are given in Figure 1.

This is one of the few studies to document the temporal impact of a government population health monitoring program. We found that SPANS was successful in advancing knowledge and understanding of the issue of child obesity, weight-related behaviors, building capacity of the academic researchers and some EUs, and influencing policy and practice decisions across a range of sectors. The actual policy and practice impacts were in line with key recommendations arising from the survey reports. The positive impacts of the survey findings were attributed to the perceived credibility of the survey by EU groups, the processes used to disseminate the findings and a range of contextual factors. Contextually, the survey had a supportive government policy, strong partnerships between CIs and EUs, and established mechanisms for dissemination of the findings across a range of government sectors.

This case study demonstrates the positive impact a population health monitoring program can have on a range of policy and practice decisions. In particular, the findings of this study have demonstrated that population health monitoring data can be used as an advocacy tool (to attract attention and funding to an issue), priority setting (identifying areas and target groups for intervention), support and justify existing programs/approaches, or identify the need for alternatives. While others have suggested that epidemiological research can be used in a conceptual way to improve understanding of a health problem [9, 10], this case study found that the survey findings were also used instrumentally to directly underpin new policies (such as banning soft drink sales in schools), and symbolically to justify existing policies and programs (such as introducing healthy school canteens), in addition to improving understanding of the issues associated with child obesity (conceptual use).

The instrumental and symbolic use of the survey findings in influencing policy decisions may reflect the fact that the EUs interviewed in this study were directly involved in the development and implementation of policies and programs relevant to the survey findings. In contrast, the study by de Goede and colleagues involved a broader range of EUs, most of whom were not directly involved in execution of health surveys. The use of the survey findings may also reflect the political imperative to take action to reduce child obesity, particularly in the early stage of child obesity prevention responses when there was an absence of other evidence on effective intervention approaches [17, 18]. It is important to note, however, that epidemiological evidence in itself does not provide solutions to health problems, and needs to be considered alongside the broader body of evidence about effective intervention and policy approaches, as highlighted by many of the EUs in this study.

It is clear from this study that population monitoring surveys are one of many factors influencing public health policy and practice. In line with previous research [5, 6, 19‐24], we found that a range of contextual factors were critical in facilitating the use of the survey findings. In particular, the 2002 NSW Child Obesity Summit [18] provided a supportive policy context which facilitated the formation of partnerships between survey researchers and EUs, and provided mechanisms and structures (such as the NSW Government Action Plan for Child Obesity Prevention 2003–2007, [16] and a Senior Officers Group with responsibility for overseeing implementation of this plan) which facilitated application of survey findings. In addition, ongoing media coverage of child obesity throughout this period reinforced the political and social relevance of these surveys [25].

The findings highlight the importance of researchers tapping into existing policy networks, structures and processes, utilizing policy windows and building partnerships and trust with policy makers and other EUs over time. This is in accordance with the findings of Innvær and colleagues who found that personal contact and timely relevance were the most commonly reported facilitators of research use in a systematic review of 24 studies of the use of evidence by health policy makers [22]. Additionally, our findings point to the importance of the dissemination process in influencing the use of findings from population health monitoring surveys, in particular, the use of active dissemination strategies, such as workshops and presentations, to encourage contact and interaction between academic researchers and EUs. The use of a range of other dissemination mechanisms such as short summary reports in lay language and media releases, as well as traditional academic publications, were all shown to be important in facilitating use of the findings by EUs. There is increasing emphasis from funding agencies on making research evidence readily available, however recent studies of public health research suggest that most dissemination activity beyond publishing academic papers appears to be undertaken in an ad hoc, unfunded fashion and that access to dissemination advice and support for researchers from funding agencies and academic institutions is lacking [26, 27]. This suggests that more emphasis should be placed on funding and supporting a range of dissemination activities on behalf of both funding bodies and academic institutions themselves.

Traditionally, tools for measuring research impacts and assessing ‘research excellence’ are based on the number and significance of academic publications (such as bibliometrics and journal impact factors). However, there is growing interest in assessing actual impact on real world policy and practice. While a number of theoretical frameworks for assessing impact have been proposed [5, 28‐32], the findings of our study suggest that the impact categories proposed in such frameworks, including that of Banzi et al.[4], can capture the practical ways in which survey data is utilized. However, the findings also indicate that it can be difficult to attribute impacts to a single piece of research, particularly the longer term societal, health and economic impacts, which almost always arise for a complex array of contributing and contextual factors. This highlights the need for alternative ways of conceptualizing research ‘impacts’. The use of ‘contribution mapping’ as proposed by Kok and colleagues [33] may provide an alternative way forward. This approach aims to assess ‘contributions’ of research as opposed to ‘impacts’ and focuses on understanding how research and knowledge utilization processes evolve amongst the key actors involved. Further clarity is also required by researchers and funding agencies about the intended contribution that population health monitoring surveys are expected to make to policy and practice, in order to assess the degree to which this was achieved.

This study has a number of strengths and limitations. In terms of strengths, we assessed impacts using multiple methods, including bibliometric analysis, interviews with survey researchers and EUs, and documentary checks. This allowed for triangulation of data in the form of case summaries. The documentary checks lend confidence that the perspectives of the EUs are credible. While the SPANS reports have been disseminated widely across the NSW education sector, the full range of users, and their responses and application of findings, are unknown, although likely to be highly variable. It must be acknowledged that the EU interviewees were purposefully sampled on the basis of their role as the primary target audience and having direct responsibility for child obesity prevention policy initiatives. There may also have been some degree of social desirability response bias, whereby EUs felt obliged to report positive or over-inflated impacts. We attempted to reduce social desirability response bias by having researchers not previously involved in SPANS conducting the interviews and undertaking the analysis. The recall of impacts was somewhat problematic for the 1997 survey, as this was conducted over 15 years ago and only one EU (out of three) was available to be interviewed. Hence we are unable to ascertain whether the impacts of the 1997 survey may be underestimated or overestimated in this case study. Overall, there has been a high degree of continuity in the CIs involved in SPANS and EUs in NSW child obesity prevention over the study period, which may also have increased difficulties in attributing influence to any single one of the three surveys, and lent a positive bias. We recognize that there is no single or straightforward way of determining the full range of impacts across multiple potential EUs; however this case study provides a practical illustration of how a Government population survey can be used.

The SPANS case study provides important new insights into how population health monitoring data can be used to influence policy and practice. This ranged from raising awareness of the issue (i.e., child obesity), identifying priority areas and target groups for interventions, underpinning new policies and supporting and justifying existing policy and/or programs. Our findings highlight the importance of survey monitoring programs being conducted by credible independent agencies, the use of a range of dissemination strategies that directly engages EUs, and the need for strong partnerships with policy makers and EUs from the inception in order to increase ownership over the findings and commitment to action. Population monitoring data, however, needs to be considered alongside other evidence about effective intervention approaches to maximize its influence on practice and policy decisions.