A systematic evaluation of payback of publicly funded health and health services research in Hong Kong

While research is believed to be essential in guiding improvements in health systems and developing new initiatives [1], there is growing recognition of the importance of measuring its returns [2‐7]. In developed countries with long histories of publicly financing health research, providing evidence of benefit has increasingly become the criterion by which research agencies are evaluated and funding continuation assessed [8, 9]. Evaluating research outcomes potentially allows the development of more effective strategies to increase the likelihood of the implementation of "successful" research [7].

Although the "best" way to evaluate research outcomes remains controversial, there is a general consensus that the evaluation should capture not only "academic" outputs (e.g. peer reviewed papers), but wider attributable health and socio-economic benefits that may include knowledge production, research targeting and capacity building, informing policy/decision making, behaviour change, product development, health and health service benefits and economic benefits [3‐6, 10]. Aiming to encompass these multiple dimensions, the "payback" framework, developed in the 1990s, is gaining popularity to become one of the most widely adopted models to evaluate health research funds internationally (Table 1) [11‐17].

Although Hong Kong currently enjoys a relatively affluent economy, public funding for academic research was only formally established in the late 1980s. Funding allocation for projects in the fields of biology and medicine typically accounts for approximately one third of the total research budget [18]. Recognizing the need for locally relevant evidence to inform health policy and practice, the government has been supporting applied health research through the Health and Health Services Research Fund (HHSRF) since 1993. The fund aims to maximize population health, improve the quality of life, and enhance the standard and cost-effectiveness of the health system through funding research that generates new knowledge in areas of human health and health services [19].

To demonstrate accountability of public funding and to provide an evidence base for assessment of continuation of funding, we undertook for the first time a systematic evaluation of outcomes of research projects supported by the fund since it was established in 1993. The objectives of the study were to quantify the outcomes of completed research projects supported by the fund using the "payback" framework, in comparison with the more established health research funds abroad. We also explored factors associated with the impact of research outcomes on health policy and provider behaviour. Our results could have relevance for research funds planning similar evaluation exercises, particularly those that are less well established.

Of the 205 questionnaires sent, 178 (86.8%) were completed and returned by investigators. The mean duration between project completion and questionnaire return was 6.34 years (median 6.25, range 1.75 to 10.92), which was significantly shorter than that for projects for which questionnaires were not returned (mean 7.76 years, median 7.75, range 3.25 to 10.83). The questionnaire response rate was inversely related to the time from project completion (p <0.001, R² = 0.75; Figure 2). In particular, significantly greater proportion of questionnaires were returned for projects completed 7 years or less compared with those completed longer than 7 years (94.0% [109/116] vs. 77.5% [69/89], odds ratio [OR] 4.51, 95% confidence interval [CI] 1.81–11.23).

Otherwise, there was no significant difference between projects for which questionnaires were returned and not returned in terms of the funding award, duration of project, or the type of administering institution (Table 2).

Research is recognized as an essential feature of health care development and is increasingly used to influence all levels of health care provision. However, in most health care systems, including that in Hong Kong, the assessment of need, delivery of care and evaluation of preventive health and medical interventions has not been a strong feature. Health services research has the potential to substantially contribute to and influence health-related decision-making whether at the policy, practice, individual patient or population level. Well-structured research programmes are fundamental to adequate monitoring of the massive investments governments make in health care and to ensuring the appropriateness of future health care provision. Pressure to assess "value for money" in the use of public sector resources for research funding has risen in recent years [2‐9]. Documenting the attributable research funding outputs or outcomes is essential for establishing the evidence base substantiating the "payback" or return on investment of public funds made available for health care research.

To our knowledge this is one of the largest studies of outcomes from a single public health research fund using the payback framework in terms of the number of individual projects evaluated. The robustness of findings of the present study has benefited from the high rate of return of evaluation questionnaires by the investigators (86.8%), reducing the risk of bias. Indeed among projects completed within the past 7 years, questionnaires were returned for nearly 95% of projects. The rate of questionnaire return dropped off significantly for projects completed more than 7 years before, which might reflect a lower incentive for investigators to respond due to the long period of time elapsed since project completion. This observation might have implication for the optimal timing to conduct studies of similar nature in future.

In spite of the fact that the total amount of funding awarded to the projects evaluated in this study is only a fraction of that awarded by more established research funds in other countries (for instance, the UK Medical Research Council's total research spending was GBP3.76 billion between 1995 and 2005) [24], our study documented a significant contribution of the HHSRF to knowledge generation through scientific publication at a level comparable to overseas research funds of a similar nature. This is demonstrated both in terms of the number of publications per funded project (HHSRF: mean 5.4 vs. Australian NHMRC [9]: mean 4.3) and 'value for money' in terms of expenditure per peer reviewed publication (HHSRF: mean HKD 167,690 vs. Australian NHMRC [9]: AUD 37,400 [approximately HKD 228,257]). However, it should be pointed out that these figures do not reflect the true total expenditure per publication, as the amount contributed by other funding sources, including those supporting the salaries of senior investigators, is unknown. Although a number of publications were in high impact journals we were unable to explicitly assess their individual impact on knowledge transfer of research into practice.

We found relatively fewer subsequent research projects (44.9%) generated by HHSRF-funded projects. This might reflect the relatively lack of funding opportunities for investigators in Hong Kong – it is estimated that only 0.69% of Hong Kong's gross domestic product (GDP) was spent on research and development in all disciplines of science and technology in 2004, while the corresponding figures were 2.78% in the UK, 2.67% in USA, 1.96% in Canada, and 1.69% in Australia [25]. Nonetheless, the HHSRF has been instrumental in supporting the development of research capacity and building a research culture in Hong Kong. In addition, HHSRF-funded projects have led to similar outcomes in terms of change in policy and self-perceived health service benefit compared with overseas funds evaluated by the same payback framework (Figure 3).

With the relatively large sample of projects evaluated, we were able to explore factors associated with research outcomes. It is perhaps not surprising to find that increased funding awards were significantly associated with greater outcomes in a range of payback categories. Perhaps more importantly, by multivariate analysis, we found that liaison with policy makers and integration of researchers in the policy formulation process, rather than publications or funding award, were the key factors influencing health behaviour change, health policy and health care benefit, confirming previous impressions [26, 27].

The intrinsic limitations of the payback framework and the way it was applied should be recognized. Since information on project outcomes was provided by the investigators retrospectively, responses may reflect 1) recall bias, 2) over or under estimation of the effect of the research outcomes and 3) measurement error as the specific impact of the effect could not be explicitly measured and verified in all cases. There is inevitably a degree of subjectivity when researchers attribute their research findings to changes in health policy and behaviour, which involve a complex process with multiple factors at play, including political and economic consideration. It is also well recognized that researchers in different traditions or cultures vary in the way they conceptualize and explain the impact of their research [28]. It is likely that the heterogeneity in characteristics of the investigators sampled in this study has diluted the effect of the outcomes. In addition, respondent fatigue resulting from the length of the detailed questionnaire might also lead to response and information bias. In our study, the overall impact might also be underestimated, as PIs of projects completed more than 7 years prior to the survey were less likely to respond. Nonetheless, despite these shortcomings, with its growing acceptance by researchers and funding agencies, the payback framework represents a useful common tool by which the multiple dimensions of health research outcomes can be quantitatively and qualitatively measured, facilitating comparison in evaluation across different funds.

In conclusion, despite the relatively young age and modest budget of the fund, the HHSRF has resulted in substantial outcomes as measured by a multi-level payback framework. As expenditure on research activities has to compete with alternative uses of scarce health services resources, the benefits arising from the investment should withstand rigorous evaluation. Quantifying the impact and payback of health services research and demonstrating the societal benefits is essential to providing the platform for continued policy support for health services research funding. Future research should focus on the overall long-term societal benefit factors that influence the uptake and translation of research into practice, and on improving the match between investigator initiated and policy directed research.