Background
Methods
The two narrative systematic reviews of impact assessment studies on which this paper is based were conducted in broadly similar ways that included systematic searching of various databases and a range of additional techniques. Both were funded by the United Kingdom National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme. |
The searches from the first review, published in 2007, were run from 1990 to July 2005 [6]. The second was a more recent meta-synthesis of studies of research impact covering primary studies published between 2005 and 2014 [9]. The search strategy used in the first review was adapted to take account of new indexing terms and a modified version by Banzi et al. [11] (see Additional file 1: Literature search strategies for the two reviews, for a full description of the search strategies). Although the updated search strategy increased the sensitivity of the search, filters were used to improve the precision and study quality of the results. The electronic databases searched in both studies included: Ovid MEDLINE, MEDLINE(R) In-Process, EMBASE, CINAHL, the Cochrane Library including the Cochrane Methodology Register, Health Technology Assessment Database, the NHS Economic Evaluation Database and Health Management Information Consortium, which includes grey literature such as unpublished papers and reports. The first review included additional databases not included in the updated review: ECONLIT, Web of Knowledge (incorporating Science Citation Index and Social Science Citation Index), National Library of Medicine Gateway Databases and Conference Proceedings Index. In addition to the standard searching of electronic databases, other methods to identify relevant literature were used in both studies. This included in the second review an independent hand-searching of four journals (Implementation Science, International Journal of Technology Assessment in Health Care, Research Evaluation, Health Research Policy and Systems), a list of known studies identified by team members, reviewing publication lists identified in major reviews published since 2005, and citation tracking of selected key publications using Google Scholar. The 2007 review highlighted nine separate frameworks and approaches to assessing health research impact and identified 41 studies describing the application of these, or other, approaches. The second review identified over 20 different impact models and frameworks (five of them continuing or building on ones from the first review) and 110 additional studies describing their empirical applications (as single or multiple case studies), although only a handful of frameworks had proven robust and flexible across a range of examples. |
Results
Author, date, location | Programme/speciality | Methods for assessing health research impact/concepts and techniques | Impacts found | Factors associated with level of impact; comments on methods and use of the findings |
---|---|---|---|---|
Adam et al., 2012 [32]; Catalonia, Spain | Catalan Agency for Health Information, Assessment and Quality – Clinical and health services research | Bibliometric analysis; surveys to researchers (99, 70 responded, 71%); interviews – researchers (15), decision-makers (8); in-depth case study of translation pathways Canadian Academy of Health Sciences framework | Overall, 40 principal investigators (PIs) (of the 70) gave 50 examples of changes; examples included 12 organisational changes of the centre/institution; two public health management; two legal/regulatory (some PIs might have given more than one of these: therefore, total for organisational/management/policy changes: possibly 17–23%, and 20% figure used in this analysis); 29 of the 70 (41%): changed clinical practice | Interactions and participation of healthcare and policy decision-makers in the projects were crucial to achieving impact; the study showed that the Agency achieved the aim of filling a gap in local knowledge needs; study provided useful lessons for informing the funding agency’s subsequent action; the studies “provide reasons to advocate for oriented research to fill specific knowledge gaps” ([32], p. 327) |
Alberta Heritage Fund for Medical Research, 2003 [33]; Alberta, Canada | Alberta Heritage Fund for Medical Research – Health research | Survey to PIs (100, 50 responded, 50%); interviews with decision makers and users Version of Payback Framework | 49% impact on policy; 39% changed behaviour; 40% health sector benefits | Research teams with decision-makers or users more successful than those without |
Bodeau-Livinec et al., 2006 [34]; France | French Committee for the Assessment and Dissemination of Technological Innovations (CEDIT) – Health technology assessment (HTA) | Semi-directive interviews with stakeholders affected by the recommendations (14); case studies used surveys in hospitals to examine impact of the recommendations (13) | Widespread interest, “used as decision-making tools by administrative staff and as a negotiating instrument by doctors in their dealings with management....ten of thirteen recommendations had an impact on the introduction of technology in health establishments” ([34], p. 161); 7 considerable, 3 moderate: total 77% | Main factor fostering compliance with recommendations “appears to be a system of regulation” ([34], p.166) Reviewed other studies: “All these experiences together with our own work suggest that the impact of HTA on practices and… introduction of new technologies is higher the more circumscribed is the target of the recommendation” ([34], p. 167) |
Brambila et al., 2007 [35]; Guatemala | Population Council – Programme of Operation Research projects in reproductive health in Guatemala | Key informant (KI) interviews; document review; site visits to health centres and non-governmental organisations implementing operational research interventions; scored 22 projects (out of 44 conducted between 1988 and 2001) on indicators: 14 process; 11 impact; 6 context Developed an approach involving process, impact and contextual factors; drew on literature such as Weiss [18] and interactive approaches | Of the 22, 13 projects intervention effective in improving results, three interventions not effective; in 14 studies implementing agency acted on results; nine interventions scaled up in same organisation; five adopted by another organisation in Guatemala; some studies led to policy changes, mainly at the programme level (total 64% impact in combined policy and practice category) | Highlighted how impact can arise from a long-term approach and the several 5-year cycles of funding “allowed for the accumulation of evidence in addition to the development of collaborative ties between researchers and practitioners, which ultimately resulted in changes to the service delivery environments” ([35], p. 242) |
Buxton et al., 1999 [36]; United Kingdom | NHS North Thames Region – Wide-ranging responsive mode R&D programme | Questionnaires to PIs (164, 115 responded, 70%) and some bibliometric analysis for all projects and case studies (19); case studies included interviews with researchers and some users Benefit scoring system based on two criteria (importance of the research to the changes, and level at which the change was made) was used to score questionnaire responses about the impacts and re-score the impact from each study on which a case study conducted Payback Framework | 41% impact on policy; 43% change in practitioner/manager behaviour; 37% led to benefits to health and health service | The survey/case study comparison suggests “greater detail and depth of the case studies often leads to a somewhat different judgement of payback, but there is no evidence of a systematic under-assessment of payback from the questionnaire approach, nor, generally, of greatly exaggerated claims being made by researchers in the self-completed questionnaires” ([36], p. 196) |
Caddell et al., 2010 [37]; Canada | IWK Health Centre, Halifax, Canada, Research Operating Grants (small grants) – Women and children’s health | Online questionnaire to PIs and co-investigators (Co-Is) (64, 39 responded, 61%) Research Impact Framework: adapted | 16% policy impact: 8% in health centre, 8% beyond; 32% said resulted in a change in clinical practice; 55% informed clinical practice by providing broader clinical understanding and increased awareness (average of 43% for practice impact); 46% improved quality of care | An association between presenting at conferences and practice impacts; authors stress link between research and excellence in healthcare: “It is essential that academic health centres engage actively in ensuring that a culture of research inquiry is maintained” ([37], p. 4) |
Donovan et al., 2014 [38];Australia | National Breast Cancer Foundation – Wide range of programmes | Documentary analysis, bibliometrics, survey of PIs (242, 153 responded, 63%), 16 case studies, cross-case analysis Payback Framework | 10% impact on policy – 29% expected to do so; 11% contributed to product development; 14% impact on practice/behaviour – 39% expected to do so | Basic research – more impact on knowledge and drug development; applied research – greater impact in other payback categories; many projects had only recently been completed – more impact expected; in launching the report the charity highlighted how it was informing their funding strategy [92] |
Expert Panel for Health Directorate of the European Commission’s Research Innovation Directorate General, 2013 [39]; European Union | European Union Framework Programmes 5, 6, and 7 – Public health projects | Documentary review: all 70 completed projects; 120 ongoing; KI interviews with particularly successful and underperforming projects (16); data extraction form constructed based on the categories from the Payback Framework, with each of the main categories broken down into a series of specific questions Payback Framework | Appendix 1: only 6 out of the 70 completed projects did not achieve the primary intended output; 42% took actions to engage or inform policymakers; 4 (6%) projects change of policy, 22% expected to do so; 7 (10%) impact on health practitioners; 6 (9%) impact on health service delivery and 6 (9%) impact on health; 1 beneficial impact on small/medium-sized enterprise | Used documentary review, therefore for completed projects had data about whole set; however, “Extensive follow-up of the post-project impact of completed projects was not possible” ([39] p. 9) Comprehensive coverage of a programme without requiring additional data from the researchers; however, also shows the limitations of such an approach in capturing later impacts |
Ferguson et al., 1998 [40]; United Kingdom | NHS Northern and Yorkshire Region – Health Services Research (HSR) (two other programmes not included here) | Desk analysis (bibliometrics), surveys to gather quantitative and qualitative data sent to all PIs and Co-Is in all three programmes: but only HSR projects asked about policy, so just the 32 HSR responses analysed here Refer to Payback Framework; no attempt to develop own | This was part of a wider analysis, but in all three areas the projects were reactive; particularly difficult to make an impact with Primary and Community Care research | |
Gold & Taylor, 2007 [41]; United States of America | Agency for Healthcare Research and Quality – Integrated delivery systems research network | Documentary review of programme as a whole and individual projects (50); descriptive interviews (85); four case studies, additional interviews No explicit framework described | Changes in operations; “Of the 50 completed projects studied, 30 had an operational effect or use” [41] (Operational effect or use is a broad term: so the 60% put into our combined impact category) | Success factors: responsiveness of project work to delivery system needs, ongoing funding, development of tools that helped users see their operational relevance |
Gutman et al., 2009 [42]; United States of America | Robert Wood Johnson Foundation – Active living research | A retrospective, in-depth, descriptive study utilising multiple methods; quantitative data derived primarily from a web-based survey of grantee investigators (PIs, Co-PIs), of the 74 projects: 68 responses analysed; qualitative data from 88 interviews with KIs The conceptual model used in the programme “was used to guide the evaluation” ([42], p. S23). Aspects of Weiss's model used for analysing policy contributions | Generally thought to be too early for much policy impact, but 25% of survey, 43% of interviewees reported a policy impact; however, policy impact in survey could be from active living research in general, not just the specific programme, and could include: “a specific interaction with policymakers (e.g. testifying, meeting with policymakers, policymaker briefings, etc.) or direct evidence of the research findings in a written policy” ([42], p. S33) | Only 16% of grants had been completed prior to the year of the evaluation; some approaches “worked well, including developing a multifaceted, ongoing, interactive relationship with advocacy and policymaker organizations” ([42], p. S32); grantees who completed both interviews and surveys generally gave similar responses, but researchers included in the random sample of interviewees gave higher percentage of policy impact than researchers surveyed; questions slightly different in the interviews than in the surveys |
Hailey et al., 1990 [43]; Australia | National Health Technology Advisory Panel – HTA reports | Looked at technologies (20) covered by HTA reports from the panel up to end of 1988. Little provided on methods – presumably desk analysis, just states comparing recommendations, assessments and policy activities No framework described | Out of the first 20 technologies covered by HTA reports there had been significant impact in 11 and probable influence in three: 70% in total | Timing crucial factor for impact; at the margin of our inclusion criteria since it is based more on panel recommendations than a programme of research, but took first 20, not a selection |
Hailey et al., 2000 [44]; Canada | Canadian province (not stated) – HTA brief tech notes | Interviews with those requesting the 20 brief HTA notes (i.e. reviews); checks on quality of the reports made using desk analysis and comments from experts No framework described | 14 (70%) had influence on policy and other decisions | These HTA brief reviews were directly and urgently requested by users; at the margin of our inclusion criteria since it is not clear to what extent it was a research programme |
Hanney et al., 2007 [6]; United Kingdom | National Health Service (NHS) – HTA programme | Multiple methods: literature review, funder documents, survey all PIs of projects between 1993 and 2003 (204, 133 responses, 65%), case studies with interviews (16) Payback Framework | Technology Assessment Reports (TARs) produced for the National Institute for Health and Clinical Excellence (NICE): 96% impact on policy, 60% on clinician behaviour; primary and secondary HTA research: 60% impact on policy, 31% on behaviour Average for programme: 73% impact on policy, 42% on behaviour; case studies showed large diversity in levels and forms of impacts and the way in which they arise | Different parts of the programme had different impact levels; key factors in achieving impact – agenda setting to meet needs of healthcare system, generally high scientific quality of research, existence of a range of ‘receptor bodies’ to receive and use findings, especially demonstrated for the NICE TARs; pre- and post-interview scoring showed reasonable correlations: suggests most survey responses not making exaggerated impact claims |
Hanney et al., 2013 [45]; United Kingdom | Asthma UK – All programmes of Asthma research | Survey of all PIs (153, 96 responses, 59%), documents, case studies (14) involving interviews and some expanding the approach to cover role of chairs and centre Payback Framework | 13% impact on policy; 17% product development; 6% health gain; but case studies reveal some important examples of influence on guidelines, some potentially major breakthroughs in asthma therapies, establishment of pioneering collaborative research centre | Many types of research and modes of funding – long-term funding of chairs led to important impacts; comparison of evidence from surveys and case studies on same projects showed generally exaggerated claims not made; study informed strategy of the medical research charity |
Hera, 2014 [46]; Africa | Africa Health Systems Initiative Support to African Research Partnerships | Documentary review; interviews at programme level; project level information – for six projects, workshops, for the remaining four a total of 12 interviews; participant observation of end-of-programme workshop and presented some preliminary findings Key element of the design – adoption of an interactive model of knowledge translation | Policy impact was created during the research process: 7 out of 10 projects reported policy impact already, “The policy dialogue is not yet complete and further uptake can be anticipated” ([46], p. 3) | “Research teams who started the policy dialogue early and maintained it throughout the study, and teams that engaged with decision-makers at local level, district and national levels simultaneously were more successful in translating research results into policy action” ([46], p. 1); timing of evaluation – too early for some impact, but programme’s interactive approach led to some policy impact during project |
Jacob & Battista, 1993 [47]; Quebec, Canada | Quebec Council on Health Care Technology Assessments (CETS) – HTA | Case study analyses of impact on decision-making and cost savings of reports in first 4 years (10) Scored for policy influence – critical incidents used Interviews (45) with scientific and political partners, and staff at CETS; documentary analysis also used Desk analysis of cost savings Developed own CETS approach | Examined impact on decision-making and cost savings; 8 of 10 reports influenced decisions | Identified a series of key features of the Quebec system that were favourable to HTAs making an impact; these include “A general receptivity to rationality in decision making…the health care system in Quebec is organized in such a way that information produced by the council can filter easily into the decision-making process” ([47], p. 571); this is an example of the receptor body playing an important role |
Jacob & McGregor, 1997 [48]; Quebec, Canada | Quebec Council on Health Care Technology Assessments (CETS) – HTA | Comprehensive case study approach; similar to above on 21 reports in circulation sufficiently long for at least some impact to be estimated Used own CETS approach | 18 of 21 reports influenced policy (86%); 8 at the highest level | Context was same as above; “The best insurance for impact is a request by a decider that an evaluation be made” ([48], p. 78) (not entirely clear if these 21 reports included 10 reports above) |
Johnston et al., 2006 [49]; United States of America | National Institute of Neurological Disorders and Stroke – All pre 2000 phase III clinical trials in this field | Data on the effects of all 28 trials from desk analysis involving reviews, contact with PIs and others, and opinions of experts (4) Health economic modelling used to estimate return on investment (ROI) ROI analysis – a key example of a monetisation study | Six trials (21%) led to improvements in health: 470,000 quality-adjusted life years in 10 years since funding of 28 trials at cost of $3.6bn; the projected net benefit was $15.2bn; yearly ROI 46% (in total 8 studies, 29%, were identified as providing impact used in the analysis: two were cost savings only) | The main purpose of this study was to assess the public ROI; however, it seems to be the only such study that attempted to identify whether any health improvements had resulted from each individual project in a programme (and it thus met our inclusion criteria) |
Kingwell et al., 2006 [50]; Australia | National Health and Medical Research Council (NHMRC) – Wide range of fields | Survey of all contactable PIs completing in 1997 using a simplified version of NHMRC end-of grant report as the survey instrument (259, 131 responses, 51%) No explicit framework | 9% affect health policy; 24% affected clinical practice, 14% public health practice (in our analysis used the 24% as not clear how many might be duplicates); commercial potential: 41% | Highlighted some projects with clinically relevant outcomes for showcasing to the community |
Kwan et al., 2007 [51]; Hong Kong | Health and Health Services Research Fund – Range of fields | Adapted Payback survey sent to PIs of completed projects (205, 178 responses, 87%); statistical analysis including multivariate analysis Payback Framework | Use in policymaking, 35%; changed behaviour, 49%; health service benefit, 42% | Multivariate analysis found that investigator participation in policy committees as a result of the research and liaison with potential users were significantly associated with health service benefit, policy and decision-making, and change in behaviour; however, set out various limitations in the methods used |
McGregor et al., 2005 [52]; Canada | HTA Unit of McGill University Health Centre, Quebec – HTA | All 16 (100%) HTA reports incorporated into hospital policy and some cost savings | Hospital’s HTA Unit combined researchers to synthesise evidence and a policy committee to make recommendations; success because “(i) relevance (selection of topics by administration with on-site production of HTAs allowing them to incorporate local data and reflect local needs), (ii) timeliness, and (iii) formulation of policy reflecting community values by a local representative committee” ([52], p. 263) | |
Milat et al., 2013 [53]; Australia | New South Wales Health Promotion Demonstration Research Grants Scheme | Semi-structured interviews with Chief Investigators (CI) (17) and end-users (29) of the 15 projects; thematic coding of interview data and triangulation with other data sources to produce case studies for each project Case studies individually assessed against four impact criteria and discussed to reach group assessment consensus Banzi Research Impact Model | 10 out of 15 (67%) were in the moderate or high categories for impact on policy and practice combined (we did not have an economic category in our analysis and therefore decided not to include the combined health, social and economic category where 33% of the projects were rated as resulting in moderate or high impact) | High impact projects' success: “due to the nature and quality of the intervention itself…, high quality research, champions who advocated for adoption, and active dissemination strategies. Our findings also highlight the need for strong partnerships between researchers and policy makers/practitioners to increase ownership over the findings and commitment to action” ([53], p. 14) |
Molas-Gallart et al., 2000 [54]; United Kingdom | Economic and Social Research Council AIDS Programme – Social aspects of AIDS | 43 interviews with researchers of all 14 completed projects, then snowball approach for users: mapped network of researchers and users and post-research activity Framework based on the interconnection of three major elements: the type of output, the diffusion channels and the forms of impact – later contributed to development of Social Impact Assessment Methods through the study of Productive Interactions [23] | 50% of researchers claimed programme provided non-academics with tools to solve problems and been used to develop policies | Concludes a two to three stage process required to assess impact (interview researchers first, then users); normal sampling techniques inadequate because impact not distributed along a normal distribution curve; detailed project-by-project qualitative analysis important |
Oortwijn et al., 2008 [55]; The Netherlands | ZonMw Health Care Efficiency Research Programme – HTA | Survey data collected from PIs (43, 34 responses, 79%); case study analysis (including 14 interviews) of five HTA projects; developed and applied a 2-round scoring system Payback Framework | 10 projects (29%) had a policy impact, including 6 being cited in guidelines; 11 projects (32%) reported implementation of new treatment strategies: counted as informed practice | The assessment was perhaps too soon after completion of the projects to witness benefits for many of projects; unlike most HTA programmes this had a large responsive mode element and most studies were prospective clinical trials |
Poortvliet et al., 2010 [56]; Belgium | The Belgium Health Care Knowledge Centre (KCE) – HTA, HSR and good clinical practice | Documentary review; two group discussions: with 11 KCE experts, with 2 KCE mangers; interviews with stakeholders (20); web-based survey – total of 88 managers reported on 126 projects; nine detailed case studies; international comparisons with three agencies using documentary/literature review and interviews (3) Developed own framework | 58% of project coordinators thought the project contributed to policy development: more for HTA than good clinical practice or HSR; 16 of the 20 stakeholders said findings influenced decision making, four said not in their organisation; 30% coordinators thought the project contributed to changes in healthcare practice | Factors linked to impact include involvement from “stakeholders in agenda and priority setting. The quality of KCE research itself is high and in general beyond discussion. The relevance of KCE research findings is generally judged as high” ([56], p. 111–2); some similarities with other/earlier findings about HTA being more likely to make impact |
Reed et al., 2011 [57]; Australia | Primary care research | Online survey to 41 contactable CIs (out of 59 projects); asked impacts expected, how many achieved; some projects excluded as still underway, other refused; 17 completed out of 27 eligible Payback Framework | Four projects (24%) influenced national/state policymaking, but 8 (47%) influenced decision making at organisational, local or regional level (combined nine separate projects (53%) had policy/organisational decision impact); despite further examples of quite high levels of impact, surveys showed “these perceived impacts affected the health service organizations, clinicians and patients who took part in the research projects” ([57], p. 4) (we included the lowest of the three figures given for this, 29%) | The high level of use for policy and organisational decision making “reflects a high level of engagement of the researchers with potential users of their research findings” ([57], p. 5) |
RSM McClure Watters et al., 2012 [58]; Northern Ireland, United Kingdom | Northern Ireland Executive: Health and Social Care Research – All fields | Desk analysis of documents and literature, consultations with stakeholders, survey informed by Payback Framework, three case studies, benchmarking. Surveys to all PIs for projects funded between 1998 and 2011 who could be contacted (169; 84 responses, 50%) Payback Framework | 19% impact on policy development; for impact on health and the healthcare system: 20% health gain; 14% improvements in service delivery; 17% increased equity (the 20% figure used in our analysis represents the most conservative overall figure); substantial leveraged funds for follow-on projects came from outside Northern Ireland | Because Northern Ireland’s government did not contribute to the United Kingdom’s NIHR, researchers were not able to apply to the NIHR programmes. This “was seen by respondents as a major constraint to research activity… research was not seen as a priority within many organisations and that many key stakeholders in the health sector did not fully engage with research or see its benefits” ([58], p. 49); as a result of the assessment, Northern Ireland decided to subscribe to the NIHR |
Sainty, 2013 [59]; United Kingdom | UK Occupational Therapy Research Foundation – Occupational therapy | PIs of completed project invited to complete a ‘personalized impact assessment form’ (equivalent to a survey) (11, 8 responded, 73%) Two researchers provided an independent review of the collated findings Becker Medical Library Model | Three projects (37.5%) reported local clinical application: “particular tools, clinical advice, or models that were the subject of research having been used in practice” ([59], p. 534) | In relation to the clinically related activities of three projects: “Important to note, was the extent to which respondents highlighted this as being in the context of the participating services or host organizations” ([59], p. 534) |
Shah & Ward, 2001 [60]; Australia | NHMRC – Public health R&D committee | Self-complete questionnaires to CIs funded in 1993 (55, 38 responses, 69%); combined with desk analysis – attempted some correlations between publications and impact No framework stated | 58% claimed research influenced policy; 69% influence on practice; 53% stated both | “Influence on policy, practice or both was not associated with peer-reviewed publication in an Australian journal” ([60], p. 558) |
Soper & Hanney, 2007 [61]; United Kingdom | NHS Implementation Methods Programme – Implementation research | Postal survey of PIs (36, 30 responses, 83%) and potential users of the three projects in maternity care (227, 100 responses, 44%); poor response from other users to electronic survey; some desk analysis; interviews with key figures Payback Framework | 30% claimed impact on policy; 27% on practice; 54% of the midwives and perinatal care researchers surveyed said the findings from at least one of the three maternity care projects had influenced their clinical practice | In this new field, the programme generated considerable enthusiasm among members of advisory and commissioning groups, and increased understanding and interest in the field; some projects made considerable impact, but IMP did not have a communications strategy and as a programme it highlighted some of complexities facing implementation. |
The Madrillon Group, 2011 [62]; United States of America | NIH – Mind body interactions and health program | Mixed methods cross-sectional evaluation design; semi-structured interviews with 100% response rate – PIs of all 44 investigator-initiated projects and all 15 centres; impacts of centres scored by adapting the scales used previous in payback studies Adapted version of Payback Framework | Projects: 34% influenced policies; 48% led to improved health outcomes; the centres and projects, “produced clear and positive effects across all five of the Payback Framework research benefits categories” ([62], p. xiii) | Some projects were still in progress and it was too early to capture all the ‘latent’ impacts; conducted innovative analysis through examining three overlapping levels (programme, centre and projects); for assessing all projects used semi-structured interviews rather than surveys |
Wisely, 2001 [63]; United Kingdom | NHS – National R&D programme on primary/secondary care interface | Survey of PIs of projects completed by April 2001 (63, 40 responded, 63%); desk analysis comparing grades for applications and quality of project Payback Framework | 35% used in policy/decision making; 27% led to changes in practice; 25% health service benefits arisen | Some indication from limited data that applications graded as excellent more likely to lead to high quality projects with impact |
Wisely, 2001 [64]; United Kingdom | NHS – National R&D programme, mother & child care | Survey of PIs of projects completed by April 2001 (39, 26 responded, 67%) Payback Framework | 27% used in policy/decision making; 31% led to changes in practice; 23% health service benefits arisen | Some PIs thought that being part of a national R&D programme helped give their project greater credibility in the eyes of potential users |
Wooding et al., 2009 [65]; United Kingdom | Arthritis Research Campaign – Wide range of arthritis research | Web-based tick list survey of PIs in 2007 of grants ending in 2002 and 2006 (136, 118 responses, 87%) Developed from the Payback Framework was subsequently named the RAND/ARC Impact Scoring System | 6 projects (5%) policy impact; 8% “generated intellectual property that has been protected or is in the process of being so” ([65], p. 37) (over 80% of grants generated new research tools) | Much of the research funded was more basic and likely to inform further research rather than directly lead to impacts; also, it was probably too soon after the end of the projects to capture all the impact that might arise |
Zechmeister & Schumacher, 2012 [66]; Austria | Institute for Technology Assessment and Ludwig Boltzmann Institute for HTA – HTA | Desk analysis identified all HTA reports aimed at use in re-imbursement or for disinvestment – 11 full HTA reports, 58 rapid assessments Descriptive quantitative analysis of administrative data and 15 interviews with administrators and payers | Five full HTA reports and 56 rapid assessments “were used for reimbursement decisions”, four full HTAs and two rapid assessments “used for disinvestment decisions and resulted in reduced volumes and expenditure” ([66], p. 77) Total of 67 out of 69 used (97%); two full HTAs no impact; other factors also played a role: in only 45% of reports “the recommendation and decisions totally consistent” ([66], p. 81) | In Austria, policymaking structures facilitate the use of HTA reports, but no mandatory requirement to do so; it is possible the decisions could have been made based on international HTA institutions, but unlikely because, to be used, HTA reports “need primarily to be in German language and they have to be produced within a time period that is strongly linked to the decision-making process” ([66], p. 77) |
Type of impact | Out of 36 studies number reporting on each impact category | Median (range) percentage achieving/claiming this impact in the studies reporting on it |
---|---|---|
Policy/organisation impact | 31 | 35% (5–100%) |
Clinician change/informed practice | 17 | 32% (10–69%) |
A combined category, e.g. policy and clinician impact, or impact on decision-making | 3 | 64% (60–67%) |
Health gain/patient benefit/improved care | 12 | 27% (6–48%) |
Discussion
The Payback Framework
|
Developed by Buxton and Hanney in 1996, the Payback Framework consists of two elements, namely a logic model of the seven stages of research from conceptualisation to impact and five categories to classify the paybacks [14]: • knowledge (e.g. academic publications) • benefits to future research (e.g. training new researchers) • benefits to policy (e.g. information base for clinical policies) • benefits to health and the health system (including cost savings and greater equity) • broader economic benefits (e.g. commercial spin-outs) Two interfaces for interaction between researchers and potential users of research (‘project specification, selection and commissioning’ and ‘dissemination’) and various feedback loops connecting the stages are seen as crucial. The Payback Framework can be applied through surveys, which can be applied to all PIs but have various limitations or to case studies. For the latter, researcher interviews are combined with document analysis and verification of claimed impacts to prepare a detailed case study containing both qualitative and quantitative information; this provides a fuller picture than surveys, but is more labour intensive.
Research Impact Framework (RIF)
Originally developed by Kuruvilla et al. [24] for academics who were interested in measuring and monitoring the impact of their own research, RIF is a ‘light touch’ checklist intended for use by individual researchers who seek to identify and select impacts from their work. Categories include • research-related impacts • policy and practice impacts • service (including health) impacts • ‘societal impact’ (with seven sub-categories) Because of its (intentional) trade-off between comprehensiveness and practicality, it generally produces a less thorough assessment than the Payback Framework and was not designed to be used in formal impact assessment studies by third parties. However, the approach proved to be highly acceptable to those researchers with whom it was applied.
Canadian Academy of Health Sciences (CAHS) Framework
CAHS Framework was developed from the Payback Framework through a multi-stakeholder consensus-building process; it is claimed to be a ‘systems approach’ that takes greater account of non-linear influences [10]. It encourages a careful assessment of context and the subsequent consideration of impacts under five categories: • advancing knowledge (measures of research quality, activity, outreach and structure) • capacity building (developing researchers and research infrastructure) • informing decision-making (decisions about health and healthcare, including public health and social care, decisions about future research investment, and decisions by public and citizens) • health impacts (including health status, determinants of health – including individual risk factors and environmental and social determinants – and health system changes) • economic and social benefits (including commercialisation, cultural outcomes, socioeconomic implications and public understanding of science) For each category, a menu of metrics and measures (66 in total) is offered, and users are encouraged to draw on these flexibly to suit their circumstances. By choosing appropriate sets of indicators, CAHS can be used to track impacts within any of the four ‘pillars’ of health research (basic biomedical, applied clinical, health services and systems, and population health – or within domains that cut across these pillars) and at various levels (individual, institutional, regional, national or international).
Monetisation models
Monetisation models, which are mostly at a relatively early stage of development [68], express returns on research investment in various ways, including as cost savings, the monetary value of net health gains via cost per quality-adjusted life year using metrics such as willingness-to-pay or opportunity cost, and internal rates of return (return on investment as an annual percentage yield). These models draw largely from the economic evaluation literature and differ principally in terms of which costs and benefits (health and non-health) they include and in the valuation of seemingly non-monetary components of the estimation. Prevailing debates on monetisation models of research impact centre on the nature of simplifying assumptions in different models and on the balance between ‘top down’ approaches (which start at a macro level and consider an aggregate health gain, usually at a national level over a specific period, and then consider how far a (national) body of research might have been responsible for it arising) or ‘bottom-up’ approaches (which start with particular research advances, sometimes all the projects in a specific programme, and calculate the health gain from them).
Societal impact assessment (SIA)
Used mainly in the social sciences, SIA emphasises impacts beyond health. Its protagonists distinguish the social relevance of knowledge from its monetised impacts, arguing that the intrinsic value of knowledge may be less significant than the varied and changing social configurations that enable its production, transformation and use. Assessment of SIA usually begins by self-evaluation by a research team of the relationships, interactions and interdependencies that link it to other elements of the research ecosystem (e.g. nature and strength of links with clinicians, policymakers and industry), as well as external peer review of these links. SIA informed the Evaluating Research in Context programme that produced the Sci-Quest model [69] and also the EU-funded SIAMPI (Social Impact Assessment Methods through the study of Productive Interactions) framework [23]. Sci-Quest was described by its authors as a ‘fourth-generation’ approach to impact assessment – the previous three generations having been characterised, respectively, by measurement (e.g. an unenhanced logic model), description (e.g. the narrative accompanying a logic model) and judgement (e.g. an assessment of whether the impact was socially useful or not). Fourth-generation impact assessment, they suggest, is fundamentally a social, political and value-oriented activity and involves reflexivity on the part of researchers to identify and evaluate their own research goals and key relationships [69]. Whilst the approach has many theoretical strengths, it has been criticised for being labour intensive to apply and difficult to systematically compare across projects and programmes.
United Kingdom Research Excellence Framework (REF)
The 2014 REF – an extensive exercise developed by the Higher Education Funding Council for England to assess United Kingdom universities’ research performance – allocated 20% of the total score to research impact [70]. Each institution submitted an impact template describing its strategy and infrastructure for achieving impact, along with several four-page impact case studies, each of which described a programme of research, claimed impacts and supporting evidence. These narratives, which were required to follow a linear and time-bound structure (describing research undertaken between 1993 and 2013, followed by a description of impact occurring between 2008 and 2013) were peer-reviewed by an intersectoral assessment panel representing academia and research users (industry and policymakers). Almost 7000 impact case studies were produced for the 2014 REF; these have been collated in a searchable online database on which further research is currently being undertaken [71]. Independent evaluation by RAND concluded that the narrative form of the REF impact case studies and their peer review by a mixed panel of experts from within and beyond academia had been a robust and fair way of assessing research impact. In its internal review of the REF, the members of Main Panel A, which covered biomedical and health research, noted that “International MPA [Main Panel A] members cautioned against attempts to ‘metricise’ the evaluation of the many superb and well-told narrations describing the evolution of basic discovery to health, economic and societal impact” [70]. |
Author, date, location | Programme/speciality | Reason for exclusion |
---|---|---|
Alberta Heritage Fund for Medical Research, 2003 [80]; Alberta, Canada | Alberta Heritage Fund for Medical Research HTA programme | The number of projects in which any impact (only on policy) was identified was described as ‘most’, which could not be included in the statistical analysis (NB: this is a different study than the one with the same author and same year that was included in the analysis as reference [33]) |
Aymerich et al., 2012 [81], Spain | Network centre for research in epidemiology and public health | Data for impact on reviews and on guidelines/other policies was combined making it impossible to identify the specific policy impact that would have been made by the contribution to guidelines, etc.; the healthcare benefits were potential not actual |
Catalan Agency for HTA and Research, 2006 [82], Catalonia, Spain | TV3 telethon for biomedical research in Catalonia: different speciality each year | Most of the data on impacts seemed to be potential impacts, and the data that were available were presented as total instances not the percentage of projects reporting the impact category |
Cohen et al., 2015 [83], Australia | National Health and Medical Research Council: intervention studies in various programmes | While it was a multi-project assessment covering 70 eligible intervention projects, they came from more than one programme and were not the total number of projects from the programmes of which they were part |
NHS Executive Trent, 1997 [84], United Kingdom | Programme of the Trent Region of the NHS: wide range of basic and applied research | The number of projects in which any impact (on policy and on practice) was identified was described just as ‘<10’, and so not included in the statistical analysis |
Shani et al., 2000 [85], Israel | Israeli Ministry of Health’s Medical Technologies Administration/Israeli Center for Technology in Health Care: HTA | The number of projects in which any impact (only on policy) was identified was described just as ‘86–100’, and so not included in the statistical analysis; also the paper was a commentary rather than a research report |
Stryer et al., 2000 [86], United States of America | Agency for Health Care and Research Quality: Outcomes and effectiveness research | The number of projects in which any impact (on policy and on practice) was identified was described as ‘limited’, and so could not be included in the statistical analysis |