Electronic supplementary material
The online version of this article (doi:10.1186/1472-6963-12-478) contains supplementary material, which is available to authorized users.
The paper reports on data from the University of Oxford, of which all three authors are employees. AMB is Head of the Medical Sciences Division; PVO is a member of the Medical Sciences Division; AO is based in the Social Sciences Division.
PVO conceived the study jointly with AMB, designed the questionnaire, collected and anonymised the data, carried out the coding and quantitative analysis, reviewed the literature on specific indicators, anonymised data from the administrative survey after collection and prior to their release to AO, and drafted the first version of the manuscript. AO reviewed the coding, contributed to planning the paper and structuring the argument, critically reviewed the first draft and contributed to the writing of the second draft, and contributed insights from studies on research assessment, impact measurement, and higher education policy, including staffing. AMB had oversight of the design and data collection for the HEFCE pilot exercise, contributed to the design, planning and co-ordination of the study, and critically reviewed all versions of the manuscript. All authors read and approved the final version of the manuscript.
Funders of medical research the world over are increasingly seeking, in research assessment, to complement traditional output measures of scientific publications with more outcome-based indicators of societal and economic impact. In the United Kingdom, the Higher Education Funding Council for England (HEFCE) developed proposals for the Research Excellence Framework (REF) to allocate public research funding to higher education institutions, inter alia, on the basis of the social and economic impact of their research. In 2010, it conducted a pilot exercise to test these proposals and refine impact indicators and criteria.
The impact indicators proposed in the 2010 REF impact pilot exercise are critically reviewed and appraised using insights from the relevant literature and empirical data collected for the University of Oxford’s REF pilot submission in clinical medicine. The empirical data were gathered from existing administrative sources and an online administrative survey carried out by the university’s Medical Sciences Division among 289 clinical medicine faculty members (48.1% response rate).
The feasibility and scope of measuring research impact in clinical medicine in a given university are assessed. Twenty impact indicators from seven categories proposed by HEFCE are presented; their strengths and limitations are discussed using insights from the relevant biomedical and research policy literature.
While the 2010 pilot exercise has confirmed that the majority of the proposed indicators have some validity, there are significant challenges in operationalising and measuring these indicators reliably, as well as in comparing evidence of research impact across different cases in a standardised manner. It is suggested that the public funding agencies, medical research charities, universities, and the wider medical research community work together to develop more robust methodologies for capturing and describing impact, including more valid and reliable impact indicators.