- Split View
-
Views
-
Cite
Cite
S. Wooding, S. Hanney, M. Buxton, J. Grant, Payback arising from research funding: evaluation of the Arthritis Research Campaign, Rheumatology, Volume 44, Issue 9, September 2005, Pages 1145–1156, https://doi.org/10.1093/rheumatology/keh708
- Share Icon Share
Abstract
Objectives. Using a structured evaluation framework to systematically review and document the outputs and outcomes of research funded by the Arthritis Research Campaign in the early 1990s. To illustrate the strengths and weaknesses of different modes of research funding.
Methods. The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews.
Results. A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced.
Conclusions. There is a wide diversity of research payback. Short focused project grants seem to provide value for money.
Introduction
Measuring the returns from research is of growing importance to research funding organizations [1–4]. This is because they need to demonstrate the benefits arising from their expenditure which (in the case of government agencies) is directly funded from the taxpayer, or (in the case of charitable organizations) directly supported by philanthropic donations and indirectly supported by the taxpayer through fiscal benefits. In addition to demonstrating accountability and good research governance, research funding organizations need to build an evidence base to inform strategic decisions on how to fund research [5].
This paper summarizes the results of an evaluation of research grants funded by the Arthritis Research Campaign (ARC) in the early 1990s. We aim to demonstrate how research outputs and outcomes can be systematically surveyed, and go on to illustrate how this information can provide an evidence base for research funding policy. A methodological paper explaining the evaluation framework adopted for the study has been published [6], as has a two-volume report which sets out our approach, results, conclusions and recommendations in considerable detail, along with descriptions of the case studies that were compiled as part of this evaluation [7, 8].
ARC is the UK's fourth largest medical research charity; its mission states that it is ‘Committed to curing arthritis’. In pursuit of this mission, ARC spends £22 million per year to support research into the causes, treatment and cure of arthritic conditions [9]. ARC supports a wide portfolio of research, including basic, clinical and allied health professional research; this research is supported primarily in a university and institute context. Definitional issues hinder precise classification, but currently about half of ARC funding supports clinical research, one-third supporting basic science. The remainder supports research in various areas, including the allied health professions and education. In 2002, to mark its 65th anniversary, ARC undertook a strategic review of its activities and impacts. The 5-yr strategic plan, Research into Practice [10], concluded ‘there seems to be a gap between the aspirations of people affected by arthritis and the ability of science and academia to meet those aspirations’. The strategy document went on to recommend that ARC needed to ‘instigate a system of rigorous retrospective evaluation on work which has already been completed, with a view to identifying opportunities for development’. This recommendation resulted in the study reported here.
Methods
The approach adopted for the evaluation was based on the ‘payback’ framework developed by the Health Economics Research Group at Brunel University [11–13]. The payback framework has two elements: the first consists of the five payback categories summarized in Table 1; the second is the payback model, shown in Fig. 1. (A detailed description of the framework is provided in a methodological paper [6], which demonstrates not only how the framework was adapted for this particular study but also how it welds together many existing approaches to evaluating aspects of health research outputs and outcomes.)
A | Knowledge |
B | Research targeting, capacity building and absorption: (i) better targeting of future research; (ii) development of research skills, personnel and overall research capacity; (iii) critical capability to utilise appropriately existing research, including that from overseas; (iv) staff development and educational benefits |
C | Informing policy and product development: (i) improved information bases on which to take political and executive decisions; (ii) informing product development |
D | Health and health sector benefits: (i) cost reduction in the delivery of existing services; (ii) qualitative improvements in the process of service delivery; (iii) increased effectiveness of services, e.g. increased health; (iv) equity, e.g. improved allocation of resources at an area level, better targeting and accessibility; (v) revenues gained from intellectual property rights |
E | Broader economic benefits: (i) wider economic benefits from commercial exploitation of innovations arising from R&D; (ii) economic benefits from a healthy workforce and reduction in working days lost |
A | Knowledge |
B | Research targeting, capacity building and absorption: (i) better targeting of future research; (ii) development of research skills, personnel and overall research capacity; (iii) critical capability to utilise appropriately existing research, including that from overseas; (iv) staff development and educational benefits |
C | Informing policy and product development: (i) improved information bases on which to take political and executive decisions; (ii) informing product development |
D | Health and health sector benefits: (i) cost reduction in the delivery of existing services; (ii) qualitative improvements in the process of service delivery; (iii) increased effectiveness of services, e.g. increased health; (iv) equity, e.g. improved allocation of resources at an area level, better targeting and accessibility; (v) revenues gained from intellectual property rights |
E | Broader economic benefits: (i) wider economic benefits from commercial exploitation of innovations arising from R&D; (ii) economic benefits from a healthy workforce and reduction in working days lost |
A | Knowledge |
B | Research targeting, capacity building and absorption: (i) better targeting of future research; (ii) development of research skills, personnel and overall research capacity; (iii) critical capability to utilise appropriately existing research, including that from overseas; (iv) staff development and educational benefits |
C | Informing policy and product development: (i) improved information bases on which to take political and executive decisions; (ii) informing product development |
D | Health and health sector benefits: (i) cost reduction in the delivery of existing services; (ii) qualitative improvements in the process of service delivery; (iii) increased effectiveness of services, e.g. increased health; (iv) equity, e.g. improved allocation of resources at an area level, better targeting and accessibility; (v) revenues gained from intellectual property rights |
E | Broader economic benefits: (i) wider economic benefits from commercial exploitation of innovations arising from R&D; (ii) economic benefits from a healthy workforce and reduction in working days lost |
A | Knowledge |
B | Research targeting, capacity building and absorption: (i) better targeting of future research; (ii) development of research skills, personnel and overall research capacity; (iii) critical capability to utilise appropriately existing research, including that from overseas; (iv) staff development and educational benefits |
C | Informing policy and product development: (i) improved information bases on which to take political and executive decisions; (ii) informing product development |
D | Health and health sector benefits: (i) cost reduction in the delivery of existing services; (ii) qualitative improvements in the process of service delivery; (iii) increased effectiveness of services, e.g. increased health; (iv) equity, e.g. improved allocation of resources at an area level, better targeting and accessibility; (v) revenues gained from intellectual property rights |
E | Broader economic benefits: (i) wider economic benefits from commercial exploitation of innovations arising from R&D; (ii) economic benefits from a healthy workforce and reduction in working days lost |
In previous studies, where the emphasis is on showing the benefits from research, a case study approach has often been used [14, 15] and such an approach has been recommended for future studies of the payback from health research [16, 17]. More widely, there is a long history of applying the case study approach to examine the utilization of research [18, 19]. Here, case studies enable narratives to be told that illuminate how the research funded in the early 1990s was (or was not) translated into practice, and thus each case potentially provides an illustrative example of the long-term outcomes from ARC research. Undertaking a number of case studies makes the evidence more robust and compelling through the replication of observations [19].
Using the evaluation framework, we constructed case studies of 16 research grants selected from 556 possible grants awarded by ARC between 1990 and 1994. With the help of the ARC's Development Committee, we identified candidate case studies to allow us to compare the effect of the mode of research support, the type of research and the bibliometric impact of the principal investigators (PIs). Case-study PIs were then contacted to see if they were willing to participate in the study: all were. Our selection of grants contained: six project grants, three programme grants, three fellowships and four institute grants; six basic grants, eight clinical grants and two allied health professional grants (classified according to the qualifications of the PIs); and nine high-impact PIs and seven mid-impact PIs (classified according to bibliometric indicators). [We were interested in whether grants of the most ‘successful’ researchers produced a spectrum of outputs different from those of less ‘successful’ colleagues. Success was determined from the PI's total publication record from 1990 to 2002. We identified the PIs' papers in the Science Citation Index and added up the journal impact factor (JIF) for each paper. We then ranked the total JIF score to identify the top and middle 10% of PIs. The top 10% were defined as ‘high’ impact, and the middle 10% as ‘mid’ impact.] With sixteen case studies we could not expect them to be representative of all ARC grants in a statistical sense [19]; however, by using a selection matrix we aimed to produce a set of case studies that mirrored the diversity of ARC funding in key dimensions and hence would be a basis for cautious generalization.
Using the information collected from document and literature reviews, semistructured key informant interviews and bibliometric analysis, each of the 16 cases was written up and published as a narrative organized according to the structure provided by the payback framework [8]. Use of a common structure facilitated comparative analysis, allowing us, for example, to identify the factors apparently associated with the successful translation of research [19]. We employed two approaches to compare across cases. The first was based on a qualitative assessment of the case studies. The second involved a novel method of scoring the case studies on the five payback categories.
We carried out the scoring process during a 2-day workshop of the research team, using a process developed for reviewing evidence and testing agreement in the formulation of clinical guidelines [20]. Team members were asked to individually score the level of payback from each case study in each category. Scoring of each of five categories was undertaken on a scale of 1–9 (where 1 was the lowest and 9 the highest score). The scores from the first round were analysed and sheets showing their distribution were circulated to the team. Following this, there was discussion about scoring discrepancies in order to resolve any misunderstandings between the team about the evidence on the outputs from the case. The team members then had the opportunity to rescore all of the case studies for that payback category, so providing a set of second-round scores. The median of these second-round scores was used for the analysis presented here.
Results
In this section, we catalogue the research paybacks identified from the 16 case studies, as summarized in Table 2. We go on to use this information to assess the effectiveness of different ways of funding research.
Case . | Knowledge production . | Research targeting, capacity building . | Informing policy and product development . | Health benefits . | Broader economic benefits . |
---|---|---|---|---|---|
A | 6 papers, receiving a total of 5 citations per year. The RCT concluded that an exercise class is more clinically effective than traditional GP management, regardless of patient preference, and is more cost-effective. Some papers describe the main findings, others discuss key methodological issues. | Further studies by the PI applied lessons from this study. The PI was involved also in the MRC-sponsored >£2 million UK BEAM Trial, which draws heavily on the study. An interviewee thought that the project's success helped encourage the ARC to support AHP research. Provided research training for team. 1 PhD. | The work was not cited in any national clinical guidelines or equivalents. But it was cited both in a number of reviews, receiving high-quality assessments, and in a task group report. At a local level it is influencing policy in various places as part of a general move towards more active management of back pain. | The study showed that exercise is more clinically effective than GP management, with some improvements on disability questionnaire scores and back pain scales with less call on health-care resources after the intervention. It is being applied in various places but sometimes to small numbers. | The study showed considerably fewer days off work in the period following the intervention for those in the intervention group than in the control group. Therefore, it would contribute to a healthy workforce. This is assumed to be happening in adoption sites but has not been assessed. |
B | 6 papers, receiving a total of 8 citations per year. The research showed that hip OA is a condition that arises through an interaction of a predisposition to the disease and mechanical insults to the hip. Hip OA is therefore an occupational disease in men whose work entails frequent heavy lifting. | The methods developed for the UK study were applied to a Japanese study on occupational lifting. All three of the co-applicants had chairs, and one of the two research nurses continued to work in the unit from where the work was undertaken. | Three papers were cited in three systematic reviews, each receiving high-quality assessment. One paper was cited in a Dutch clinical guideline and by the Industrial Injuries Advisory Council (IIAC) Assessment on whether hip OA in farmers was a prescribed disease. | Adoption of IIAC recommendation will result in monetary benefit to farmers suffering from OA, at a cost to the taxpayer, improving economic equity. | |
C | 13 papers, receiving a total of 118 citations per year. A detailed understanding of the mechanisms of matrix metalloproteinase (MMP) inhibition by tissue inhibitors of matrix metalloproteinases (TIMPs), coupled with information on the domain structure of the TIMP protein. | PI's industrial collaborator is now a senior research manager with Celltech. Grant supported assistant's PhD: the foundation of his career in pharmaceutical industry. Reagents produced have been sent out to over 100 researchers. MMPs initially seen as promising target, leading to drug development. | At least one drug development programme in industry continues, but no registered products have yet been developed. | No evidence of health benefits. | No evidence of broader economic benefits. |
D | 11 papers, receiving a total of 17 citations per year. Increased understanding of how the immune system deals with chlamydial infection and the causes of reactive arthritis. Three of the papers were peripheral to the grant subject and do not directly concern chlamydial infection or reactive arthritis. | The PI's career has continued to advance, but outside the field of arthritis. PI developed a mouse model of chlamydial infection, a method for identifying novel vaccine targets and possible candidate peptide for a vaccine against chlamydia, but this work was not developed further. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
E | 44 papers, receiving a total of 73 citations per year. Referee commented in the programme renewal that ‘the major criticism is that no blinding insights had come out of the programme’. Site visitors felt that the team should publish in more international journals. PI contributed to two books. | Two postdoctoral researchers established themselves as researchers. A large number of current clinical leaders and several basic scientists trained on the programme. PI became leader in health services research. | Citation in one European guideline. No evidence of impact on policy. Some commercial interest in biomechanical processes in OA currently being trialled (although not directly linked to PI's earlier effort to interest footwear manufacturers in his findings). | No current benefit beyond small number of patients treated with the team's new taping technique. Subsequent research by team into screening using genetic and biochemical markers could lead to more effective and targeted treatment. | No evidence of broader economic benefits. |
F | 23 papers, receiving a total of 32 citations per year. Showed that collagen X was a definite marker for endochondral ossification and identified specific mutations in collagen X that cause chondrodysplasia type Schmid. | Two research assistants obtained their PhDs because of the programme. Two of the postdoctoral researchers obtained positions at the university, while the third was offered a job at a prestigious organization in the USA. | The identification of specific mutations in collagen X was used to develop a diagnostic test for chondrodysplasia type Schmid. Currently, this test is run as a clinical service for the European skeletal dysplasia network. | Confirmation of chondrodysplasia type Schmid improves the management of care, and specifically prevents surgical intervention that may be applicable to other forms of chondrodysplasia. | No evidence of broader economic benefits. |
G | 32 papers, receiving a total of 56 citations per year. Showed that complement plays an important role in processing immune complexes and this may be related to SLE. The importance of anti-C1q antibodies in patients with SLE was also demonstrated. | PI established a successful research career and is in the process of setting up a medical school. Junior research assistant completed his PhD and is now an MRC-supported senior research fellow. The other research assistants continue to work in the field of rheumatology. | The development of a diagnostic test that enabled the measuring of antibodies to C1q in SLE patients. The test is still used today at the Royal Postgraduate Medical School. | Prevalence of SLE is low. The test allows diagnostics of one form of SLE (C1q deficiency). | No evidence of broader economic benefits. |
H | 20 papers, receiving a total of 108 citations per year. The PI and his team showed that the rheumatoid process appears not to be grossly abnormal, however its consequences are; it is the interaction between T cells and stromal cells that causes the problem. | MRC Centre of Immune Regulation has a major focus on rheumatology as a result of the fellowship. Fellowship initiated several subsequent project and programme grants. PI is now head of department. Academic collaborators are now leading scientists in the field. 7 PhDs. | The observation that RGD peptides were triggering apoptosis informed the development of KGD peptides, a class of drugs used to treat deep-vein thrombosis, but not useful for arthritis. Established long-term collaboration with industrial partners. | No evidence of health benefits. | No evidence of broader economic benefits. |
I | 11 papers, receiving a total of 32 citations per year. Although the research hypothesis was not addressed in the timeframe of the fellowship, the research did contribute to a number of methodological developments. | The PI completed an MD and later become a consultant in rheumatology and general medicine. The lessons from the fellowship were applied to a subsequent project grant, assessing the effect of exercise on people with OA subsequently informed British and European guidelines. | A grading system for radiographic assessment of OA was developed and used in graduate clinical education. | It is now common for clinicians to recommend the physical training of muscles for people with OA. | No evidence of broader economic benefits. |
J | 13 papers, receiving a total of 10 citations per year. These publications showed that proprioception plays an important role in ACLD deficiency, and that a rehabilitation programme to enhance proprioception was more effective than traditional muscle strengthening and exercise. | The PI was awarded a PhD on the basis of the work. | Informed management of ACLD. | Small benefit due to improved management of ACLD. | Unquantified return in reduction of days off work. |
K | 7 papers, receiving a total of 29 citations per year. Showed treatment with aspirin and heparin gives higher rate of live births in women suffering recurrent miscarriage associated with antiphospholipid antibodies than aspirin alone. | The trial generated a number of further lines of research, including the effect of antiphospholipid antibodies on in vitro fertilization. The research registrar was awarded a MD. The PI established an international profile. | The British Royal College of Obstetricians and Gynaecologists guideline and the US, Dutch and Australian guidelines all recommend combination therapy of aspirin plus heparin and all cite the trial as the evidence for this recommendation. | The trial led to the conclusion that a combined treatment increased the chances of a live birth for women who had tested positive for antiphospholipid antibodies. A number of these women would not have completed a pregnancy successfully without this treatment. | The treatment is complex and potentially expensive. However, as the number of live births increases, fewer costs are involved because of a decrease in repetition of pregnancies. |
L | 41 papers, receiving a total of 330 citations per year. The paper describing the first clinical trial demonstrated the efficacy and safety of treatment. The results of a 4-centre double blind trial of 73 patients published in 1994 confirmed these findings. | Work promoted research in the area and promoted the use of biologicals in therapy. In wider RA research, work led to a paradigm shift, revealing that, in RA's complicated multistep process, one single molecule can have profound effects on pathogenesis. 3 PhDs and MDs. | The research led to the development of three drugs which have been licensed in the UK. Anti-TNF treatment has been included in a technology appraisal from the NHS National Institute for Clinical Excellence and adopted by international consensus groups in Europe and the USA. | The potentially considerable benefits to RA patients of anti-TNF treatment have been limited by low take-up due to perceived high cost of treatment. | Broader economic benefits have resulted from commercial exploitation of anti-TNF drugs. |
M | 15 papers, receiving a total of 20 citations per year. The research identified a number of specific genetic regions and markers for RA. By the end of 1996, 170 sibling pair families had been identified, confirming linkage to HLA. | No evidence of research targeting or capacity building. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
N | 16 papers, receiving a total of 75 citations per year. In the institute's submission for a new core grant in 1996, one reviewer noted that the work led to ‘few publications in first class journals’. The PI suggested that in terms of papers the impact had been low. | The PI argued that in terms of papers the impact had been low, but the work had had a strategic influence on the field. The PI left the institute in 1994, which heralded the disintegration of the research team. He has subsequently refocused his research interests. 3 PhDs. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
O | 17 papers, receiving a total of 21 citations per year. The core publications describe the identification of dinucleotide repeat polymorphism as the α-1-antitrypsin locus and give a summary of the first 100 multi-case RA families with affected sibling pairs. | The fellowship benefited the PI's career and RA genetics capacity at the institute. 1 PhD. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
P | 27 papers, receiving a total of 40 citations per year. NOAR has provided groundbreaking work in understanding how common RA is and the factors that are associated with it. | The PI, co-applicants and lead rheumatologist have become leaders in their field. Six people have obtained PhD/MDs as a result of their work on NOAR. NOAR has increased research capacity in Norfolk. The Danes have set up a similar registry, based on NOAR. | The incidence and prevalence figures from NOAR are widely cited in clinical guidelines. Recent data from NOAR on the benefit of early referral. Early referral of RA patients is one of the two quality standards set for rheumatology by the Royal College of Physicians. | No evidence of health benefits. | No evidence of broader economic benefits. |
Case . | Knowledge production . | Research targeting, capacity building . | Informing policy and product development . | Health benefits . | Broader economic benefits . |
---|---|---|---|---|---|
A | 6 papers, receiving a total of 5 citations per year. The RCT concluded that an exercise class is more clinically effective than traditional GP management, regardless of patient preference, and is more cost-effective. Some papers describe the main findings, others discuss key methodological issues. | Further studies by the PI applied lessons from this study. The PI was involved also in the MRC-sponsored >£2 million UK BEAM Trial, which draws heavily on the study. An interviewee thought that the project's success helped encourage the ARC to support AHP research. Provided research training for team. 1 PhD. | The work was not cited in any national clinical guidelines or equivalents. But it was cited both in a number of reviews, receiving high-quality assessments, and in a task group report. At a local level it is influencing policy in various places as part of a general move towards more active management of back pain. | The study showed that exercise is more clinically effective than GP management, with some improvements on disability questionnaire scores and back pain scales with less call on health-care resources after the intervention. It is being applied in various places but sometimes to small numbers. | The study showed considerably fewer days off work in the period following the intervention for those in the intervention group than in the control group. Therefore, it would contribute to a healthy workforce. This is assumed to be happening in adoption sites but has not been assessed. |
B | 6 papers, receiving a total of 8 citations per year. The research showed that hip OA is a condition that arises through an interaction of a predisposition to the disease and mechanical insults to the hip. Hip OA is therefore an occupational disease in men whose work entails frequent heavy lifting. | The methods developed for the UK study were applied to a Japanese study on occupational lifting. All three of the co-applicants had chairs, and one of the two research nurses continued to work in the unit from where the work was undertaken. | Three papers were cited in three systematic reviews, each receiving high-quality assessment. One paper was cited in a Dutch clinical guideline and by the Industrial Injuries Advisory Council (IIAC) Assessment on whether hip OA in farmers was a prescribed disease. | Adoption of IIAC recommendation will result in monetary benefit to farmers suffering from OA, at a cost to the taxpayer, improving economic equity. | |
C | 13 papers, receiving a total of 118 citations per year. A detailed understanding of the mechanisms of matrix metalloproteinase (MMP) inhibition by tissue inhibitors of matrix metalloproteinases (TIMPs), coupled with information on the domain structure of the TIMP protein. | PI's industrial collaborator is now a senior research manager with Celltech. Grant supported assistant's PhD: the foundation of his career in pharmaceutical industry. Reagents produced have been sent out to over 100 researchers. MMPs initially seen as promising target, leading to drug development. | At least one drug development programme in industry continues, but no registered products have yet been developed. | No evidence of health benefits. | No evidence of broader economic benefits. |
D | 11 papers, receiving a total of 17 citations per year. Increased understanding of how the immune system deals with chlamydial infection and the causes of reactive arthritis. Three of the papers were peripheral to the grant subject and do not directly concern chlamydial infection or reactive arthritis. | The PI's career has continued to advance, but outside the field of arthritis. PI developed a mouse model of chlamydial infection, a method for identifying novel vaccine targets and possible candidate peptide for a vaccine against chlamydia, but this work was not developed further. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
E | 44 papers, receiving a total of 73 citations per year. Referee commented in the programme renewal that ‘the major criticism is that no blinding insights had come out of the programme’. Site visitors felt that the team should publish in more international journals. PI contributed to two books. | Two postdoctoral researchers established themselves as researchers. A large number of current clinical leaders and several basic scientists trained on the programme. PI became leader in health services research. | Citation in one European guideline. No evidence of impact on policy. Some commercial interest in biomechanical processes in OA currently being trialled (although not directly linked to PI's earlier effort to interest footwear manufacturers in his findings). | No current benefit beyond small number of patients treated with the team's new taping technique. Subsequent research by team into screening using genetic and biochemical markers could lead to more effective and targeted treatment. | No evidence of broader economic benefits. |
F | 23 papers, receiving a total of 32 citations per year. Showed that collagen X was a definite marker for endochondral ossification and identified specific mutations in collagen X that cause chondrodysplasia type Schmid. | Two research assistants obtained their PhDs because of the programme. Two of the postdoctoral researchers obtained positions at the university, while the third was offered a job at a prestigious organization in the USA. | The identification of specific mutations in collagen X was used to develop a diagnostic test for chondrodysplasia type Schmid. Currently, this test is run as a clinical service for the European skeletal dysplasia network. | Confirmation of chondrodysplasia type Schmid improves the management of care, and specifically prevents surgical intervention that may be applicable to other forms of chondrodysplasia. | No evidence of broader economic benefits. |
G | 32 papers, receiving a total of 56 citations per year. Showed that complement plays an important role in processing immune complexes and this may be related to SLE. The importance of anti-C1q antibodies in patients with SLE was also demonstrated. | PI established a successful research career and is in the process of setting up a medical school. Junior research assistant completed his PhD and is now an MRC-supported senior research fellow. The other research assistants continue to work in the field of rheumatology. | The development of a diagnostic test that enabled the measuring of antibodies to C1q in SLE patients. The test is still used today at the Royal Postgraduate Medical School. | Prevalence of SLE is low. The test allows diagnostics of one form of SLE (C1q deficiency). | No evidence of broader economic benefits. |
H | 20 papers, receiving a total of 108 citations per year. The PI and his team showed that the rheumatoid process appears not to be grossly abnormal, however its consequences are; it is the interaction between T cells and stromal cells that causes the problem. | MRC Centre of Immune Regulation has a major focus on rheumatology as a result of the fellowship. Fellowship initiated several subsequent project and programme grants. PI is now head of department. Academic collaborators are now leading scientists in the field. 7 PhDs. | The observation that RGD peptides were triggering apoptosis informed the development of KGD peptides, a class of drugs used to treat deep-vein thrombosis, but not useful for arthritis. Established long-term collaboration with industrial partners. | No evidence of health benefits. | No evidence of broader economic benefits. |
I | 11 papers, receiving a total of 32 citations per year. Although the research hypothesis was not addressed in the timeframe of the fellowship, the research did contribute to a number of methodological developments. | The PI completed an MD and later become a consultant in rheumatology and general medicine. The lessons from the fellowship were applied to a subsequent project grant, assessing the effect of exercise on people with OA subsequently informed British and European guidelines. | A grading system for radiographic assessment of OA was developed and used in graduate clinical education. | It is now common for clinicians to recommend the physical training of muscles for people with OA. | No evidence of broader economic benefits. |
J | 13 papers, receiving a total of 10 citations per year. These publications showed that proprioception plays an important role in ACLD deficiency, and that a rehabilitation programme to enhance proprioception was more effective than traditional muscle strengthening and exercise. | The PI was awarded a PhD on the basis of the work. | Informed management of ACLD. | Small benefit due to improved management of ACLD. | Unquantified return in reduction of days off work. |
K | 7 papers, receiving a total of 29 citations per year. Showed treatment with aspirin and heparin gives higher rate of live births in women suffering recurrent miscarriage associated with antiphospholipid antibodies than aspirin alone. | The trial generated a number of further lines of research, including the effect of antiphospholipid antibodies on in vitro fertilization. The research registrar was awarded a MD. The PI established an international profile. | The British Royal College of Obstetricians and Gynaecologists guideline and the US, Dutch and Australian guidelines all recommend combination therapy of aspirin plus heparin and all cite the trial as the evidence for this recommendation. | The trial led to the conclusion that a combined treatment increased the chances of a live birth for women who had tested positive for antiphospholipid antibodies. A number of these women would not have completed a pregnancy successfully without this treatment. | The treatment is complex and potentially expensive. However, as the number of live births increases, fewer costs are involved because of a decrease in repetition of pregnancies. |
L | 41 papers, receiving a total of 330 citations per year. The paper describing the first clinical trial demonstrated the efficacy and safety of treatment. The results of a 4-centre double blind trial of 73 patients published in 1994 confirmed these findings. | Work promoted research in the area and promoted the use of biologicals in therapy. In wider RA research, work led to a paradigm shift, revealing that, in RA's complicated multistep process, one single molecule can have profound effects on pathogenesis. 3 PhDs and MDs. | The research led to the development of three drugs which have been licensed in the UK. Anti-TNF treatment has been included in a technology appraisal from the NHS National Institute for Clinical Excellence and adopted by international consensus groups in Europe and the USA. | The potentially considerable benefits to RA patients of anti-TNF treatment have been limited by low take-up due to perceived high cost of treatment. | Broader economic benefits have resulted from commercial exploitation of anti-TNF drugs. |
M | 15 papers, receiving a total of 20 citations per year. The research identified a number of specific genetic regions and markers for RA. By the end of 1996, 170 sibling pair families had been identified, confirming linkage to HLA. | No evidence of research targeting or capacity building. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
N | 16 papers, receiving a total of 75 citations per year. In the institute's submission for a new core grant in 1996, one reviewer noted that the work led to ‘few publications in first class journals’. The PI suggested that in terms of papers the impact had been low. | The PI argued that in terms of papers the impact had been low, but the work had had a strategic influence on the field. The PI left the institute in 1994, which heralded the disintegration of the research team. He has subsequently refocused his research interests. 3 PhDs. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
O | 17 papers, receiving a total of 21 citations per year. The core publications describe the identification of dinucleotide repeat polymorphism as the α-1-antitrypsin locus and give a summary of the first 100 multi-case RA families with affected sibling pairs. | The fellowship benefited the PI's career and RA genetics capacity at the institute. 1 PhD. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
P | 27 papers, receiving a total of 40 citations per year. NOAR has provided groundbreaking work in understanding how common RA is and the factors that are associated with it. | The PI, co-applicants and lead rheumatologist have become leaders in their field. Six people have obtained PhD/MDs as a result of their work on NOAR. NOAR has increased research capacity in Norfolk. The Danes have set up a similar registry, based on NOAR. | The incidence and prevalence figures from NOAR are widely cited in clinical guidelines. Recent data from NOAR on the benefit of early referral. Early referral of RA patients is one of the two quality standards set for rheumatology by the Royal College of Physicians. | No evidence of health benefits. | No evidence of broader economic benefits. |
GP, general practitioner; NOAR, Norfolk Arthritis Register; RCT, randomized controlled trial; ACLD, Anterior Cruciate Ligament Deficiency, a disorder of the knee.
Case . | Knowledge production . | Research targeting, capacity building . | Informing policy and product development . | Health benefits . | Broader economic benefits . |
---|---|---|---|---|---|
A | 6 papers, receiving a total of 5 citations per year. The RCT concluded that an exercise class is more clinically effective than traditional GP management, regardless of patient preference, and is more cost-effective. Some papers describe the main findings, others discuss key methodological issues. | Further studies by the PI applied lessons from this study. The PI was involved also in the MRC-sponsored >£2 million UK BEAM Trial, which draws heavily on the study. An interviewee thought that the project's success helped encourage the ARC to support AHP research. Provided research training for team. 1 PhD. | The work was not cited in any national clinical guidelines or equivalents. But it was cited both in a number of reviews, receiving high-quality assessments, and in a task group report. At a local level it is influencing policy in various places as part of a general move towards more active management of back pain. | The study showed that exercise is more clinically effective than GP management, with some improvements on disability questionnaire scores and back pain scales with less call on health-care resources after the intervention. It is being applied in various places but sometimes to small numbers. | The study showed considerably fewer days off work in the period following the intervention for those in the intervention group than in the control group. Therefore, it would contribute to a healthy workforce. This is assumed to be happening in adoption sites but has not been assessed. |
B | 6 papers, receiving a total of 8 citations per year. The research showed that hip OA is a condition that arises through an interaction of a predisposition to the disease and mechanical insults to the hip. Hip OA is therefore an occupational disease in men whose work entails frequent heavy lifting. | The methods developed for the UK study were applied to a Japanese study on occupational lifting. All three of the co-applicants had chairs, and one of the two research nurses continued to work in the unit from where the work was undertaken. | Three papers were cited in three systematic reviews, each receiving high-quality assessment. One paper was cited in a Dutch clinical guideline and by the Industrial Injuries Advisory Council (IIAC) Assessment on whether hip OA in farmers was a prescribed disease. | Adoption of IIAC recommendation will result in monetary benefit to farmers suffering from OA, at a cost to the taxpayer, improving economic equity. | |
C | 13 papers, receiving a total of 118 citations per year. A detailed understanding of the mechanisms of matrix metalloproteinase (MMP) inhibition by tissue inhibitors of matrix metalloproteinases (TIMPs), coupled with information on the domain structure of the TIMP protein. | PI's industrial collaborator is now a senior research manager with Celltech. Grant supported assistant's PhD: the foundation of his career in pharmaceutical industry. Reagents produced have been sent out to over 100 researchers. MMPs initially seen as promising target, leading to drug development. | At least one drug development programme in industry continues, but no registered products have yet been developed. | No evidence of health benefits. | No evidence of broader economic benefits. |
D | 11 papers, receiving a total of 17 citations per year. Increased understanding of how the immune system deals with chlamydial infection and the causes of reactive arthritis. Three of the papers were peripheral to the grant subject and do not directly concern chlamydial infection or reactive arthritis. | The PI's career has continued to advance, but outside the field of arthritis. PI developed a mouse model of chlamydial infection, a method for identifying novel vaccine targets and possible candidate peptide for a vaccine against chlamydia, but this work was not developed further. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
E | 44 papers, receiving a total of 73 citations per year. Referee commented in the programme renewal that ‘the major criticism is that no blinding insights had come out of the programme’. Site visitors felt that the team should publish in more international journals. PI contributed to two books. | Two postdoctoral researchers established themselves as researchers. A large number of current clinical leaders and several basic scientists trained on the programme. PI became leader in health services research. | Citation in one European guideline. No evidence of impact on policy. Some commercial interest in biomechanical processes in OA currently being trialled (although not directly linked to PI's earlier effort to interest footwear manufacturers in his findings). | No current benefit beyond small number of patients treated with the team's new taping technique. Subsequent research by team into screening using genetic and biochemical markers could lead to more effective and targeted treatment. | No evidence of broader economic benefits. |
F | 23 papers, receiving a total of 32 citations per year. Showed that collagen X was a definite marker for endochondral ossification and identified specific mutations in collagen X that cause chondrodysplasia type Schmid. | Two research assistants obtained their PhDs because of the programme. Two of the postdoctoral researchers obtained positions at the university, while the third was offered a job at a prestigious organization in the USA. | The identification of specific mutations in collagen X was used to develop a diagnostic test for chondrodysplasia type Schmid. Currently, this test is run as a clinical service for the European skeletal dysplasia network. | Confirmation of chondrodysplasia type Schmid improves the management of care, and specifically prevents surgical intervention that may be applicable to other forms of chondrodysplasia. | No evidence of broader economic benefits. |
G | 32 papers, receiving a total of 56 citations per year. Showed that complement plays an important role in processing immune complexes and this may be related to SLE. The importance of anti-C1q antibodies in patients with SLE was also demonstrated. | PI established a successful research career and is in the process of setting up a medical school. Junior research assistant completed his PhD and is now an MRC-supported senior research fellow. The other research assistants continue to work in the field of rheumatology. | The development of a diagnostic test that enabled the measuring of antibodies to C1q in SLE patients. The test is still used today at the Royal Postgraduate Medical School. | Prevalence of SLE is low. The test allows diagnostics of one form of SLE (C1q deficiency). | No evidence of broader economic benefits. |
H | 20 papers, receiving a total of 108 citations per year. The PI and his team showed that the rheumatoid process appears not to be grossly abnormal, however its consequences are; it is the interaction between T cells and stromal cells that causes the problem. | MRC Centre of Immune Regulation has a major focus on rheumatology as a result of the fellowship. Fellowship initiated several subsequent project and programme grants. PI is now head of department. Academic collaborators are now leading scientists in the field. 7 PhDs. | The observation that RGD peptides were triggering apoptosis informed the development of KGD peptides, a class of drugs used to treat deep-vein thrombosis, but not useful for arthritis. Established long-term collaboration with industrial partners. | No evidence of health benefits. | No evidence of broader economic benefits. |
I | 11 papers, receiving a total of 32 citations per year. Although the research hypothesis was not addressed in the timeframe of the fellowship, the research did contribute to a number of methodological developments. | The PI completed an MD and later become a consultant in rheumatology and general medicine. The lessons from the fellowship were applied to a subsequent project grant, assessing the effect of exercise on people with OA subsequently informed British and European guidelines. | A grading system for radiographic assessment of OA was developed and used in graduate clinical education. | It is now common for clinicians to recommend the physical training of muscles for people with OA. | No evidence of broader economic benefits. |
J | 13 papers, receiving a total of 10 citations per year. These publications showed that proprioception plays an important role in ACLD deficiency, and that a rehabilitation programme to enhance proprioception was more effective than traditional muscle strengthening and exercise. | The PI was awarded a PhD on the basis of the work. | Informed management of ACLD. | Small benefit due to improved management of ACLD. | Unquantified return in reduction of days off work. |
K | 7 papers, receiving a total of 29 citations per year. Showed treatment with aspirin and heparin gives higher rate of live births in women suffering recurrent miscarriage associated with antiphospholipid antibodies than aspirin alone. | The trial generated a number of further lines of research, including the effect of antiphospholipid antibodies on in vitro fertilization. The research registrar was awarded a MD. The PI established an international profile. | The British Royal College of Obstetricians and Gynaecologists guideline and the US, Dutch and Australian guidelines all recommend combination therapy of aspirin plus heparin and all cite the trial as the evidence for this recommendation. | The trial led to the conclusion that a combined treatment increased the chances of a live birth for women who had tested positive for antiphospholipid antibodies. A number of these women would not have completed a pregnancy successfully without this treatment. | The treatment is complex and potentially expensive. However, as the number of live births increases, fewer costs are involved because of a decrease in repetition of pregnancies. |
L | 41 papers, receiving a total of 330 citations per year. The paper describing the first clinical trial demonstrated the efficacy and safety of treatment. The results of a 4-centre double blind trial of 73 patients published in 1994 confirmed these findings. | Work promoted research in the area and promoted the use of biologicals in therapy. In wider RA research, work led to a paradigm shift, revealing that, in RA's complicated multistep process, one single molecule can have profound effects on pathogenesis. 3 PhDs and MDs. | The research led to the development of three drugs which have been licensed in the UK. Anti-TNF treatment has been included in a technology appraisal from the NHS National Institute for Clinical Excellence and adopted by international consensus groups in Europe and the USA. | The potentially considerable benefits to RA patients of anti-TNF treatment have been limited by low take-up due to perceived high cost of treatment. | Broader economic benefits have resulted from commercial exploitation of anti-TNF drugs. |
M | 15 papers, receiving a total of 20 citations per year. The research identified a number of specific genetic regions and markers for RA. By the end of 1996, 170 sibling pair families had been identified, confirming linkage to HLA. | No evidence of research targeting or capacity building. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
N | 16 papers, receiving a total of 75 citations per year. In the institute's submission for a new core grant in 1996, one reviewer noted that the work led to ‘few publications in first class journals’. The PI suggested that in terms of papers the impact had been low. | The PI argued that in terms of papers the impact had been low, but the work had had a strategic influence on the field. The PI left the institute in 1994, which heralded the disintegration of the research team. He has subsequently refocused his research interests. 3 PhDs. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
O | 17 papers, receiving a total of 21 citations per year. The core publications describe the identification of dinucleotide repeat polymorphism as the α-1-antitrypsin locus and give a summary of the first 100 multi-case RA families with affected sibling pairs. | The fellowship benefited the PI's career and RA genetics capacity at the institute. 1 PhD. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
P | 27 papers, receiving a total of 40 citations per year. NOAR has provided groundbreaking work in understanding how common RA is and the factors that are associated with it. | The PI, co-applicants and lead rheumatologist have become leaders in their field. Six people have obtained PhD/MDs as a result of their work on NOAR. NOAR has increased research capacity in Norfolk. The Danes have set up a similar registry, based on NOAR. | The incidence and prevalence figures from NOAR are widely cited in clinical guidelines. Recent data from NOAR on the benefit of early referral. Early referral of RA patients is one of the two quality standards set for rheumatology by the Royal College of Physicians. | No evidence of health benefits. | No evidence of broader economic benefits. |
Case . | Knowledge production . | Research targeting, capacity building . | Informing policy and product development . | Health benefits . | Broader economic benefits . |
---|---|---|---|---|---|
A | 6 papers, receiving a total of 5 citations per year. The RCT concluded that an exercise class is more clinically effective than traditional GP management, regardless of patient preference, and is more cost-effective. Some papers describe the main findings, others discuss key methodological issues. | Further studies by the PI applied lessons from this study. The PI was involved also in the MRC-sponsored >£2 million UK BEAM Trial, which draws heavily on the study. An interviewee thought that the project's success helped encourage the ARC to support AHP research. Provided research training for team. 1 PhD. | The work was not cited in any national clinical guidelines or equivalents. But it was cited both in a number of reviews, receiving high-quality assessments, and in a task group report. At a local level it is influencing policy in various places as part of a general move towards more active management of back pain. | The study showed that exercise is more clinically effective than GP management, with some improvements on disability questionnaire scores and back pain scales with less call on health-care resources after the intervention. It is being applied in various places but sometimes to small numbers. | The study showed considerably fewer days off work in the period following the intervention for those in the intervention group than in the control group. Therefore, it would contribute to a healthy workforce. This is assumed to be happening in adoption sites but has not been assessed. |
B | 6 papers, receiving a total of 8 citations per year. The research showed that hip OA is a condition that arises through an interaction of a predisposition to the disease and mechanical insults to the hip. Hip OA is therefore an occupational disease in men whose work entails frequent heavy lifting. | The methods developed for the UK study were applied to a Japanese study on occupational lifting. All three of the co-applicants had chairs, and one of the two research nurses continued to work in the unit from where the work was undertaken. | Three papers were cited in three systematic reviews, each receiving high-quality assessment. One paper was cited in a Dutch clinical guideline and by the Industrial Injuries Advisory Council (IIAC) Assessment on whether hip OA in farmers was a prescribed disease. | Adoption of IIAC recommendation will result in monetary benefit to farmers suffering from OA, at a cost to the taxpayer, improving economic equity. | |
C | 13 papers, receiving a total of 118 citations per year. A detailed understanding of the mechanisms of matrix metalloproteinase (MMP) inhibition by tissue inhibitors of matrix metalloproteinases (TIMPs), coupled with information on the domain structure of the TIMP protein. | PI's industrial collaborator is now a senior research manager with Celltech. Grant supported assistant's PhD: the foundation of his career in pharmaceutical industry. Reagents produced have been sent out to over 100 researchers. MMPs initially seen as promising target, leading to drug development. | At least one drug development programme in industry continues, but no registered products have yet been developed. | No evidence of health benefits. | No evidence of broader economic benefits. |
D | 11 papers, receiving a total of 17 citations per year. Increased understanding of how the immune system deals with chlamydial infection and the causes of reactive arthritis. Three of the papers were peripheral to the grant subject and do not directly concern chlamydial infection or reactive arthritis. | The PI's career has continued to advance, but outside the field of arthritis. PI developed a mouse model of chlamydial infection, a method for identifying novel vaccine targets and possible candidate peptide for a vaccine against chlamydia, but this work was not developed further. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
E | 44 papers, receiving a total of 73 citations per year. Referee commented in the programme renewal that ‘the major criticism is that no blinding insights had come out of the programme’. Site visitors felt that the team should publish in more international journals. PI contributed to two books. | Two postdoctoral researchers established themselves as researchers. A large number of current clinical leaders and several basic scientists trained on the programme. PI became leader in health services research. | Citation in one European guideline. No evidence of impact on policy. Some commercial interest in biomechanical processes in OA currently being trialled (although not directly linked to PI's earlier effort to interest footwear manufacturers in his findings). | No current benefit beyond small number of patients treated with the team's new taping technique. Subsequent research by team into screening using genetic and biochemical markers could lead to more effective and targeted treatment. | No evidence of broader economic benefits. |
F | 23 papers, receiving a total of 32 citations per year. Showed that collagen X was a definite marker for endochondral ossification and identified specific mutations in collagen X that cause chondrodysplasia type Schmid. | Two research assistants obtained their PhDs because of the programme. Two of the postdoctoral researchers obtained positions at the university, while the third was offered a job at a prestigious organization in the USA. | The identification of specific mutations in collagen X was used to develop a diagnostic test for chondrodysplasia type Schmid. Currently, this test is run as a clinical service for the European skeletal dysplasia network. | Confirmation of chondrodysplasia type Schmid improves the management of care, and specifically prevents surgical intervention that may be applicable to other forms of chondrodysplasia. | No evidence of broader economic benefits. |
G | 32 papers, receiving a total of 56 citations per year. Showed that complement plays an important role in processing immune complexes and this may be related to SLE. The importance of anti-C1q antibodies in patients with SLE was also demonstrated. | PI established a successful research career and is in the process of setting up a medical school. Junior research assistant completed his PhD and is now an MRC-supported senior research fellow. The other research assistants continue to work in the field of rheumatology. | The development of a diagnostic test that enabled the measuring of antibodies to C1q in SLE patients. The test is still used today at the Royal Postgraduate Medical School. | Prevalence of SLE is low. The test allows diagnostics of one form of SLE (C1q deficiency). | No evidence of broader economic benefits. |
H | 20 papers, receiving a total of 108 citations per year. The PI and his team showed that the rheumatoid process appears not to be grossly abnormal, however its consequences are; it is the interaction between T cells and stromal cells that causes the problem. | MRC Centre of Immune Regulation has a major focus on rheumatology as a result of the fellowship. Fellowship initiated several subsequent project and programme grants. PI is now head of department. Academic collaborators are now leading scientists in the field. 7 PhDs. | The observation that RGD peptides were triggering apoptosis informed the development of KGD peptides, a class of drugs used to treat deep-vein thrombosis, but not useful for arthritis. Established long-term collaboration with industrial partners. | No evidence of health benefits. | No evidence of broader economic benefits. |
I | 11 papers, receiving a total of 32 citations per year. Although the research hypothesis was not addressed in the timeframe of the fellowship, the research did contribute to a number of methodological developments. | The PI completed an MD and later become a consultant in rheumatology and general medicine. The lessons from the fellowship were applied to a subsequent project grant, assessing the effect of exercise on people with OA subsequently informed British and European guidelines. | A grading system for radiographic assessment of OA was developed and used in graduate clinical education. | It is now common for clinicians to recommend the physical training of muscles for people with OA. | No evidence of broader economic benefits. |
J | 13 papers, receiving a total of 10 citations per year. These publications showed that proprioception plays an important role in ACLD deficiency, and that a rehabilitation programme to enhance proprioception was more effective than traditional muscle strengthening and exercise. | The PI was awarded a PhD on the basis of the work. | Informed management of ACLD. | Small benefit due to improved management of ACLD. | Unquantified return in reduction of days off work. |
K | 7 papers, receiving a total of 29 citations per year. Showed treatment with aspirin and heparin gives higher rate of live births in women suffering recurrent miscarriage associated with antiphospholipid antibodies than aspirin alone. | The trial generated a number of further lines of research, including the effect of antiphospholipid antibodies on in vitro fertilization. The research registrar was awarded a MD. The PI established an international profile. | The British Royal College of Obstetricians and Gynaecologists guideline and the US, Dutch and Australian guidelines all recommend combination therapy of aspirin plus heparin and all cite the trial as the evidence for this recommendation. | The trial led to the conclusion that a combined treatment increased the chances of a live birth for women who had tested positive for antiphospholipid antibodies. A number of these women would not have completed a pregnancy successfully without this treatment. | The treatment is complex and potentially expensive. However, as the number of live births increases, fewer costs are involved because of a decrease in repetition of pregnancies. |
L | 41 papers, receiving a total of 330 citations per year. The paper describing the first clinical trial demonstrated the efficacy and safety of treatment. The results of a 4-centre double blind trial of 73 patients published in 1994 confirmed these findings. | Work promoted research in the area and promoted the use of biologicals in therapy. In wider RA research, work led to a paradigm shift, revealing that, in RA's complicated multistep process, one single molecule can have profound effects on pathogenesis. 3 PhDs and MDs. | The research led to the development of three drugs which have been licensed in the UK. Anti-TNF treatment has been included in a technology appraisal from the NHS National Institute for Clinical Excellence and adopted by international consensus groups in Europe and the USA. | The potentially considerable benefits to RA patients of anti-TNF treatment have been limited by low take-up due to perceived high cost of treatment. | Broader economic benefits have resulted from commercial exploitation of anti-TNF drugs. |
M | 15 papers, receiving a total of 20 citations per year. The research identified a number of specific genetic regions and markers for RA. By the end of 1996, 170 sibling pair families had been identified, confirming linkage to HLA. | No evidence of research targeting or capacity building. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
N | 16 papers, receiving a total of 75 citations per year. In the institute's submission for a new core grant in 1996, one reviewer noted that the work led to ‘few publications in first class journals’. The PI suggested that in terms of papers the impact had been low. | The PI argued that in terms of papers the impact had been low, but the work had had a strategic influence on the field. The PI left the institute in 1994, which heralded the disintegration of the research team. He has subsequently refocused his research interests. 3 PhDs. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
O | 17 papers, receiving a total of 21 citations per year. The core publications describe the identification of dinucleotide repeat polymorphism as the α-1-antitrypsin locus and give a summary of the first 100 multi-case RA families with affected sibling pairs. | The fellowship benefited the PI's career and RA genetics capacity at the institute. 1 PhD. | No evidence of informing policy and product development. | No evidence of health benefits. | No evidence of broader economic benefits. |
P | 27 papers, receiving a total of 40 citations per year. NOAR has provided groundbreaking work in understanding how common RA is and the factors that are associated with it. | The PI, co-applicants and lead rheumatologist have become leaders in their field. Six people have obtained PhD/MDs as a result of their work on NOAR. NOAR has increased research capacity in Norfolk. The Danes have set up a similar registry, based on NOAR. | The incidence and prevalence figures from NOAR are widely cited in clinical guidelines. Recent data from NOAR on the benefit of early referral. Early referral of RA patients is one of the two quality standards set for rheumatology by the Royal College of Physicians. | No evidence of health benefits. | No evidence of broader economic benefits. |
GP, general practitioner; NOAR, Norfolk Arthritis Register; RCT, randomized controlled trial; ACLD, Anterior Cruciate Ligament Deficiency, a disorder of the knee.
Knowledge production
In total, we identified 302 papers attributable to the case study grants, and between the beginning of the case studies and 2002 these received a total of 975 citations per year. The researchers funded by the case study grants had presented their results at numerous national and international conferences; however, as there were no comprehensive records of these presentations, we could not collate usable information on them.
The largest number of papers were produced by case studies E and L, with over 40 papers each. The highest levels of citation were achieved by case studies C, H and L; the collection of papers from these case studies all received over 100 citations per year. In considering measures of knowledge production based on citation, it is important to remember the disadvantages suffered by the allied health professional field of research. Of the papers attributed to these grants, case studies A and I, only 53% were published in journals for which citation information existed in the Science Citation Index (SCI); furthermore, many of the citations of those articles that are included will be missed, as the citations will have been in journals not covered by the SCI.
Research targeting and capacity building
The majority of case studies recorded significant paybacks in terms of research targeting and capacity building. A major element was the career development of either the PIs or postgraduate and postdoctoral research assistants who were employed on the research grants. For example, 24 scientists were awarded either a PhD or an MD as a result of the research undertaken on the 16 grants; for case studies I and J, these were awarded to the PI. A number of the PIs were subsequently appointed to senior academic positions, including research chairs, clinical consultants, heads of Medical Research Council (MRC) units, and the dean of a medical school. It is inappropriate to attribute wholly such outcomes directly to the ARC funding, but many of the PIs suggested it had been a contributory factor.
A further element of research capacity was the transfer of technological know-how—in case study O, this involved genetic mapping from a group acknowledged as leader in the field, which hosted the PI for a 6-month training fellowship before she returned to her host institution. Similarly, in case study P, a number of interviewees commented that the establishment of the Norfolk Arthritis Register increased (arthritis) research capacity locally.
In terms of research targeting, case studies A, B, C, H, K and L seem to have the major impact. As summarized in Table 2, this included: informing a >£2 million MRC-sponsored clinical trial (case study A); pharmaceutical companies' interest in recombinant TIMPs (tissue inhibitors of metalloproteinases) (case study C); a Japanese study on occupational lifting (case study B); subsequent research on the effect of antiphospholipid antibodies on in vitro fertilization (case study K); and a significant, possibly paradigm-changing, shift in the use of biologicals as therapeutic agents (case study L).
Finally, it is worth noting the development of reagents (case study C) and a mouse model (case study D) as further examples of research payback. In the former, although records were not kept, the PI estimated that the reagents produced during the grant were sent to over 100 other research groups.
Informing policy and product development
As with the previous two categories, we identified a range of different types of payback that informed either policy development or product development. These include: being cited on systematic reviews; being cited on clinical guidelines and other forms of policy guidance; and the development of specific clinical tests.
Four papers that were attributed to case studies A and B were cited subsequently in systematic reviews. Moreover, in both of these examples, the quality of the original papers was assessed by the authors of the systematic reviews and the studies scored well against the criteria used. Seven papers from the case studies were cited on clinical guidelines or technology appraisals relating to arthritis. Case study P had the highest level of citation on clinical guidelines, with three papers cited on arthritis-related guidelines and one paper cited by the non-arthritis-related guideline. Case study L, which relates to the use of anti-TNF therapy in RA, had two papers cited in a National Institute for Clinical Excellence technology appraisal. Case studies E and K both had one paper that was cited by arthritis-related guidelines. For case study E, this was a European guideline. For case study K, the findings were an important element in the British Royal College of Obstetricians and Gynaecologists guidelines recommending a combination therapy of aspirin plus heparin to prevent recurrent miscarriages for women who have antiphospholipid syndrome (APS).
An especially interesting example of payback arose from case study B. This grant looked at the role of occupational activity and hip osteoarthritis (hip OA) and concluded that agricultural workers were at an increased risk of hip OA. This work was cited in the IIAC assessment of whether hip OA in farmers should be a prescribed disease (and thus receive Industrial Injuries Benefit); the Council's subsequent advice to the Secretary of State was that this was justified. In addition to the paper, one of the lead PIs gave a presentation to the Council.
In terms of informing product development, three case studies identified potential drug targets: case study C stimulated industrial research in the area of matrix metalloproteinase inhibitors; case study H indirectly informed the development of a class of drugs used outside the arthritis field to treat deep-vein thrombosis; and case study L tested the hypothesis that RA could be treated by using targeted TNF-α inhibitors.
Finally, two case studies, F and G, identified specific diagnostic tests, although these were for very rare conditions. The first was for specific mutations in collagen X and is used to test for chondrodysplasia type Schmid. The second enabled the measurement of antibodies to C1q in patients with systemic lupus erythematosus (SLE).
Health and health sector benefits
Three grants have clearly had significant potential health and health sector benefits. Case study K concluded that a combined treatment of aspirin and heparin for women with APS, which affects about 15 in 10 000 women, reduces the risk of recurrent miscarriages and provides these women with a far higher chance of a successful pregnancy.
Case study L provided some approximate estimates of the numbers of RA patients benefiting, in terms of improved symptom relief and physical functioning. To date, take-up has been limited, due to the high cost of treatment and doubts about the drugs' cost-effectiveness except in restricted groups of patients [21]. The prevalence of RA in Western populations is estimated at 0.8% and there is an uptake of approximately 2% of patients in the UK, 6% in Scandinavia and 15% in the USA, this suggests that approaching 10 000 patients in the UK and 350 000 patients in the USA are receiving anti-TNF treatment. Results of trials suggest that between 29% [22] and around 70% [23, 24] may enjoy a health improvement due to the anti-TNF treatment.
Although case study P concerned the epidemiology and prevalence of RA, it was, nevertheless, seen to be providing health benefits by its identification of other risk factors, such as heart disease, for RA patients; this information was improving patient care.
For the more basic research, it is still to early too judge what the total benefits may be. For example, for case study C, there are still active drug development programmes that were stimulated by the research. In other cases, such as case studies E and F, improved understanding of the disease processes of OA may lead to future improvements in treatment; similar benefits may come out of case study H for RA patients. Finally, there are case studies where the technology and techniques developed have laid the groundwork for current basic research that eventually may lead to patient benefit: case study O has led to work in pharmacogenomics that may benefit RA patients.
Broader economic benefits
For cases A, B, J and K, there are unquantified returns in the reduction of days off work and the value of production gained from having a more healthy workforce. In addition, for case study L, there are economic returns to those companies that have licensed drugs. However, we were unable to identify evidence to quantify the broader economic returns arising from this ARC-sponsored research.
Cross case analysis: the effectiveness of different modes of funding
In order to assess the strengths and weakness of different modes of funding, we undertook comparative analyses of the case studies. The cross-case analyses were based on a qualitative examination of common types of payback (Table 3), supported by the quantitative assessment of scores summarized in Fig. 2.
. | Project . | Programme . | Fellowship . | Institute . |
---|---|---|---|---|
Knowledge production | 56 papers with a total of 187 citations per year | 82 papers with a total of 126 citations per year | 63 papers with a total of 196 citations per year | 101 papers with a total of 466 citations per year |
Research training and capacity building | 4 PhD/MDs awarded from work on grant (case studies A, C, K and J) | 2 PhD/MDs awarded from work on grant (case study F) | 9 PhD/MDs awarded from work on grants (case studies G, H and I) | 13 PhD/MDs awarded from work on grant (case studies L, O and P) |
Provision of reagents (case study C) | Contributed to the establishment of MRC for Centre Immune Regulation (case study H) | Development of technological know-how in genetic mapping (case study O) | ||
Informed Japanese study on occupational lifting (case study B) | ||||
Informed subsequent work looking at the effect of APL antibodies on in vitro fertilisation (case study K) | Informed shift in the use of biologicals as therapeutic targets (case study L) | |||
Informed >£2 million MRC RCT (case study A) | ||||
Development of mouse model (case study D) | ||||
Informing policy and product development | Citation on systematic reviews (case studies A and B) | Cited in European guideline (case study E) | Test for SLE (case study G) | Widely cited in a number of guidelines in setting context (case study P) |
Directly supported RCOG guideline recommending a combination therapy of aspirin and heparin for women with APS (case study K) | Test for chondrodysplasia type Schmid (case study F) | Informed development of a class of drugs to treat deep-vein thrombosis (case study H) | Tested hypothesis that RA could be treated using TNF-α inhibitors (case study L) | |
Cited in Dutch guideline (case study B) | ||||
Cited in IIAC assessment of whether hip OA in farmers should be a prescribed disease (case study B) | ||||
Informed management of ACLD (case study J) | ||||
Health and health sector benefits | Combined treatment of aspirin plus heparin reduced the risk of miscarriage for women with APS (case study K) | Significant benefits to large fraction of RA patients limited by low take-up due to perceived high cost of treatment (case study L). | ||
Study results improved management of back pain patients through study days run by PI (case study A) | Understanding of prevalence of arthritic disease (case study P) | |||
Improved rehabilitation of ACLD patients (case study J) | ||||
Wider economic benefits | Unquantified return in reduction of days of work (case studies A, B, J and K) | Economic return to companies licensed to produce drug (case study L) | ||
Confounding factors | ||||
Number of grants | 6 | 3 | 3 | 4 |
Median size of grant | c. £90 000 | c. £480 000 | c. £250 000 | c. £450 000 |
Median duration of grant | 2.5 yr | 5 yr | 5 yr | 4 yr |
Median research age | 3.5 yr | 17 yr | 7 yr | 16.5 yr |
. | Project . | Programme . | Fellowship . | Institute . |
---|---|---|---|---|
Knowledge production | 56 papers with a total of 187 citations per year | 82 papers with a total of 126 citations per year | 63 papers with a total of 196 citations per year | 101 papers with a total of 466 citations per year |
Research training and capacity building | 4 PhD/MDs awarded from work on grant (case studies A, C, K and J) | 2 PhD/MDs awarded from work on grant (case study F) | 9 PhD/MDs awarded from work on grants (case studies G, H and I) | 13 PhD/MDs awarded from work on grant (case studies L, O and P) |
Provision of reagents (case study C) | Contributed to the establishment of MRC for Centre Immune Regulation (case study H) | Development of technological know-how in genetic mapping (case study O) | ||
Informed Japanese study on occupational lifting (case study B) | ||||
Informed subsequent work looking at the effect of APL antibodies on in vitro fertilisation (case study K) | Informed shift in the use of biologicals as therapeutic targets (case study L) | |||
Informed >£2 million MRC RCT (case study A) | ||||
Development of mouse model (case study D) | ||||
Informing policy and product development | Citation on systematic reviews (case studies A and B) | Cited in European guideline (case study E) | Test for SLE (case study G) | Widely cited in a number of guidelines in setting context (case study P) |
Directly supported RCOG guideline recommending a combination therapy of aspirin and heparin for women with APS (case study K) | Test for chondrodysplasia type Schmid (case study F) | Informed development of a class of drugs to treat deep-vein thrombosis (case study H) | Tested hypothesis that RA could be treated using TNF-α inhibitors (case study L) | |
Cited in Dutch guideline (case study B) | ||||
Cited in IIAC assessment of whether hip OA in farmers should be a prescribed disease (case study B) | ||||
Informed management of ACLD (case study J) | ||||
Health and health sector benefits | Combined treatment of aspirin plus heparin reduced the risk of miscarriage for women with APS (case study K) | Significant benefits to large fraction of RA patients limited by low take-up due to perceived high cost of treatment (case study L). | ||
Study results improved management of back pain patients through study days run by PI (case study A) | Understanding of prevalence of arthritic disease (case study P) | |||
Improved rehabilitation of ACLD patients (case study J) | ||||
Wider economic benefits | Unquantified return in reduction of days of work (case studies A, B, J and K) | Economic return to companies licensed to produce drug (case study L) | ||
Confounding factors | ||||
Number of grants | 6 | 3 | 3 | 4 |
Median size of grant | c. £90 000 | c. £480 000 | c. £250 000 | c. £450 000 |
Median duration of grant | 2.5 yr | 5 yr | 5 yr | 4 yr |
Median research age | 3.5 yr | 17 yr | 7 yr | 16.5 yr |
RCT, randomized controlled trial; RCOG, Royal College of Obstetricians and Gynaecologists; APL, antibodies that react with phospholipids; ACLD, Anterior Cruciate Ligament Deficiency, a disorder of the knee.
. | Project . | Programme . | Fellowship . | Institute . |
---|---|---|---|---|
Knowledge production | 56 papers with a total of 187 citations per year | 82 papers with a total of 126 citations per year | 63 papers with a total of 196 citations per year | 101 papers with a total of 466 citations per year |
Research training and capacity building | 4 PhD/MDs awarded from work on grant (case studies A, C, K and J) | 2 PhD/MDs awarded from work on grant (case study F) | 9 PhD/MDs awarded from work on grants (case studies G, H and I) | 13 PhD/MDs awarded from work on grant (case studies L, O and P) |
Provision of reagents (case study C) | Contributed to the establishment of MRC for Centre Immune Regulation (case study H) | Development of technological know-how in genetic mapping (case study O) | ||
Informed Japanese study on occupational lifting (case study B) | ||||
Informed subsequent work looking at the effect of APL antibodies on in vitro fertilisation (case study K) | Informed shift in the use of biologicals as therapeutic targets (case study L) | |||
Informed >£2 million MRC RCT (case study A) | ||||
Development of mouse model (case study D) | ||||
Informing policy and product development | Citation on systematic reviews (case studies A and B) | Cited in European guideline (case study E) | Test for SLE (case study G) | Widely cited in a number of guidelines in setting context (case study P) |
Directly supported RCOG guideline recommending a combination therapy of aspirin and heparin for women with APS (case study K) | Test for chondrodysplasia type Schmid (case study F) | Informed development of a class of drugs to treat deep-vein thrombosis (case study H) | Tested hypothesis that RA could be treated using TNF-α inhibitors (case study L) | |
Cited in Dutch guideline (case study B) | ||||
Cited in IIAC assessment of whether hip OA in farmers should be a prescribed disease (case study B) | ||||
Informed management of ACLD (case study J) | ||||
Health and health sector benefits | Combined treatment of aspirin plus heparin reduced the risk of miscarriage for women with APS (case study K) | Significant benefits to large fraction of RA patients limited by low take-up due to perceived high cost of treatment (case study L). | ||
Study results improved management of back pain patients through study days run by PI (case study A) | Understanding of prevalence of arthritic disease (case study P) | |||
Improved rehabilitation of ACLD patients (case study J) | ||||
Wider economic benefits | Unquantified return in reduction of days of work (case studies A, B, J and K) | Economic return to companies licensed to produce drug (case study L) | ||
Confounding factors | ||||
Number of grants | 6 | 3 | 3 | 4 |
Median size of grant | c. £90 000 | c. £480 000 | c. £250 000 | c. £450 000 |
Median duration of grant | 2.5 yr | 5 yr | 5 yr | 4 yr |
Median research age | 3.5 yr | 17 yr | 7 yr | 16.5 yr |
. | Project . | Programme . | Fellowship . | Institute . |
---|---|---|---|---|
Knowledge production | 56 papers with a total of 187 citations per year | 82 papers with a total of 126 citations per year | 63 papers with a total of 196 citations per year | 101 papers with a total of 466 citations per year |
Research training and capacity building | 4 PhD/MDs awarded from work on grant (case studies A, C, K and J) | 2 PhD/MDs awarded from work on grant (case study F) | 9 PhD/MDs awarded from work on grants (case studies G, H and I) | 13 PhD/MDs awarded from work on grant (case studies L, O and P) |
Provision of reagents (case study C) | Contributed to the establishment of MRC for Centre Immune Regulation (case study H) | Development of technological know-how in genetic mapping (case study O) | ||
Informed Japanese study on occupational lifting (case study B) | ||||
Informed subsequent work looking at the effect of APL antibodies on in vitro fertilisation (case study K) | Informed shift in the use of biologicals as therapeutic targets (case study L) | |||
Informed >£2 million MRC RCT (case study A) | ||||
Development of mouse model (case study D) | ||||
Informing policy and product development | Citation on systematic reviews (case studies A and B) | Cited in European guideline (case study E) | Test for SLE (case study G) | Widely cited in a number of guidelines in setting context (case study P) |
Directly supported RCOG guideline recommending a combination therapy of aspirin and heparin for women with APS (case study K) | Test for chondrodysplasia type Schmid (case study F) | Informed development of a class of drugs to treat deep-vein thrombosis (case study H) | Tested hypothesis that RA could be treated using TNF-α inhibitors (case study L) | |
Cited in Dutch guideline (case study B) | ||||
Cited in IIAC assessment of whether hip OA in farmers should be a prescribed disease (case study B) | ||||
Informed management of ACLD (case study J) | ||||
Health and health sector benefits | Combined treatment of aspirin plus heparin reduced the risk of miscarriage for women with APS (case study K) | Significant benefits to large fraction of RA patients limited by low take-up due to perceived high cost of treatment (case study L). | ||
Study results improved management of back pain patients through study days run by PI (case study A) | Understanding of prevalence of arthritic disease (case study P) | |||
Improved rehabilitation of ACLD patients (case study J) | ||||
Wider economic benefits | Unquantified return in reduction of days of work (case studies A, B, J and K) | Economic return to companies licensed to produce drug (case study L) | ||
Confounding factors | ||||
Number of grants | 6 | 3 | 3 | 4 |
Median size of grant | c. £90 000 | c. £480 000 | c. £250 000 | c. £450 000 |
Median duration of grant | 2.5 yr | 5 yr | 5 yr | 4 yr |
Median research age | 3.5 yr | 17 yr | 7 yr | 16.5 yr |
RCT, randomized controlled trial; RCOG, Royal College of Obstetricians and Gynaecologists; APL, antibodies that react with phospholipids; ACLD, Anterior Cruciate Ligament Deficiency, a disorder of the knee.
The four modes of funding provided by ARC (in the early 1990s) were considered. Project grants are designed for work on a specific research question over the duration of around 3 yr; programme grants support a portfolio of more speculative long-term research over a period of 5 yr; fellowships aim to develop the career of an individual; and institute grants help to maintain a centre of arthritis research with both clinical and basic strands, for 4–5 yr at a time.
Analysis of Table 3 and Fig. 2 suggests that, despite lower costs and shorter duration, at a median size of grant of £90 000 over 2.5 yr, project grants yield considerable payback over a range of categories, and do not appear to underperform the other three funding modes (programme grants at a median cost of approximately £480 000 over 5 yr, fellowships at approximately £250 000 over 5 yr, institute grants at approximately £450 000 over 4 yr). Funding provided to institutes resulted in the widest range of levels of payback and also encompassed the highest-scoring study. Project grants and institute funding include the case studies with the broadest health benefits (case studies K and L, respectively). By comparison, programme grants (roughly comparable to institute grants in median funding amounts and periods) perform poorly, with the payback recorded as limited to two PhDs, a guideline citation and a clinical test.
Conclusions and policy implications
From our analysis of 16 case studies, we drew six observations and, based on these, made six policy observations and recommendations for the ARC. The six conclusions are summarized in the key messages box. Given space limitations, in this paper we have reported on the results that underpin two of these observations: there is a diversity of research payback; and short focused project grants seem to provide value for money. However, before commenting on these findings, we discuss a number of limitations of our approach, to illustrate some of the remaining challenges to be faced in evaluating research.
The first caveat is the use, selection of and generalization from case studies. Given the approach that we adopted and the budget, we had to restrict the number of case studies to 16. This raises the question of whether the results of this study can be generalized to all ARC research or indeed beyond that. This, in itself, then becomes a debate on the legitimacy of qualitative research and the use of case studies in general. Yin, who has written a seminal text on case study research, argues that ‘the case study is but one of several ways of doing social science research … [each with] peculiar advantages and disadvantages’ [19, p.1]. In terms of scientific generalization, Yin makes the point that ‘scientific facts are rarely based on single experiments’ and that the ‘investigator's goal [with case study research] is to expand and generalise theories (analytic generalisation) and not to enumerate frequencies (statistical generalisation)’ [19, p.10].
A second issue is the attribution of payback to ARC-funded work. It is undoubtedly true that there is a range of unquantified and (largely) unspecified non-ARC inputs into the research evaluated in the 16 case studies. This issue was explored in some depth at a 1999 international workshop on research evaluation [25], and it was concluded that there are many complications in identifying the impact of specific funding. How far it is desirable to attempt this in detail depends partly on the purposes for which the evaluation is being conducted.
A third issues relates to the robustness and consistency of the scoring system that we developed for the payback profiles (Fig. 2). Although we believe that this system is a major step towards being able to operationalize and embed a multidimensional research monitoring and evaluation system, we would stress that it was developmental. We need to have a better understanding of the reliability of the scoring system, a tighter definition of the scoring scale and an understanding of the quantity and type of information necessary to produce informed and consistent scores. All of these issues are future research topics.
Finally, there is the issue of elapsed time. In deciding on a time window for research evaluation, a compromise needed to be found between the quality of records, the ability of researchers to recall their activities and allowing enough time for the outcomes of research to develop [26]. In this study we used a 10-year lag, which seemed to be an optimal trade-off between these two requirements. However, it was not long enough to allow a fair comparison between basic and clinical research. Previous work examining how long it takes for basic research to inform clinical guidelines suggests that even after four generations of citation, taking around 17 yr, less than 20% of the cited research is basic [27]. And it is worth noting that the anti-TNF work, case study L, would have been basic research in the mid-1980s whilst case study O has resulted in further pharmacogenetics work that may benefit patients with RA in the future.
Diversity of payback
There is strong evidence from our analysis that there is a wide range of research paybacks and that many of these would not have been readily identified without the structured in-depth case study approach that was employed in this evaluation. In Table 2 we summarize the various types of payback identified from the 16 case studies. While this is an impressive list, it should be remembered that our sample of 16 case studies was biased towards more successful investigators, because we selected high- and mid-impact investigators rather than a full spread. Nevertheless, these returns come from only 16 (or 3%) of 556 possible grants, and, even if cautiously extrapolated, illustrate a diverse and significant return to ARC's investment in research.
Given this diversity of outcome, it is important that the ARC and other research funders use an assessment method that captures a broad range of payback. The payback categories developed in this study provide a structured basis for any further monitoring or evaluation of the outputs and outcomes of research funded by the ARC. This wide-ranging assessment is important, as there is a tendency in research evaluation to rely heavily on bibliometric analysis and, as shown in this study, this can seriously underestimate payback.
The importance of a multifaceted approach to research evaluation was confirmed by Cronin and Normand [28] in their review of the literature. First, they concluded that there was consensus that all criteria needed a qualitative and quantitative information base; and secondly that a number of different criteria should be used. This was best articulated by Martin and Irvine [29], who proposed the concept of ‘converging partial indicators’; that is, a series of incomplete indicators that indicate the reliability of the assessment when they are all pointing in the same direction.
The relative value of project, programme and institute grants
The cross-case analysis presented in Fig. 2 and Table 3 shows a significant and diverse range of payback arising from the six project grants that we examined. This included, for example, 62 papers, four PhD/MDs, informing four clinical guidelines or policy documents, reduced risks in miscarriages for women with APS, and unquantified reductions in days of work. By comparison, the three programme grants resulted in 82 papers, two PhDs, one citation in a clinical guideline and the development of a clinical test for chondrodysplasia type Schmid. This does not appear to show the much higher returns that might have been expected from programme grants. However, it should be borne in mind that: (i) programme grants, by their very nature, might be expected to be more speculative and long term; and (ii) we only examined three such grants. [We are also subsequently told that the award system for such grants has become far more rigorous and competitive since the early 1990s, when these grants were first awarded.]
Of all the observations we have made from our analysis, this was the most unexpected and surprising. There is a widespread view in science policy that long-term stable funding is preferential, although all types of support mechanisms have been shown to contribute significantly to 13 clinical advances in cancer research [30]. However, there are a number of potential confounding factors that mean the conclusion regarding project grants needs to be treated with caution. For example, four of the six project grants were patient-focused, and three of the project grants involved a randomized controlled trial. This suggests perhaps that the projects were more mission-oriented and therefore likely from the outset to lead to faster payback. Conversely, the fact that patient-focused, randomized controlled trial-type work is being funded through projects illustrates the importance of maintaining a mechanism for funding short-term focused research of this nature. This unexpected finding needs to be tested elsewhere. It was also notable that the institute case studies showed the widest range in outcomes, including the most successful case study. This could imply that institute funding allows the pursuit of speculative and innovative research that, when successful, has large payback.
For a number of decades, research funding organizations have been interested in assessing the payback from their research. In response to this demand, analysts have developed a number of methods to evaluate research. In this study we have combined a number of different approaches to assess the long-term payback from ARC-funded research. We would like to conclude by noting that the rationale for supporting this study was well founded. Research funding agencies such as the ARC need a firm evidence base to support policy and decision-making. Indeed, other research funding agencies should follow ARC's lead and accept the need for a firm evidence basis for their policies through supporting studies such as these [5].
As is normal with a project of this scale and complexity, the project team would like to acknowledge the invaluable support provided by a number of groups and individuals. First, ARC's Development Committee—that is, Patrick Sissons, Ann Raven, Fergus Logan, Madeleine Devey, Michael Patnick, Mary Collins, Matthew Brown, Mike Hurley and Tony Woolf—provided excellent guidance during all stages of the study. We would also like to acknowledge the role of ARC staff, most notably Fergus Logan, Madeleine Devey and Michael Patnick, for initiating the study and for answering our numerous queries. The ongoing development of the Health Economics Research Group's payback framework has been supported by its Policy Research Programme grant from the Department of Health. We would like to reserve our final acknowledgements to all those scientists—46 in total—who were willing and able to act as our experimental subjects for this study. This involved either being the subject of a case study or being interviewed as a stakeholder of a case study. As is the nature of this type of evaluation, it is inappropriate to name these individuals, but without their support and commitment this study would not have been possible.
This work was carried out for and funded by the Arthritis Research Campaign. As such we acted as consultants for the Arthritis Research Campaign. All the authors continue to seek financial support to undertake studies researching and evaluating the effectiveness of research funding organizations.
References
Committee on Science, Engineering and Public Policy (COSEPUP).
Lewison G, Devy ME. Bibliometric methods for the evaluation of arthritis research.
National Institutes of Health.
Grant J, Hanney S, Buxton M. Academic medicine: time for reinvention. Research needs researching.
Hanney S, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the long-term outcomes of research: the impact of funding by the UK's Arthritis Research Campaign.
Wooding S, Hanney S, Buxton M, Grant J.
Wooding S, Anton S, Grant J, Hanney S, Hoorens S, Lierens A, Shergold M.
Arthritis Research Campaign. Research. http://www.arc.org.uk/research/default.htm. Accessed 7 April
Arthritis Research Campaign. Research into practice. A five-year strategic plan. http://www.arc.org.uk/about_us/resprac.htm. Accessed 7 April
Buxton M, Hanney S. How can payback from health services research be assessed?
Buxton M, Hanney S, Packwood T, Roberts S, Youll P. Assessing benefits from Department of Health and National Health Service research and development.
Hanney S, Soper B, Buxton M.
Jacob R, McGregor M. Assessing the impact of health technology assessment.
Yin R, Moore G. Lessons on the utilization of research from nine case experiences in the natural hazards field.
Hanney S, Gonzalez-Block M, Buxton M, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment.
Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research.
Williams D, Rank AD. Measuring the economic benefits of research and development: the current state of the art.
Fitch F, Berstein SJ, Dolores Aguilar M et al. The RAND/UCLA appropriateness method user's manual, MR-1269-DG-XII/RE. Santa Monica, CA: RAND Corporation,
National Institute for Clinical Excellence.
Maini R, St Clair EW, Breedveld F et al. Infliximab (chimeric anti-tumor necrosis factor alpha monoclonal antibody) versus placebo in rheumatoid arthritis patients receiving concomitant methotrexate: a randomised phase III trial.
Arthritis Research Campaign. Life after anti-TNF. http://www.arc.org.uk/newsviews/arctdy/118/afterTNF.htm. October
Robin, F. Randomised double-blind comparison of chimeric monoclonal antibody to tumour necrosis factor (ca2) versus placebo in rheumatoid arthritis.
Croxson B, Hanney S, Buxton M. Routine monitoring of performance: what makes health research and development different?
Bozeman B, Kingsley G. R&D value-mapping: a new approach to case study-based evaluation.
Grant J, Cottrell R, Cluzeau F, Fawcett G. Evaluating ‘payback’ on biomedical research from papers cited in clinical guidelines: applied bibliometric study.
Cronin E, Normand C.
Martin B, Irvine J. Assessing basic research: some partial indicators of scientific progress in radio astronomy.
Author notes
RAND Europe, Cambridge and 1Health Economics Research Group, Brunel University, London, UK.
Comments