Background
Estimating the returns from biomedical and health research
Study/features | Mushkin (1979) [[4]] | Funding first (2000) [[6]] | Access economics (2003) [[8]] | Access economics (2008) [[9]] | HERG et al.(2008) [[3]] | Access economics (2011) [[10]] |
---|---|---|---|---|---|---|
How health gains were assessed | Top-down by disease category: overall gain in each category not linked to specific intervention. Attributed 20 to 30% of total gain to R&D. Reduced morbidity difficult to assess because little reduction in days off work because of sickness. Adjusted the raw data, for example, by applying historical Army and Navy data as an index to record the decline in sickness. | Top-down: overall gain in mortality not linked to specific interventions. Attributed roughly one-third of the total gain to R&D, plus ‘some fraction of the credit for the other two-thirds.’ | Top-down: overall gain in mortality and morbidity not linked to specific interventions. Attributed 50% of the total gain to R&D. | Top-down: as in the 2003 study, overall gain in mortality and morbidity not linked to specific interventions. Attributed 50% of the total gain to R&D. | Bottom-up: identified research-based interventions, then quantified health impact. | Top-down: overall gain in mortality and morbidity for five disease areas not linked to specific interventions. Attributed 50% of the total gain to R&D. |
How health gains were valued | Human capital approach, that is, values attached to lives saved between one period and the next, based on potential future earnings, plus calculation of value of potential working time no longer lost due to sickness. | Used a comparatively high ‘willingness-to-pay’ value derived from labour economics. | Used the same comparatively high ‘willingness-to-pay’ value as Funding First. | Used a higher ‘willingness-to-pay’ estimate than the 2003 study, this time derived from a meta-analysis of international studies. | Used a comparatively low, but arguably realistic, value of health gain by adopting the figure implied by the current level of NHS spending, that is, the opportunity cost of a QALY within the current NHS budget. | Used a lower ‘willingness-to-pay’ estimate than that used in the 2008 study, in line with Department of Finance and Deregulation guidance. |
Proportion of national health gain allocated to national research | Not discussed as a major issue; we assumed it to be 100%. | Not discussed as a major issue in Funding First; we assumed it to be 100%. | Used proportion of global research conducted in Australia (2.5%) to determine the proportion of the total research-based health gain to attribute to Australian research. | Uses bibliometric analysis-based estimate of Australian share of global research output in clinical medicine (3.04%). | An analysis of citations of UK research on UK clinical guidelines suggests average best estimate of 17% linked to UK research. | Uses an updated bibliometric analysis-based estimate of Australian share of global research output in clinical medicine (3.14%). |
Costs of health care considered? | No, at least not as a separate item to net-off against the value of the health gains. | No in initial headline figures, but Yes in later analysis: ‘the gain in the value of life, net what was spent to attain the longer life, is just 15 percent smaller.’ | No, did not net-off the healthcare costs required to achieve the health gains. | No, did not net-off the healthcare costs required to achieve the health gains. | Yes, did net-off the health care costs required to achieve the health gains. | Did not net-off health care delivery costs, but did consider avoided health system expenditure due to gains in wellbeing. |
Considered elapsed time between research and health gains? | Yes: 10 years. | Acknowledged time lags between research and benefits but this was apparently not brought into calculations. | No, compared research expenditure and health benefits in the same year. This implies the health gains from research are instant. | Yes: 40 years, with range of 20 to 60 years used for sensitivity analyses. | Yes: an analysis of citations of UK research on UK clinical guidelines suggested average best estimate of 17 years lag. | Yes: same assumption of 40 years as was used in 2008 study. No sensitivity analysis around elapsed time. |
How the overall rate of return calculated | IRR of 47%. | Not brought together to provide an overall IRR. | An overall benefit/costs ratio for health research of 2.40. | An overall benefit/costs ratio for health research of 2.17. | IRR of 9% for CVD research combined with 30% for GDP benefits. | Benefit-cost ratios for five disease areas: 6.1 (CVD); 2.7 (cancer); 1.1 (SIDS); 1.2 (asthma); and 0.7 (muscular dystrophy). |
Methods
Overall conceptual approach
-
a time series of the public and charitable funding of cancer-related research;
-
a time series of the NMB of cancer health gains, derived from the monetised health benefits and the healthcare costs for selected interventionsa;
-
an estimate of the elapsed time between the investment (research funding) and return (health gain) associated with those interventions; and
-
an estimate of the amount of health gain that should be attributed to public and charitable research investment in cancer-related research in the UK.
Estimating public and charitable funding of cancer-related research
-
Medical Research Council (MRC) spending on cancer research averaged 9.8% of their total investment (range: 4.6% to 16.7%) between 1970/1 and 2009/10.
-
Wellcome Trust cancer funding was more erratic, ranging between 1%c and 38%, with an average of 14.5% of expenditure being on cancer research.
-
The proportion of peer-reviewed research papers in oncology as a percentage of all UK biomedical outputs averaged 9.2% (range: 8.5% to 9.5%) between 1988 and 1995 [25].
-
The proportion of peer-reviewed research papers in oncology research (as a percentage of all NHS research outputs) was 12% between 1990 and 1997 [26].
-
The proportion of mainstream quality-related (QR) funding allocations by the Higher Education Funding Council for England for ‘Cancer studies’ (that is, Unit of Assessment 02) between 2009 and 2012 was around 6% of the total biomedical allocation (that is, Unit of Assessments 01 to 15 and 44).d
Estimating the NMB from cancer-related research
Identifying the key cancer interventions
Identifying estimates of per-patient NMB for the set of cancer interventions
-
Smoking prevention/cessation
-
Screening programmes: cervical, breast, and bowel cancer.
-
Treatment of: breast, colorectal and prostate cancer.
Smoking prevention/cessation
Screening programmes
Treatment programmes
Constructing a time series (1991 to 2010) of usage of cancer interventions
Analysis of UK clinical guidelines to estimate elapsed time and rate of attribution
Estimation of the rate of return
Results
Public and charitable funding of UK cancer-related research, 1970 to 2011
Net monetary benefit
QALYs (thousands) | ||||||||
---|---|---|---|---|---|---|---|---|
Year | Treatment | Screening | Smoking reduction | Total | ||||
Prostate cancer | Breast cancer | Colorectal cancer | Cervical cancer | Bowel cancer | Breast cancer | |||
1991 | 8 | 46 | 6 | 71 | – | 2 | 144 | 277 |
1992 | 9 | 48 | 6 | 70 | – | 2 | 144 | 279 |
1993 | 10 | 46 | 6 | 68 | – | 2 | 145 | 276 |
1994 | 11 | 48 | 6 | 67 | – | 2 | 145 | 279 |
1995 | 10 | 45 | 6 | 65 | – | 2 | 145 | 273 |
1996 | 11 | 46 | 6 | 66 | – | 2 | 146 | 276 |
1997 | 11 | 46 | 6 | 63 | – | 3 | 146 | 274 |
1998 | 11 | 50 | 6 | 59 | – | 2 | 147 | 276 |
1999 | 13 | 53 | 7 | 56 | – | 2 | 147 | 279 |
2000 | 15 | 53 | 7 | 55 | – | 2 | 148 | 281 |
2001 | 17 | 55 | 7 | 54 | – | 2 | 149 | 285 |
2002 | 20 | 56 | 8 | 53 | – | 2 | 150 | 290 |
2003 | 22 | 59 | 8 | 53 | – | 2 | 151 | 295 |
2004 | 24 | 61 | 9 | 56 | – | 2 | 152 | 305 |
2005 | 21 | 62 | 10 | 60 | – | 2 | 154 | 309 |
2006 | 22 | 62 | 12 | 61 | 2 | 2 | 155 | 316 |
2007 | 23 | 65 | 13 | 60 | 5 | 2 | 157 | 324 |
2008 | 27 | 68 | 14 | 60 | 7 | 2 | 158 | 337 |
2009 | 26 | 71 | 15 | 63 | 9 | 2 | 159 | 345 |
2010 | 25 | 74 | 15 | 65 | 12 | 3 | 161 | 354 |
Total | 339 | 1112 | 173 | 1225 | 35 | 43 | 3003 | 5930 |
Costs (GB£ million) | ||||||||
---|---|---|---|---|---|---|---|---|
Year | Treatment | Screening | Smoking reduction | Total | ||||
Prostate cancer | Breast cancer | Colorectal cancer | Cervical cancer | Bowel cancer | Breast cancer | |||
1991 | 199 | 665 | 181 | 41 | – | 34 | -277 | 844 |
1992 | 220 | 687 | 190 | 40 | – | 37 | -277 | 897 |
1993 | 241 | 658 | 185 | 39 | – | 40 | -278 | 887 |
1994 | 272 | 684 | 185 | 39 | – | 42 | -278 | 944 |
1995 | 252 | 646 | 175 | 37 | – | 42 | -279 | 874 |
1996 | 278 | 660 | 182 | 38 | – | 43 | -280 | 921 |
1997 | 269 | 684 | 181 | 36 | – | 53 | -281 | 943 |
1998 | 283 | 720 | 187 | 34 | – | 50 | -282 | 993 |
1999 | 336 | 753 | 214 | 32 | – | 47 | -283 | 1098 |
2000 | 391 | 746 | 211 | 32 | – | 45 | -285 | 1140 |
2001 | 456 | 755 | 206 | 31 | – | 43 | -287 | 1204 |
2002 | 519 | 764 | 202 | 30 | – | 43 | -282 | 1276 |
2003 | 571 | 794 | 178 | 30 | – | 44 | -255 | 1361 |
2004 | 614 | 817 | 179 | 32 | – | 44 | -254 | 1432 |
2005 | 545 | 850 | 182 | 34 | – | 44 | -252 | 1403 |
2006 | 572 | 851 | 188 | 35 | -5 | 45 | -258 | 1428 |
2007 | 596 | 881 | 177 | 35 | -10 | 46 | -252 | 1473 |
2008 | 613 | 919 | 181 | 35 | -15 | 48 | -242 | 1538 |
2009 | 606 | 950 | 185 | 36 | -20 | 48 | -237 | 1569 |
2010 | 569 | 986 | 186 | 38 | -25 | 56 | -240 | 1569 |
Total | 8403 | 15469 | 3755 | 704 | -75 | 894 | -5358 | 23793 |
Net monetary benefit (£ million) | ||||||||
---|---|---|---|---|---|---|---|---|
Year | Treatment | Screening | Smoking reduction | Total | ||||
Prostate cancer | Breast cancer | Colorectal cancer | Cervical cancer | Bowel cancer | Breast cancer | |||
1991 | 8 | 490 | -33 | 1729 | – | 7 | 3885 | 6085 |
1992 | 8 | 506 | -35 | 1699 | – | 7 | 3889 | 6075 |
1993 | 9 | 485 | -33 | 1659 | – | 8 | 3890 | 6018 |
1994 | 10 | 504 | -32 | 1646 | – | 8 | 3894 | 6030 |
1995 | 9 | 475 | -28 | 1592 | – | 8 | 3906 | 5963 |
1996 | -3 | 483 | -28 | 1609 | – | 9 | 3920 | 5989 |
1997 | -3 | 458 | -28 | 1546 | – | 11 | 3933 | 5916 |
1998 | -3 | 531 | -29 | 1447 | – | 10 | 3947 | 5902 |
1999 | -10 | 566 | -40 | 1374 | – | 9 | 3966 | 5866 |
2000 | -15 | 579 | -36 | 1350 | – | 9 | 3989 | 5876 |
2001 | -25 | 630 | -30 | 1310 | – | 9 | 4018 | 5910 |
2002 | -8 | 647 | -5 | 1283 | – | 9 | 4037 | 5963 |
2003 | -14 | 677 | 24 | 1294 | – | 9 | 4035 | 6025 |
2004 | -7 | 705 | 49 | 1376 | – | 9 | 4064 | 6195 |
2005 | -15 | 694 | 67 | 1460 | – | 9 | 4099 | 6314 |
2006 | -16 | 697 | 106 | 1484 | 64 | 9 | 4138 | 6481 |
2007 | -23 | 736 | 149 | 1469 | 128 | 9 | 4167 | 6635 |
2008 | 51 | 786 | 171 | 1474 | 192 | 10 | 4192 | 6876 |
2009 | 46 | 816 | 178 | 1528 | 256 | 10 | 4216 | 7050 |
2010 | 66 | 854 | 183 | 1596 | 320 | 11 | 4253 | 7282 |
Total | 65 | 12318 | 566 | 29927 | 960 | 179 | 80437 | 124452 |
Estimating the elapsed time
Estimating the amount of health gains that can be attributed to UK research
Estimating the IRR from cancer-related research
Analysis | IRR, % |
---|---|
Base casea
| 10.1 |
Research funding estimate | |
Low | 10.8 |
High | 8.7 |
‘Value’ of a QALY, GB£ | |
20,000 | 8.0 |
30,000 | 11.7 |
50,000 | 16.1 |
70,000 | 18.9 |
Elapsed time, years | |
10 | 14.6 |
20 | 7.4 |
Attribution to UK research, % | |
10 | 6.1 |
25 | 13.0 |
Effect of smoking cessation | |
Decrease NMB by 25% | 8.7 |
Increase NMB by 25% | 11.2 |
Omitting benefit of smoking reduction | 2.4 |
Discussion
What this paper contributes
Key assumptions and caveats
-
Our base-case value of a QALY is £25,000. Obviously, and as demonstrated by our sensitivity analysis, the IRR is sensitive to the assumed value of the health gain measured as QALYs. Our base-case assumption is consistent with our analysis of the returns to CVD research, and reflects the mid-point in the range of values (of £20,000 to £30,000) cited as the normal criteria for acceptance of interventions by NICE [51]. More recently, NICE has increased this threshold, up to around £50,000, for certain treatments that provide end-of-life benefits, particularly late-stage cancer treatments [52]. At the same time it has seemed to re-emphasise that the £20,000 threshold should apply unless there are special circumstances. Although this leaves uncertainty about the most appropriate value here (as reflected in our sensitivity analysis), conceptually the argument remains that this ‘opportunity cost’ value of a QALY should apply to an assessment of research in that investing in health-related research can be seen as an alternative to spending the money directly on current health care. We note, however, that other studies in the US and Australia have used much higher values, reflecting individual willingness to pay for health gains, and we have illustrated in a sensitivity analysis the effect of using a value of the order of three times GDP per capita [53].
-
The total NMB for interventions not covered is assumed to be zero. Our IRR calculation assumes that all other cancer treatment developments/interventions that we have not specifically included have, in aggregate, no effect on the NMB, because for these, the monetised value of the health benefit is equal to the cost of delivering the benefit. In reality, there may be some areas that we have not covered for which the NMB is negative because of the high cost of treatment and low incremental health gain. Conversely, there are may be other areas that generate a significant number of QALYs at a relatively low cost. We are not in a position to know whether the net effect of the interventions we did not examine is positive, negative, or zero.
-
The total net flow of knowledge between disciplines is zero. We have assumed that the flow of knowledge is the same into and out of different research fields, and from each research field into the cognate treatment areas. However, we know that research is unpredictable and diffuse, and there may be research disciplines that contribute more than they gain from other areas. One could argue that some of the reduction in mortality from diseases other than cancer that arises as a result of the reduction in smoking (e.g. CVD) which we have excluded, should in fact be included as having been achieved as an additional advantage arising from the evidence of the effect of smoking on lung cancer.
-
All health gain from treatments is captured in the estimates of the health gain from specific interventions. We have assumed that in principle the health gain from improved service configuration and all other supportive service changes (including diagnostics and imaging) should be captured in the estimates of the gains from specific interventions. In practice, our estimates of QALY gains are mainly derived from UK-relevant health technology assessments that are extrapolated from trial data, which may provide an imperfect estimate of the gain when the interventions are used in routine NHS practice.
-
The definitions of the cancer-related research used by the research funders captures basic research that may have contributed to developments in this area. This is clearly the case for the cancer-specific funders such as CRUK, as we included all the research they funded. For MRC funding, we relied on the funder classification which, as discussed in Additional file 1, was broad and thus should include basic research. For the Wellcome Trust, which accounts for around 10% of total cancer funding, we had to rely on search terms. We scanned the list of grant titles selected through this search strategy, and this list suggests that fundamental research is being included, although we cannot guarantee that it all is in fact included. For the remaining two funders – the Funding Councils and the DH/NHS – this would not be an issue, as their time series were derived through an estimate of cancer research activity.
-
The knowledge cycle time and attribution rate were largely determined through bibliometric analysis of clinical guidelines. As part of this study, and reported separately, we undertook a series of case studies that qualitatively explored how research translates into health benefit [24]. This work demonstrates the complexity of biomedical and health innovation, especially when trying to measure the time it takes for research to develop into health benefits. Although the bibliometric approach provides us with an empirical estimate of both the elapsed time and the rate by which we can attribute UK research to UK health gain, it inevitably is a gross simplification of a complex process.
-
We have made various assumptions about the baseline treatment against which we were looking in research-based developments. For example, in estimating the net health gain from breast cancer treatments, we did not include benefits from standard mastectomy but just estimated the benefits from subsequent developments.
-
There is a risk that we may have double-counted the NMB for individuals who are treated as a result of screening. Conceptually, the benefits of screening include the downstream NMB of treatments that result from the screening. However, a number of issues minimise the likelihood of our double counting. First, we did not include (in the treatment calculation) all the benefits of treating an individual disease (for example, breast cancer) but only the additional benefits of improved (research-based) treatments, so any additional people who get ‘basic treatment’ as a result of screening were counted only as an advantage to screening. Second, the benefits and the future treatment costs of a woman entering a screening programme (which is when we estimated the future QALYs and present value of associated net costs) occur in a future year, often many years ahead, so in taking a 20 year period, there is limited scope for counting both. If we had perfect data and were looking at all treatment benefits over a much longer period, we could in principle look only at the benefits of treatments that would encapsulate all the QALY benefits of screening.
-
We have evidence of linkage between research and health gains but no formal evidence of causality. Our analysis relied on the reasonable assumption that these health benefits would not have occurred without the evidence from medical research, and we have illustrated the often complex nature of those linkages in case studies [24]. At one level we have addressed this issue of causality by our bottom-up approach, adding together the benefits demonstrated through clinical trials of new interventions. For these, causality from worldwide medical research is all but a truism. However, even for these, we had to assume that a proportion of the benefit (based on the UK contribution to publications cited in guidelines) arose from UK research. It is possible that some or even all of these interventions might have come into use in the UK even if there had been no UK cancer research, but it is improbable that the same level and timing of benefits would have arisen. Causality could be argued to be less direct for the benefits of the reduction in smoking, which made the largest contribution to the total NMB. It is possible, but implausible, that changes in smoking behaviour might have arisen in the absence of any evidence of the health effects. Certainly, our case studies [24] show that there was an extended lag between the initial evidence of harms to smokers and changes in behaviour, and the UK government probably needed the cumulative evidence that has emerged over several decades, and in particular the evidence of the harms of environmental tobacco smoke, to make the legislative changes in the face of very considerable resistance. There are also additional uncertainties around the magnitudes of NMB from smoking. Of the total £124 billion total NMB, £80 billion (or 65%) arose from reductions in smoking, and the numbers for the increased proportion of the population who were non-smokers or ex-smokers is based on self-reported survey data. In the sensitivity analysis (Table 5), if the NMB from smoking reduction was decreased or increased by (an arbitrary) 25%, the IRR would reduce to 8.7% or increase to 11.2% respectively. Omitting the benefits from smoking reduction entirely reduces the IRR to 2.4%. However, it should be stressed that we estimated only the mortality effects on lung cancer and excluded effects on other cancers (and other disease areas) from smoking, all of which would mean we probably underestimated the impact of smoking reduction. However, taking a perspective of NHS costs only, we have not included costs to other parts of the economy from the various measures to reduce smoking [56].
-
Variable quality of data on the effectiveness of screening. The three national screening programmes are important elements in our estimates. The clinical and cost-effectiveness evidence for bowel cancer screening is high-quality and trial-based, but the evidence for cervical screening, and even more so for breast cancer screening, is less robust. The recent review [33] of the clinical evidence has provided some clarity to the contentious issue of the net benefits of breast screening, and underpins the relatively simple economic model that we used as the basis of our estimate of NMB, but there is considerable uncertainty around these estimates.
-
There is a lack of robust clinical effectiveness and cost-effectiveness data for some interventions, especially for longstanding treatments. This was a general problem with well-established surgical techniques (for example, total mesorectal excision, for which no cost-effectiveness evidence could be found) and similarly for some of the hormonal therapies (for example, tamoxifen and goserelin).
-
There are a large number of areas of cancer that we did not consider in our analysis. Our analysis was based on a prioritised list of cancer types generated from both expert opinion and epidemiological data. By necessity, this meant we did not look at a number of areas (and as noted above, assumed the NMB arising from these areas to be zero).
-
Elapsed time was an important variable in determining the IRR, but one that is conceptually difficult to measure [24]. We wanted to measure the time between research investment and health gain, but neither of these events occurs at one defined point. Research investment may occur over a period, although in many cases, given a typical pattern of investment starting with pilot trials, and building to larger-scale studies and finally randomised controlled trials, the bulk of the research investment may come late in the overall investment period. The point at which the bulk of the health gain occurs is even more difficult to define, and will depend on a range of factors, such as the type of intervention and the way in which it is implemented. The issue of time lags was identified in the original 2008 report, which suggested that further research is needed.
-
Past performance is not an indicator of future performance. The IRR is based on past performance, and cannot be a guarantee of future returns, particularly for increased levels of research spending. This means that research advocates need to use the estimates provided in this paper very cautiously if wishing to extrapolate them as indicators of likely future returns from research expenditure. Given the near doubling in cancer-related research funding since the turn of the century (Figure 3), there will need to be a similar increase in NMB in the coming decade to maintain the current returns. It is worth noting that the NMB of bowel screening is not fully reflected in the IRR because this screening is of recent introduction, so there is additional benefit that will be realised in the future. Likewise, pharmaceutical interventions are typically priced to maximise the value of the benefit at time of introduction, so the NMB is close to zero. During the coming decade, some of the expensive drugs will come off patent and may be available more cheaply, thus contributing to an increase in the NMB; however, other new and expensive ‘on patent’ drugs may well be used in preference.
-
We estimated average returns from cancer research, not the marginal returns. From this analysis, we are not able to say whether the rate of return would have been different if research spending had been higher or lower, and whether at the margin the returns to research investment are increasing or diminishing.
-
The analysis should not be used to make comparative assessments about the value of research into particular interventions/cancers. Our approach examined a portfolio of interventions/cancer types and we would caution that the detailed data may not be sufficiently robust to make comparisons between interventions within specific cancers.
Future research requirements
-
A deeper understanding of the international flows of knowledge. In our model, we estimated the extent to which UK research influences UK practice, using citations on clinical guidelines, and this figure was used in estimating the IRR. However, there is a need for a more nuanced understanding of these knowledge flows and their impact on international health gains; for example, UK research is contributing to health gains beyond the UK. As a result, our current figure underestimates the global value of UK R&D. A study that aimed to measure the health gains, net of healthcare costs, in the rest of the world as a result of UK medical research would address this. At a European level, it would also be interesting to explore how the investments of different European countries in biomedical and health research leads to health gains in other European countries, thereby reinforcing the notion of European solidarity.
-
An improved estimate of spillover effects for UK biomedical and health research. Public and charitable biomedical and health research expenditure not only leads to health gains, but also makes an important contribution to the national economy. Much of the evidence base for estimating a spillover effect of 30% comes from studies undertaken in the 1960s and 1970s, and/or relates specifically to agriculture research. More recent analyses for medical research are largely based on US data. Furthermore, in this study, we also assumed that the spillovers are independent of disease area but we have no empirical evidence to support whether that assumption is justified or not. Future research should aim to provide empirical estimates of the effects of biomedical and health research for the UK economy, ideally at a disease-specific level.
-
Examine another disease area or time period in which smoking reduction is likely to have a minimal impact. As illustrated in Table 5, the IRR for cancer research is very dependent on the effect of smoking reduction. It would be valuable to undertake an investigation in another clinical area in which smoking is not important to see whether similar rates of return are found.