Health economic evaluations (HEEs) on vaccines and vaccination programmes should always be considered by decision-making bodies when considering inclusion of a new vaccine into the national programme to avoid suboptimal allocation of resources. |
Proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs in HEEs. However, vaccines should only be treated differently where they really are different (e.g. indirect effects). |
Funders and decision-makers should recognize that proper and valid HEEs (of vaccines) demand time and resources. |
1 Introduction
2 Methods
2.1 Systematic Literature Review and Preparation of a Workshop
Topics | Vaccines and infectious diseases | Economic evaluation | Guidelines | Methods | Decision making |
---|---|---|---|---|---|
Search terms (in Title OR Abstract) | Vaccine OR Vaccination OR Vaccinate OR Vaccinating OR Vaccinated OR Immunization OR Immunisation OR Immunize OR Immunise OR Infectious disease OR Communicable disease OR Preventable disease | Cost OR Cost effectiveness OR Cost utility OR Cost benefit OR Benefit cost OR Cost saving OR Economic OR Pharmacoeconomic OR Pharmacoeconomics OR Budget impact OR Efficiency OR Efficient OR Monetary OR Financial OR ICER OR QALY | Guideline OR Guide OR Good-practice OR Good OR practice OR Good research practice OR Standards OR Standard OR Recommendation OR Recommendations OR Framework OR Frameworks OR Primer OR Consensus | Methoda OR Methodsa OR Methodological OR Decision analytic OR Decision analysis OR Decision analyses OR Modela OR Modelsa OR Modellinga OR Modelinga OR Model based OR Simulation OR Simulation OR Mathematical OR Transmission OR Dynamic OR Discounting OR Interaction OR Herd immunity OR Herd protection OR Herd effects OR Indirect effects OR Population-wide benefits OR Waning | Decision making OR Reimbursement OR Fourth hurdle OR Payer OR Pricing OR Funding OR Willingness to pay OR Threshold OR Value for money OR Social value OR Social preferences OR Public health |
Search branch 1 | • | AND • | AND • | ||
Search branch 2 | • | AND • | AND • | ||
Search branch ‘methods’ (1 + 2) | • | AND • | AND • | AND • | |
Search branch ‘decision making’ (3) | • | AND • | AND • |
-
‘Guidelines’ (search branch 1) OR
-
‘Methods’ (search branch 2) OR
-
‘Decision making’ (search branch 3).
-
they did not have a methodological purpose, or
-
they did not have a vaccine context, or
-
they considered exclusively non-industrialized countries.
2.2 Conduct of a Workshop
3 Results
3.1 Identified Studies
3.2 Identified Aspects for Discussion
3.3 Modelling Methods and Health Economics
3.3.1 Model Choice
-
For infectious disease modelling, the sole use of static models should always be justified. Static models can be used as a conservative estimate when there is no evidence for harm (e.g. age shifts with adverse effects) if indirect effects are ignored [4, 30, 34, 81, 82]. WHO has developed a flow chart that provides assistance when choosing an adequate type of model (Fig. 3) [4].×
-
A challenge is to handle realistic demographic predictions in models (with a long time horizon) because of migration, demographic changes and scarcity of contact studies in (special) populations.
-
Stochastic models;
-
Simulate a more realistic world.
-
Can follow an individual’s life course, which is easier for decision-makers to understand.
-
In stochastic models, the randomness is of first-order uncertainty; therefore, they provide an alternative to account for heterogeneity (if events are not rare) in subgroups as it is done in a deterministic model [81].
-
Model calibration and the probabilistic sensitivity analysis (PSA) become more challenging and computationally intensive, hence, transparency might suffer.
-
Data sources may not be accessible, as more fine-grained (non-aggregated) data are needed.
-
A remaining challenge is to find adequate ways to conduct efficient uncertainty analyses on stochastic models [83].
3.3.2 Time Horizon of Models
Evaluation strategy | Target population | Time horizon | Start of evaluation |
---|---|---|---|
Category 1 | Entire population | Fixed TH (several years or decades) | From implementation of vaccination |
Category 2 | Entire population | 1 year | From steady state |
Category 3 | Cohorta
| Cohort’s lifetime or fixed TH | From implementation of vaccination |
Category 4 | Cohorta
| Cohort’s lifetime or fixed TH | From steady state |
-
The time horizon of a dynamic model should last until the steady state is achieved in order to deliver valuable results. Hence, the model’s time horizon should not be defined prior to the analysis.
-
Future research must analyse the impact of these different strategies.
-
A model usually should be run to epidemiological equilibrium, but it ideally should reproduce historical epidemiological (and demographic) values that may not be in equilibrium. There is a need for validation of pre-vaccination as well as post-vaccination epidemiology (whatever is available/applicable). However, this is often technically challenging and may suffer from lack of data.
-
A surveillance system should be implemented to monitor the impact of a vaccination programme. The results can be used to compare the real impact with the impact predicted in the model that was developed before the vaccine was implemented and to evaluate the model.
3.3.3 Natural History of Disease
3.3.4 Measures of Vaccine-Induced Protection
-
There is a ‘sequential’ [targeting the first endpoint only, e.g. VE protecting against herpes zoster (HZ) only]
-
A ‘non-sequential’ [targeting all VE-relevant endpoints independently, e.g. VE protecting against HZ and also against HZ complication post-herpetic neuralgia (PHN)] approach
-
per protocol (PP) and
-
intention to treat (ITT).
-
The degree of protection (or leaky protection) is the percentage of (partial) protection in successfully vaccinated individuals (e.g. 100 % of vaccinated individuals have a protection of 50 %).
-
Take (or ‘all or nothing’) describes the percentage of successfully vaccinated individuals with full protection (e.g. 50 % of vaccinated individuals have a protection of 100 %).
-
The model structure should account for the type of VE measure incorporated. VE in terms of reducing susceptibility to infection is fundamentally different to VE reducing infectiousness. These different aspects of VE have a differential impact on the results. Modelling can be used to estimate unknown parameters including VE estimates by using, for example, a Bayesian framework utilizing Markov Chain Monte Carlo inference [93]. More studies assessing VE against infectiousness are warranted (e.g. challenge studies).
-
Model structure and the decision-maker’s research question determine the use of a sequential or non-sequential approach.
-
ITT data, when available, should be taken for base-case analyses and PP data for uncertainty analyses. However, the use of PP data for the base-case is sufficient when the difference between ITT and PP data is completely explained by the different proportions of susceptible individuals in the study population, since this is ideally incorporated in a model. PP data should be preferably chosen if a specific result on vaccine-dose compliance and/or completion of a vaccine course is of relevance.
-
The choice of representing VE with degree of protection versus take depends on the type of protection conferred by the vaccine of interest. When there is no evidence on whether the vaccine confers a leaky or an all-or-nothing protection, different approaches to account for vaccine efficacy (leaky or all-or-nothing or combination) should be used.
-
The quantitative relationship between immune response and the degree of vaccine-induced protection against clinical disease is often unclear. Validated surrogates can be considered if no clinical endpoints are available.
-
The impact of negative vaccine effects, both at an individual level (i.e. adverse events) and at a population level (i.e. replacement or age shift) needs to be considered. Cases of vaccine-preventable diseases and cases of adverse events are equally relevant outcomes.
-
Vaccine manufacturers currently have little incentive to collect some specific clinical data (e.g. head-to-head comparisons of different vaccine products); however, these are relevant for modelling and public health. For comparison of different studies, standardized case definitions for clinical outcomes are needed, and methods for the implementation of such indirect comparisons into models should be developed/standardized.
3.3.5 Duration of Vaccine-Induced Protection
-
If vaccine waning is not well understood, then different waning scenarios should be considered in an uncertainty analysis and their impact compared. Immunological memory can be integrated.
-
The availability of detailed trial data on vaccine-induced protection and waning at the patient level would enable more rapid and less uncertain economic evaluations following the marketing of a new vaccine.
3.3.6 Indirect Effects Apart from Herd Protection
-
Ecological effects such as intra-population immune boosting following exposure to a pathogen (cf. varicella-zoster virus), replacement of pathogen strains covered by a vaccine by strains not covered (e.g. pneumococcal serotypes) and antibiotic resistance should be part of uncertainty analyses whenever they are possibilities.
-
Ideally, pathogen replacement, eradication, genetic selection in host, changes in behaviour (e.g. screening uptake, risk behaviour such as unprotected sex or social mixing), weakening of maternal immunity, and using vaccination as a platform for adding other interventions should also be considered wherever relevant.
3.3.7 Target Population
-
Survey-based mixing data are considered most adequate and should be used wherever possible. Synthetic methods need to be validated against survey data where available.
-
Future research should better measure contact patterns for children and parents, and evaluate what kind of contact is relevant. Knowledge is still lacking about how well contact patterns represent occasions for transmission for specific infections. It is therefore important to assess which subset of mixing data (e.g. contacts involving touching) provides the best fit to relevant observational data for the disease in question (e.g. seroprevalence data). Research funders need to understand that contact matrices are important for dynamic model projections, and that these differ between countries or regions.
3.3.8 Model Calibration and Validation
-
The manual calibration approach should be based on a structured process, and the algorithm should be reported. However, the random or optimized approach is considered to be more adequate [53, 105‐107]. A random calibration approach may have an identifiability issue. Hence, researchers should make sure that the shape and range of the posterior distribution is plausible.
-
A plain visual validation is not considered sufficient. Instead, the use of goodness-of-fit criteria is recommended.
-
The dataset used for validation should be independent from that used for calibration (ideally even with different endpoints). An alternative is to hold back a portion of the calibration data (e.g. test/training datasets). An alternative option is a cross-model validation approach in which the same data are used on different models. Lack of data points might be a limitation.
3.3.9 Handling Uncertainty
-
All identifiable sources of uncertainty should be accounted for, if not by PSA then by other analyses. The parameter distributions used in PSA need to be justified. Transparency is important because dynamic models can have many ‘deep parameters’ (i.e. parameters that are not directly observable, such as the probability of infection transmission per contact event). Calibrated parameters also need to capture information about uncertainty [87, 112].
-
Decision-makers need to understand the relevance of uncertainty analyses. PSA can help to identify future research priorities. However, uncertainty measures in calibrated parameters remain challenging.
-
Structural uncertainty should be parameterized where possible, but uncertainty in normative aspects such as perspective, vaccine price and discounting should not be analysed in PSA [87, 112]. Vaccine coverage should be varied between desirable and undesirable levels. Uncertainty in contact patterns should be parameterized wherever possible (see Table 3).Type of uncertaintySensitivity analysisScenario analysisParameter uncertaintyMethodological/normative uncertaintyStructural/model uncertaintyDeterministic sensitivity analysisYesYesYesYesPSAYesNANANAExamples• Efficacy• Costs• Transmission dynamic vs. discrete-event simulations• Discount rate• Presence of a immune state (SIS vs. SIR)• Pathogen strain competition and replacement• Coverage• Target age/risk group• Vaccine price
3.3.10 Discounting
-
The majority of, but not all, experts recommended differential discount rates for costs and effects in HHEs (exclusively in cost-utility and cost-effectiveness analysis) if the model’s time horizon is long (e.g. >20 years).
-
The discount rate of health effects could be around 50 % of the discount rate for costs. However, for a more evidence-based recommendation, empirical research has to be conducted [39].
-
Constant discount rates over time should not be applied in models with a long time horizon (e.g. >20 years) according to a majority of experts.
-
Since discount rates and discount approaches usually have a major impact on results of HEE of vaccines, the variation of these aspects need to be analysed (see Sect. 3.3.9) and explained to decision-makers.
3.3.11 Health-Related Quality of Life and Quality-Adjusted Life-Years as Outcome Measures
-
HR-QOL of carers should routinely be considered in uncertainty analysis in both payers’ and societal perspectives. However, input data might be scarce.
-
Utility in anticipation and fear of adverse events are also difficult to consider due to limited data. When appropriate data are available, they might be considered in uncertainty analyses.
3.3.12 Cost Components
-
Indirect costs of carers should be considered for both perspectives. From a payers’ perspective, they are considered as sick pay (if the payer has to cover these costs) and from a societal perspective as productivity loss.
-
If set-up costs (e.g. for campaigns) are not included in the vaccine price (i.e. promotion and distribution of a vaccine is not done by its manufacturer), they should be considered in the perspective that covers these costs.
3.3.13 Perspectives
-
A societal perspective should ideally be taken for the base-case analysis when considering infectious diseases (i.e. not for vaccines exclusively), unless this contradicts national guidelines.
3.4 Decision Making
3.4.1 Purposes of Health Economic Evaluations in Decision Making
3.4.2 Integration of Health Economic Results in Decision-Making Processes
3.4.3 Key Parameters that Should be Varied in Uncertainty Analyses
3.4.4 Vaccination-Specific Aspects of Reporting Results
-
Discounted and also undiscounted results should be presented.
-
Cumulative results should be reported at various time points over a model’s construed decision horizon, including a longitudinal view up to the end of the defined time horizon of the model.
-
Results from various relevant perspectives.
-
Cost-effectiveness acceptability curves (CEACs).
-
Best- and worst-case scenarios.
-
Absolute values and ICERs for all disease-specific outcomes.
-
A report of an HEE should describe the validation/calibration process, the strength of evidence behind the input data, and should discuss the potential variation of results in uncertainty analyses.
-
The most recent questionnaire [71] assessing the credibility of a modelling study needs at least one infectious disease-specific addition: “If applicable, why was a dynamic model not used?”
4 Summary
-
In general, international standards as laid down in established guidelines should be applied and adopted to specific problems where necessary.
-
HEEs on vaccines and vaccination programmes should be considered by decision-making bodies such as NITAGs when considering inclusion of a new vaccine into the national programme to avoid inferior allocation of resources.
-
A mechanical use of a threshold without considering other criteria may not be necessary. However, information about incremental costs and incremental outcomes of relevant vaccination strategies and ICERs (with adequate comparator(s)) should be delivered to decision-making bodies.
-
Other interventions (e.g. drugs in preventive medicine) often share similar characteristics as vaccines. Vaccines should only be treated differently where they really are different (e.g. indirect effects).
-
HEEs must be objective, systematic and transparent.
-
HEEs should be as complex as necessary but as simple as possible.
-
Ideally, infectious disease models should be dynamic.
-
Models should focus on patient-relevant clinical endpoints wherever possible. But surrogates may be used if no clinical endpoints are available, preferably if they are validated. Both relying on surrogates and disregarding them is risky. The uncertainty concerning surrogates should be made clear to decision-makers.
-
Future costs and outcomes should be discounted. There are arguments for differential and over time decreasing discounting (not only for vaccines).
-
Information about costs and outcomes from the societal perspective is relevant. This perspective should ideally be reported in addition to the payer perspective.
-
A broad set of utility-generating characteristics (such as carer quality of life, utility in anticipation) may be adequate. However, more research is needed.
-
All uncertainties should be accounted for. Uncertainty analysis plays an important role.
-
Methodological problems need to be solved. However, it is not necessary to reject HEE per se because of the methodological challenges.
-
Funders and decision-makers should recognize that HEEs (of vaccines) demand time and resources.