Cost effectiveness of pediatric pneumococcal conjugate vaccines: a comparative assessment of decision-making tools

The availability of a seven-valent pneumococal conjugate vaccine (PCV7), which is efficacious in preventing invasive pneumococcal disease (IPD), pneumonia, and acute otitis media (AOM) [1‐3], has prompted policymakers to ponder whether or not this vaccine should be included in the national immunization program. The more recent licensure of higher valent PCV vaccines (10-valent and 13-valent pneumococal conjugate vaccines) makes this a harder task still. Cost-effectiveness (CE) analyses can be useful to aid decision making. Numerous economic evaluation studies of PCV-7 have been published since 2000 [4‐29]. Despite the availability of a comprehensive critical literature review [30], there is a lack of critical review of the computational reliability and convergent validity of accessible PCV models [31]. Indeed, there is a constant need to verify the actual model structures, model attributes, input parameters, key assumptions and whether different models will produce the same outputs if standardized input parameters are used. Moreover it is not clear what the strengths and possible pitfalls of the available PCV7 models are and the potential to use it for new PCV combinations. In order to guide policymakers in particular from low-income and middle-income countries (LMIC) on the optimal use of decision-making tools, in 2009 the World Health Organization (WHO) invited model developers from both the public and private sectors to submit their tools for a systematic comparative assessment. Rather than having the aim of recommending a single tool, the overall objective of the assessment was to provide decision makers with a menu of CE tools and their characteristics for their optimal use for vaccine adoption decisions.

This paper critically reviews a number of potentially influential and accessible cost-effectiveness tools. We assess the differences and similarities between models and evaluate to which extent the outputs differ given the use of standardized input parameters.

To the best of our knowledge, this research work is the first study conducted with an aim to compare accessible models developed by both the public and private sectors. This review provides insights into differences and similarities of model attributes and key assumptions to understand disease dynamics and key drivers of the results. Importantly, having a full understanding why different tools produce different results will help identify data needs and pitfalls important to the users of such models. Although the models we reviewed were developed to determine value for money of vaccination program, different vaccines were assessed: PneumoADIP assessed the CE of PCV, TriVac assessed the CE of PCV, Haemophilus influenzae B (Hib) and rotavirus vaccines and SUPREMES appraised pneumococcal Haemophilus influenzae protein D-conjugate vaccine (PHiD-CV).

The different model approaches translated in different data input requirements, which made it a challenge to use a standardized data set for the tool comparison exercise. The incidence-based cohort approaches adopted in the PneumoADIP and the PAHO TriVac models are more data demanding and may be more intuitively appealing than the SUPREMES model. The latter model evaluates under static conditions an entire population at specific time point using the total population size and it specific age distribution to estimate the impact of a new vaccine program on total population health during a fixed time period (usually 1 year) and therefore requires fewer data inputs. This feature makes the SUPREMES model useful if the vaccine causes intermediate and important changes to the population and if budget-impact estimates after vaccine introduction are of interest to the decision maker. In addition, the model is easier use and can be applied widely to those countries with limited data but may be more difficult to justify as users would need to agree with several implicit assumptions embedded in the modeling approach.

It is important to note that while all three models were at an advanced stage of development at the time of the comparison exercise, models of this kind are prone to continued extension and elaboration, and weaknesses or absent features listed in this article may be corrected or improved with time. This comparison process can serve a useful purpose in this regard, by giving modelers the opportunity to identify key omissions and improve the accuracy, transparency and usability of their models. For example, since the evaluation in August 2009 the TRIVAC model now includes otitis media outcomes, advanced options for probabilistic sensitivity analysis and scenario analysis, and key assumptions about waning efficacy and age-appropriate vaccination. The present version of the PneumoADIP model allows discount rates to be varied while GSK now has developed a cohort-based version of the SUPREMES model to deal with the different vaccine impact on disease events using cohort-based or population-based model methods [38]. In addition, while none of the three models had been through a validation process at the time of the comparison exercise, the TRIVAC model has since been evaluated by an external expert panel [41]. As models and cumulative effects analysis (CEA tools evolve continued appraisal is needed.

The differences between these models are not only due to the approach and data requirements, but are difficult to distinguish without fully transparent user manuals. The TriVac model has an advanced user-friendly and self-guided function enabling users to properly interpret the research findings.

All three models include the possibility to incorporate the herd immunity effects of the vaccination program, however there are important differences in how this is incorporated and the underlying assumptions in each model. Herd immunity effects have already been shown to be influential in various analyses published before 2006 [30]. However, the closely related effects of serotype replacement, which can partly nullify the beneficial impacts of herd immunity are not taken into account in any of these models. All three models have shown that they acknowledge an important component of outcome, the morbidity or health-related quality of life, because they present the results in terms of incremental costs per DALY averted (PneumoADIP and TriVac) or per QALY gained (SUPREMES).

From the user's perspective, it is very important to be able to identify influential parameters. A series of sensitivity analyses using tornado diagrams has shown that vaccine efficacy against pneumonia, serotype coverage and replacement, vaccine coverage, and disease burden parameters including incidence and case fatality of pneumococcal diseases are the key driver of the outcomes. Unfortunately, evidence to inform one or more of these parameters is often lacking in low-income and middle-income countries and frequently parameters are populated through generalizations made using data from industrialized countries. Vaccination costs was also a parameter of major importance (though not included in the standard sensitivity analyses).

The outputs for TriVac and PneumoADIP were found to be very similar, while the outputs of the SUPREME model were markedly different from the other two models. Given the differences in various aspects such as parameters included in the models, the baseline value used in the model, and model structure, it is not possible to distinguish the sources that could explain the differences in findings. Because discounting is very influencing in this long-term outcome modeling, the lack of discounting in the SUPREME model could be considered a crucial factor that might explain, to a certain extent, the differences in QALYs gained and cost differences. To be more specific, the future outcomes earned by cohort vaccinated previously are measured at the same time for cohort currently being vaccinated.

The main aim of our comparison was not to make the difference between models understandable to model developers but foremost to potential users of these models at the country level. Understanding the differences and similarities of these CE tools will enhance the intended use of these tools by decision makers in terms of feeling ownership of results. This structured tool assessment exercise has shown that the models we compared differed in terms of model structure and key assumptions. Using a standardized input data set for three PCV cost-effectiveness analysis tools revealed that vaccine cost, efficacy and assumptions for critical clinical endpoints are the most influential parameters. Comparison of CE tools should help local policymakers understand the different approaches used in these models, and their advantages and disadvantages in relation to the data they have at their disposal to conduct and economic analysis. For instance, due to the complex nature of the transmission and ecology of pneumococci, many simplifying assumptions were implicitly made about the transmission dynamics in each of these models. All assumptions and simplifications should be well understood, and be transparent to users of such models. Models should be as transparent as possible according to the criteria and requirements presented in the WHO CEA guidelines [37]. By identifying the most relevant key parameters influencing the results of these models, country and region-specific data collection could be focused as appropriately as possible. Priority should be given to improve the data to inform key parameters. In addition, efforts to improve and develop transmission dynamic models with realistic age structures that better describe the natural history of Streptococcus pneumoniae infection and progression to pneumococcal disease and capable of improved investigation of herd immunity and serotype replacement effects are also important. Lastly, the fact that the tools are evaluated and other tools are publicly available enables different stakeholders to learn from each other's experiences. Adherence to the WHO guide of economic evaluations of immunization programs may also facilitate this process [37].