Did we see it Coming? An Evaluation of the Swedish Early Awareness and Alert System

As already stretched healthcare budgets are put under additional pressure with the introduction of new expensive medicines [1‐4], payers are increasingly seeking greater value for their money [5‐10]. In addition, timely access to new and meaningfully better medicines to improve patient outcomes is considered a priority in many countries [11]. The early identification of new emerging technologies prior to their introduction can enable timely assessments to be undertaken and activities established to facilitate rational uptake and utilization. However, scrutinizing the research and development pipeline to identify medicines that may impact healthcare is notoriously difficult and requires considerable expertise. Horizon scanning systems [also known as early awareness and alert (EAA) systems] [12] can provide the necessary support to health technology assessment bodies and payers.

Early awareness and alert activities are now recognized as a key element in processes for the introduction of new medicines [7, 13]. In view of this, the structure, processes, outputs, and impact of an EAA system should be regularly evaluated and tailored to meet customer needs [14, 15]. It is reasonable to assume that an established EAA system carries out evaluations of its work as part of improvement initiatives. In Sweden, for example, feedback from stakeholder interviews was used to improve both the process and outputs of the Swedish EAA System [16]. However, such evaluations are typically internal and of a qualitative nature. Furthermore, they may not be readily accessible to wider audiences, thereby hampering the opportunity to learn from experiences across systems. To date, EAA systems with published evaluations include the UK’s National Institute for Health Research Horizon Scanning Research and Intelligence Centre [17‐19], the Austrian Horizon Scanning Programme in Oncology [20], and the US Agency for Healthcare Research and Quality Healthcare Horizon Scanning System [21].

The Swedish EAA System has been performing EAA activities at the national level since 2010. It was established to support the long-term planning of the county councils and to optimize the readiness of the healthcare system to the introduction of new medicines. A detailed description of the Swedish EAA System is available elsewhere [22]. Both regional and national decision makers use the outputs of the system, including early assessment reports produced for prioritized medicines, to keep informed and prepared for the introduction of new medicines. The purpose of this study is to assess whether the Swedish EAA System identified and prioritized new medicines that would go on to have substantial economic impact.

During the study period (2010–2015), we identified 462 medicinal products that were authorized or refused by EMA and an additional 64 medicinal products whose initial authorization application was withdrawn by the pharmaceutical company. After applying our exclusion criteria, a study population of 253 medicinal products was yielded (Fig. 1). Most commonly, in almost three out of four cases, the excluded medicines were either generics or known active substances intended for use in an already approved indication. All 253 medicinal products were identified by the Swedish EAA System.

During the study period, 104 early assessment reports were published (Fig. 1). Of these, 33 reports were excluded from our analyses [see Electronic Supplementary Material (ESM)]; most frequently for covering extensions of indications of already authorized medicinal products. This left us with 71 prioritized medicines (positive index test).

Table 1 provides a description of the total study population and the prioritized medicines. The majority of the medicinal products processed by EMA were classified as new active substances (86%) and/or were granted marketing authorization (83%); among the medicines prioritized by the Swedish EAA System these proportions were slightly higher at 93% and 90%, respectively. Antineoplastic and immunomodulating agents were the most common group of medicines overall (29%) and among the prioritized medicines (51%). Early assessment reports were published around 3 months ahead of the date of marketing authorization [median (interquartile range) number of days: 87 (40–187)].

In total, based on analyses of sales data, we classified 21 new medicinal products as having substantial economic impact on the healthcare system (positive reference test). An overview of these medicines is provided in Table 2.

A tabular comparison of the prioritizations made by the Swedish EAA System and the economic impact is provided in Table 3 with corresponding measures of prioritization accuracy in Table 4. The Swedish EAA System prioritized 16 medicinal products for a sensitivity of 76%. The other prioritized medicines (n = 55) were among the 232 medicinal products that did not exceed €4 million in sales (corresponding to a specificity of 76%).

Subgroup and sensitivity analyses were generally in line with the main analyses. The accuracy of prioritization, in terms of sensitivity, was highest for antineoplastic/immunomodulating agents (100%) and new active substances (76%). Conversely, the lowest accuracy of prioritization was observed when using the lower sales threshold (65%). Detailed data are presented in the ESM.

We found that the Swedish EAA System identified all new medicines that would go on to have substantial economic impact and prioritized (i.e., produced early assessment reports for) most of these medicines. The few non-prioritized medicines whose sales exceeded the threshold were identified by the system but not selected for an early assessment report because similar medicines had been already prioritized and/or marketed earlier (e.g., Sovaldi® had already been selected for a report prior to Daklinza® and Harvoni®, thus the latter two were not prioritized). Overall, these findings indicate that decision makers in Sweden have been accurately informed about impending new medicines prior to their launch. This awareness allowed, for example, for special activities to be set up to facilitate a rational and cost-effective introduction of new medicines into clinical practice both at the regional [25, 26] and national [27] level. Notably, all new medicines comprising the study population were identified by the Swedish EAA System. For many of the non-prioritized medicines, for which no early assessment report was produced, information was summarized and disseminated in other ways (e.g., a list of all identified orphan medicines is regularly updated and provided to both regional and national stakeholders).

Potential economic impact is however only one of many criteria that the Swedish EAA System weighs in the decision to prioritize a medicine. For example, factors contributing to the prioritization of obesity medicines [e.g., phentermine/topiramate (Qsiva®)] included the patient population size, safety aspects, and anticipated media interest. New medicines that are seen as innovative methods of treating a disease are also prioritized. Bezlotoxumab (Zinplava®) as passive immunization against Clostridium difficile toxins, ipilimumab (Yervoy®) for the treatment of advanced melanoma, olaparib (Lynparza®) as maintenance treatment of platinum-sensitive recurrent BRCA-mutated ovarian cancer, and ranibizumab (Lucentis®) for the treatment of visual impairment due to diabetic macular edema were primarily prioritized for this reason [16]. The fact that the sales figures do not necessarily reflect all prioritization criteria explains the relatively low positive predictive value (range: 12–35%), i.e., fewer than one in three prioritized medicines went on to have a substantial economic impact. The prioritization of such medicines may however be equally important. For example, new innovative methods to treat diseases may have systemic effects on the delivery of healthcare services and require preparation ahead of the introduction. Consequently, given the holistic approach to prioritization employed by the Swedish EAA System combined with their efforts to improve efficiency (e.g., the Swedish EAA System typically does not prioritize medicines with existing mechanisms of action for indications or orphan medicines), it is difficult to propose revisions to their working methods that would lead to improved sensitivity and positive predictive value.

As a byproduct of our analyses, we obtained an overview of all new medicines assessed by EMA during the period 2010–15. We identified 217 new active substances that were submitted to EMA during these years. One fourth of these were in the oncology/immunology field, and a considerable number of medicinal products were also introduced for metabolic (diabetes and rare diseases) and infectious diseases. Correspondingly, over half of the medicines prioritized by the Swedish EAA System were in oncology or immunology, which is similar to outputs from other EAA systems [19, 20].

Apart from providing an assessment of how the Swedish EAA System has performed so far in identifying and prioritizing new medicines that would have substantial economic impact, this study also contributes to the scarce literature on the evaluation of EAA activities and systems. As outlined in the EuroSCAN methods toolkit, evaluation is an important element of an established EAA system [14]. However, few reports on such evaluations have been published to date [15, 17‐21, 28]. This could possibly be explained by a lack of scientific interest and involvement, but also by methodological challenges, including the definition of a reference standard to be used for comparisons. The ideal reference standard should be able to reflect the prioritization criteria used by an EAA system but it may not be feasible to obtain a single measure that captures the criteria in their entirety (e.g., the Swedish EAA System’s prioritization criteria range from “potential to affect treatment costs” to “legally, ethically, or politically interesting”).

In addition to sales data, we also considered using other variables, including the size of the patient population, health technology assessment recommendation, and level of innovation. However, a reliable estimation of the size of the patient population proved to be difficult primarily owing to a lack of data on patient counts, particularly for narrow indications. Results of health technology assessments are only available for medicines that are included in the pharmaceutical benefits scheme, but many of those prioritized are intended for hospital use only for which, during the study period, no national appraisal was made. Finally, we also explored approaches to measure the level of innovation but found no universally accepted standard that could be used in our study.

Using sales data as a reference standard appears to capture both directly and indirectly some of the prioritization criteria used, including the size of the patient population, the potential to clinically improve patient outcomes, and the potential to affect treatment costs. In addition, sales objectively reflect which new medicines are actually adopted in clinical practice. However, we encountered a number of limitations with using sales data as a reference standard. We acknowledge that our choice of the timing of sales (i.e., second and third year) and the cutoff value is a pragmatic approach to operationalize our analyses.

Measuring sales early in the product life cycle may underestimate the number of impactful medicines. Reasons for this include medicines with slower uptake and medicines initially marketed in narrower indications (e.g., non-vitamin K antagonist oral anticoagulants). We believe, however, that early sales are well suited to capture medicines with a strong immediate impact that warrant preparedness by the healthcare system. The prioritization accuracy was also sensitive to the cutoff values used in our analyses. In the main analyses, we used a pre-specified threshold of approximately 0.1% of yearly total pharmaceutical sales. While this may not seem like a high level, in absolute terms the impact can be substantial, especially for specialist medicines used in a limited number of hospitals. Moreover, the aggregate nature of sales data does not allow us to attribute sales to a given indication for medicinal products that are used in multiple indications.

We also excluded all new vaccines from our analyses as no complete sales data were available because of separate distribution chains. While no vaccines were prioritized during the study period, more recently, in 2017, the Swedish EAA System prioritized the herpes zoster vaccine Shingrix® [29]. Recent progress in vaccine research and development may translate into an increase in breakthrough therapies [30] that warrants the inclusion of vaccines in future evaluations.

Finally, there is a possibility that an early assessment report could have influenced sales as the two measures are not independent (e.g., taking into account the information provided by the Swedish EAA System, the New Therapies Council or individual counties may have recommended not to use a given medicine). Analyses of  information on positive and negative recommendations could have been a possible way to address this; however, national-level recommendations were not provided until recently and, moreover, it was not possible to systematically collect data on all decisions taken at a county level.

This study was carried out as an independent evaluation of the Swedish EAA System. While all authors have knowledge of this particular EAA system and the role of EAA activities in the process of introduction of new medicines in general, none of the authors has been involved in decision making on which medicines to prioritize. Furthermore, our study is the first evaluation of an EAA system to use sales data as a reference standard to define which new medicines impacted the healthcare system. Despite the aforementioned limitations of such a one-dimensional measure, we believe it is nonetheless of value to explore its utility and share the results of this study with stakeholders both in Sweden and internationally.

To summarize, our findings provide reassurance to decision makers who rely on the outputs of the Swedish EAA System. Moreover, this study also contributes to the scarce literature on evaluations of EAA activities and systems and thus provides valuable insights to stakeholders willing to establish or evaluate their own EAA activities.