Different Weights of the Evidence-Based Medicine Triad in Regulatory, Health Technology Assessment, and Clinical Decision Making

In the early 1970s, Archie Cochrane described his fundamental clinical decision dilemma: “I had considerable clinical choice of therapy: my trouble was that I did not know which to use and when” [1]. Consequently, EBM and the respective triad was gradually established. As David Sackett, one of its founding fathers, has put it, EBM is: “a systematic approach to clinical problem solving by the integration of best research evidence with clinical expertise and patient values” [2]. Since those early days, decision requirements gradually expanded from patient-centric clinical decision making to a broader, population-based view, including the development of clinical guidelines, regulatory assessments, and, more recently, health technology assessments (HTAs), respective appraisals, and related reimbursement decisions. The different scope of decision situations (i.e. patient-centric vs. population-based) necessitates a different weight of the three pillars of evidence within EBM.

Regulatory bodies, such as the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA), are the most critical public decision makers in the early lifecycle of new products. EBM forms the conceptual framework for their decisions. However, the early phases in the lifecycle of new products are characterized by limited ‘real-world experience’ and a limited understanding of ‘patient preferences’, which are both not formally required to obtain approval for a new technology. For the purpose of this manuscript, we used the term ‘real-world experience’ to describe the body of real-world clinical experience and evidence gathered by clinicians when treating their patients. While initial clinical experience may also be developed within a clinical trial setting, the term primarily includes clinical information and experience gathered outside the controlled, standardized, well-documented results of clinical trials. Regulatory decisions primarily rely on data from pivotal clinical trials, which we shall refer to as ‘research evidence’. Randomized clinical trials (RCTs) are still the gold standard of research evidence. However, a recent review of pharmaceutical regulatory approvals found that, between 1999 and 2014, 76 drug approvals for indications were granted without evidence from RCTs [3]. The traditional RCT-based research evidence paradigm is being challenged by the establishment of accelerated regulatory pathways, a shift towards continuous development of clinical data with the integration of comprehensive information into ‘Big Data’, and the targeting of smaller patient populations through precision medicine, or development for orphan and ‘super-orphan’ indications [4]. The EMA’s guidance on adaptive pathways is taking this discussion to the next level. The three guiding principles of adaptive pathways are (1) iterative, stepwise generation of evidence, (2) development and consideration of real-world data as supplement to clinical trial data, and (3) early involvement of patients and HTA bodies [5]. This represents a reliable and pragmatic approach by regulators to balance the benefit and risk of innovative medicines in a new, more targeted research environment.

Regulatory bodies commonly use the phrase ‘totality of evidence’ in describing what is to be considered for any regulatory decision [4]. This term usually refers to a systematic review of the existing body of knowledge in the early phase of a product, which can entail not only clinical but also pre-clinical data [6]. While this approach is more inclusive than, for example, many HTA-body assessments that primarily rely on RCT evidence, it still reflects only the totality of ‘research evidence’, as real-world expertise and data on patient preferences are largely unavailable in those early phases.

The decision competency of HTA bodies is reaching far beyond the regulatory assessment of benefit and risk. HTA bodies are the gatekeepers of reimbursement for new products. Economic aspects are coming into play that, by their nature, necessitate a conflict. Richard Smith, the former editor of the British Medical Journal, pointed to this conflict of interest when arguing, that NICE “corrupts the concept of evidence-based medicine” [7]. Until today, NICE keeps a ‘Save Money’ link on its homepage, providing insights of the savings they generate [8]. This focus creates a general conflict with the aims of EBM. The founders of EBM have anticipated this challenge and mentioned that there is a risk of fundamentally misunderstanding the financial consequences of EBM, as it ‘may raise rather than lower the costs of care’ [2]. However, efforts are made to better align the assessments by regulatory authorities (i.e. EMA) and HTA bodies with the aim of providing patients with early and broad access to innovative medicines [9].

HTA bodies have a different scope on the available ‘research evidence’ than regulatory bodies. They rather focus on the available clinical data. Pre-clinical data and data on e.g. mechanism of action, which are part of the totality of data that regulatory bodies are referring to, are of less relevance to HTA bodies. Furthermore, they apply a very strict ranking of available evidence, with anything but RCTs having a difficult time in their assessments.

With regards to the EBM triad, the German IQWiG institute is rather explicit about its scope. Thus, predictions of success for individual cases are considered principally not possible. Additionally, ‘experiences based on individual cases are unsuitable for a benefit assessment, as it is not possible to ascribe the results of an individual case to the effect of an intervention’ [10]. ‘Real-world experience’, which is based on experience in individual patients, is therefore mostly excluded from the benefit assessments. In fact, IQWiG mostly selects physicians who have not participated in the respective clinical development programs as advisors for respective benefit assessments. It is hard to determine when real-world experience should overrule RCT results. However, RCTs are always limited to certain questions and a certain patient population, while clinical reality reflects the whole variety of ‘real world’. As such, the concept of denying patients access to potentially beneficial treatments based solely upon an HTA decision-maker’s interpretation of RCT results alone is flawed.

‘Patient preferences’ are a challenging area for HTA bodies. While the German Federal Joint Committee (FJC) continuously supports and requests those data, none of the available assessments has been granted an additional benefit solely based on the patient-reported outcomes [11]. The generic European Quality of Life Instrument (EQ-5D) is requested as a base for utility calculations by NICE but is not highly regarded by FJC and the French Haut Autorite de Sante (HAS). Furthermore, it needs to be taken into account that utilities, as measured by the EQ-5D, are reflecting the societal perception on a certain condition, rather than the individual preferences of a patient suffering from a certain condition. Current public-private efforts such as the PREFER initiative (Patient Preferences in Benefit and Risk Assessments during the Treatment Life Cycle) are aiming to systematically examine and include the patient perspective throughout the regulatory and HTA process [12].

Optimizing clinical decision making was the initial aim of the founders of EBM [1, 2]. To achieve this goal, they aimed to integrate the three pillars: ‘research evidence’, ‘real-world expertise’, and ‘patient preferences’. Finding the right balance was considered critical to practicing good medicine. “Without clinical expertise, practice risks becoming tyrannized by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient” [2]. Soon the question came up, if one of the three pillars should be considered first among the equals. Haynes et al [13] suggested a higher ranking for real-world expertise, in the sense that it should encompass all other sources of evidence, consider the trade-offs of the various perspectives and finally determine the best way forward in the interest of the patient. However, this patient-centric view on EBM gradually separated from the population-centric approaches of regulatory authorities, HTA bodies, and guideline committees [14] leading to the perception that EBM is a movement in crisis [15]. According to Greenhalgh et al [16] EBM, as it is applied by HTA bodies, regulatory bodies and guideline committees, tends to devalue the patient and carer experience. They therefore suggest a couple of approaches to increase the emphasis on these two pillars of EBM, which are ultimately relevant for good clinical decision making.

Roger’s theory on ‘Diffusion of Innovations’ [17] indicates that knowledge and data are usually the first steps in the adoption curve of new technologies. Experience of peers and own experience occur later in the lifecycle but are the key drivers for successful implementation. In EBM, decisions on a public health level (i.e. regulatory and HTA organizations) occur early in the lifecycle of new products. Their primary source of evidence is research data. Real-world experience and preferences of individual patients are scarce in those early days, but are decisive in the later phases of the adoption curve. The art of appropriate decision making in a regulatory and HTA environment is to consider all available data but always keep an open mind for the other two essential pillars of EBM. The lack of strong input from a real-world clinical experience perspective, or a patient preference perspective, within HTA decision making needs to be proactively addressed and managed in order to avoid, for example, unavailability of beneficial treatments for patients (see Table 1). The example of osimertinib, an innovative oncology product that has been pulled from the German market after a negative benefit assessment, indicates such a system failure. Although largely supported by the physician community [18], the product did not receive a positive benefit rating by the FJC, as the comparative RCT data were not yet available.

Those failures are the consequence of an imbalanced approach with regards to the three pillars of EBM and require correction in the future. In a most recent appraisal in ROS1-positive lung cancer, the FJC acknowledges, that despite the fact that no additional benefit has been demonstrated, crizotinib may be a relevant therapy option in individual cases [19]. From our point of view, this indicates an improved reflection of the three pillars of evidence in the FJC’s decision making.