Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients

Healthcare decisionmaking is a complex process requiring simultaneous consideration of a number of elements including scientific judgment, economics and ethics. The cost-effectiveness (CE) model has become a prime model for healthcare resource allocation and decisionmaking globally. It was developed to support decisionmaking by integrating into unified metrics some of the key elements considered to be important. Although the methods developed in this field are valuable for examining the consequences of new healthcare interventions, the focus on CE ratios (e.g. cost per quality-adjusted life year [QALY]) has contributed to a "black box" syndrome, both at the clinical and policy levels[1, 2] In addition, healthcare decisions need to be based on a wider set of considerations that are not part of the CE model such as current need, lack of treatment and disease severity [3‐6].

A number of multicriteria models have emerged to support deliberation and assist consideration of the numerous factors implicated in healthcare decisionmaking [7‐15]. Some elements of decisionmaking can be quantified, and multicriteria decision analysis (MCDA) provides a way to account for multiple streams of information [16]. MCDA is emerging as a tool that goes beyond cost-effectiveness by allowing integration of more elements, such as disease severity [16‐18]. In addition, MCDA provides a mechanism that allows decisionmakers to gain insight into their priorities and values [19]. However, not all elements of decision are quantifiable (e.g., ethics, historical context) and may be difficult to incorporate into an MCDA model. Culyer [20] suggested a process that blends algorithmic (quantitative) and deliberative (non-quantitative) approaches. Such a comprehensive framework should allow explicit consideration of all elements of decision by a wide range of stakeholders [21] to provide accountability for reasonableness [22].

Another critical point is how to inform decisionmakers on those elements of decision, the goal of health technology assessment (HTA) activities--currently carried out by governmental agencies, public and private payers and manufactures around the world [5, 23]. HTA is as useful as the data available to build it, highlighting the critical impact of clinical trial design, which is heavily used to assess efficacy, safety, patient reported outcomes and economic outcomes [4], and the transparent reporting of results [1]. To fulfill their roles, HTA producers should also inform socio-ethical dimensions of new interventions [24]. However, although ethical evaluation helps stakeholders realize the consequences of implementing a healthcare intervention at the micro (patient), meso (institution) and macro (society) levels [25], only 47% of the International Network of Agencies for Heath Technology Assessment (INAHTA) member organizations reported including ethics in their assessments [26].

A decisionmaking framework bridging HTA with MCDA was proposed [27] that provided a pragmatic link between HTA and healthcare policy and clinical decisionmaking. In a proof-of-concept study, the preliminary framework was applied to 10 drugs and tested by 13 Canadian stakeholders during a panel session (submitted manuscript). In the current study, a complex case was tested to further explore the non-quantifiable elements of decision, to develop a comprehensive framework supporting consideration of all elements of decision, and to explore the validity of this approach. The use of growth hormone (GH) to treat patients with Turner syndrome (TS) was selected because of the complexity of the considerations surrounding expensive hormone injections over long periods of time to augment height in growth-delayed children affected by this syndrome. It was also chosen because it is an approved therapy and is used worldwide in developed countries for Turner syndrome, a genetic condition affecting between 1 in 2000 and 1 in 2500 live born females [28, 29].

The current debate on transparency and legitimacy of healthcare decisionmaking [41, 42] calls for a more systematized approach to evidence review and decisionmaking. In particular, ethical and colloquial (e.g., not based on scientific evidence) considerations are inherent to the thought process underlying healthcare decisionmaking [21, 43] but are often not explicitly acknowledged or discussed, let alone communicated to other stakeholders. The EVIDEM framework proposes a comprehensive set of components of decision to consider in making healthcare choices, and transparent access to available data on which these considerations are based. This approach is envisioned to help identify the issues at play and increase legitimacy and transparency of decisions.

Testing the EVIDEM framework using the complex case of growth hormone for patients with Turner syndrome allowed development of a complementary tool (Extrinsic Value Tool), thus expanding the comprehensive nature of the framework. Applying the framework to this case clearly exposed limitations of the cost-effectiveness paradigm and highlighted the importance of systematically considering all relevant aspects of an intervention to best reach an informed decision. It also revealed gaps in available data and a need to better align research questions with data needs. Exploratory validation of the approach provided support for the inclusion of most framework components and showed good agreement of MCDA value estimates at the individual level (test-retest) but large variations across panelists reflecting different perspectives and personal values.

This case study exemplifies how MCDA can provide a means to integrate a wider range of components into the decisionmaking process [16] and help consider each of them explicitly. The use of growth hormone is complex and raises many questions, including far reaching issues such as social perception of short stature (What is short stature? Why is it an issue? What is the value of an incremental gain in stature?) In addition, consideration of the ethical components of the Extrinsic Value Tool brought up questions of fairness towards Turner syndrome patients versus other individuals with short stature [33, 44]. Decisionmaking that solely relies on the cost per QALY ignores these issues, leaving them at best to ad hoc discussions. The clear inadequacy of the available utility data and the great difference between discounted and non-discounted incremental cost-effectiveness ratios (because high costs are incurred during childhood, but benefits accumulate over adult life-time) observed for this case study cast additional doubt upon reliance on the cost per QALY measure and highlight the usefulness of a more comprehensive approach.

In a first step, applying the framework allows eliciting the personal values of the committee/panel members (weighting). Variations across individuals are expected and were observed in the case study reflecting different perspectives and potential bias. Of note, panelist weights were lowest for "Clinical guidelines", suggesting limited impact of these, perhaps related to the current debate surrounding guideline limitations [45].

In a second step, combining panelist weights and scores for the intervention, produces an MCDA value estimate--a panel-specific comprehensive measure integrating relative value (i.e., comparative efficacy, safety, patient reported outcomes, impact on treatment budget and other spending, cost-effectiveness), absolute value (i.e., severity of disease, size of population affected, types of benefit at population and patient level, quality of evidence), as well as clinical guidelines and the limitations of alternative interventions. In this case study, the MCDA estimate was 41% of maximum value, with major drivers including "Improvement of efficacy" (14% of total MCDA estimate), "Disease severity" (11%) and "Quality of evidence" (22%, for Q2 and Q3 combined). Although not directly comparable due to different panel compositions, this MCDA estimate is at the lower range (42% - 64%) of estimates obtained during the proof-of-concept study for 10 medicines approved in Canada (submitted manuscript). Further research is needed to position MCDA estimates within a frame of reference.

The process of considering the quantifiable components of decision lays the groundwork for the third step in which qualitative ethical and healthcare system related factors are being considered. The MCDA estimate is not intended to be used in a formulaic approach [16], but rather as an attempt to tease out all relevant quantifiable components and to then consider the impact of other ethical and contextual elements influencing overall value. In this case study, issues identified included the medical utility of height gain (Utility), provision of psychosocial support as an alternative (Efficiency), policy towards rare diseases with respect to discrimination and priority setting (Fairness, Political context) and potential misuse in toddlers (System capacity/appropriate use). Although these issues had mixed effects on the perceived value of the intervention under scrutiny, panelists noted that the process helps ensure awareness of all historical, political, system-related and ethical elements that may impact the decision. Overall, the framework encouraged a better analysis of the issues that troubled panelists subconsciously, rendering them more explicit in this assessment.

Exploring the validity of the framework revealed that panelists would consider most of its components to assess a healthcare intervention, thus providing some support for the current set of components. Although validation in terms of agreement across panelists was not possible since the tool is designed to capture differences of personal values, exploration of intra-rater agreement between weights and scores obtained during test and retest revealed a fair to good agreement at the individual level (ICC = 0.6). Weights were, to a larger degree than scores, responsible for test-retest disagreement in value estimates, suggesting that panelists were still wrestling with their own values, or interpreted the components of the MCDA VM differently. Alternatively, this may have arisen because the panel was not yet comfortable enough with the process; this latter hypothesis is testable by examining individual test-retest data over time. For weights, largest variations were observed for disease-related components suggesting some difficulty in deciding on the importance of disease severity (D1) and size of population (D2). It may also reflect difficulty in positioning Turner syndrome on a scale encompassing all diseases, since specialists typically focus on their practice and tend to rank diseases and interventions within the range of patients they see [25]. For scores, the largest degree of test-retest disagreement at the individual level was observed for cost-effectiveness-based components, highlighting the difficulties in making sense of CE ratios and frustration associated with the systematic negative effect of discounting on cost/QALY ratios for interventions for children. Although CE ratios were included in the framework to make a connection with the current predominant model, this may not be necessary and separately considering definite expenditures (e.g., drugs costs - E1) and potential savings (e.g., hospital stay - E3) may be more informative and better aligned with the natural thought process underlying decisionmaking.

Discussing amongst a range of stakeholders all the components of decisionmaking on the basis of a structured HTA report that presents the available evidence (in its broad sense, i.e., both scientific and colloquial) for each component of decision was seen by the panelists as a useful means of revealing data gaps and identifying research questions. This can lead to improved study design and hopefully ensure better alignment of data generation with data needs at the micro, meso and macro levels. In this case, the process revealed the need for defining truly relevant outcome measures. From the patient perspective, height increase was seen as useful if it could permit normal functioning by bringing the patient from very small to small, highlighting the importance of patient involvement in sound decisionmaking [46‐48]. Comparative data concerning alternative strategies to improve adaptation of individuals with short stature (e.g., psychological support) was also limited, but seen as important to assess the value of growth hormone, in line with previous studies [49, 50].

The EVIDEM framework is proposed as a step beyond the current cost-effectiveness model, combining efficient HTA, pragmatic MCDA, and an ethical framework to ensure systematic consideration of all components of decisionmaking and evidence available, and to provide a transparent record of how these elements have been used to reach a decision. It constitutes a concrete step towards addressing some of the elements identified as key to successful decisionmaking [42] and for accountability for reasonableness [22]. Systematically considering all pieces of the puzzle for healthcare interventions by all stakeholders leads to difficult questions, helps to position interventions across a wide range of options, including preventive and curative programs as advocated by Nord [3], and may enhance social responsibility and reduce bias.

The framework is primarily designed to stimulate and make more explicit the natural thinking process underlying decisionmaking and deliberation. Although decisionmaking contexts are diverse, we propose that the same comprehensive set of components can be considered in a broad range of circumstances because they can be easily adapted to user needs at both policy and clinical levels. HTA reports organized in the format of this framework to inform components deemed necessary can be used to stimulate reflection and deliberation. MCDA estimates can also be used to establish a ranking system that can encompass a broad range of healthcare interventions and is consistent with user priorities and values. The framework is applicable to decisions on healthcare interventions for which there are many comparators since it provides a pragmatic reporting of comparative effectiveness; this is currently being tested for pain management. The framework is also applicable for research planning, communication and knowledge translation; the last is currently being explored through a web-based open-access collaborative registry structuring data on healthcare interventions (high level synthesis, details and full text sources) using the framework components of decision. Further testing and validation is needed to build up MCDA approaches combined with pragmatic HTA in healthcare decisionmaking.