Balancing costs and benefits at different stages of medical innovation: a systematic review of Multi-criteria decision analysis (MCDA)

There seems to be remarkable lack of consistency in methodology employed by the various studies that undertook the MCDA. There was no reasonable process to make clear why different authors decided to use a certain methodology. The selection of criteria was more systematic but still very different across studies. This resulted in different numbers and sort of criteria in the examined studies. Additionally, the scaling of performance scores raises questions about the methodological validity of these scales.

In contrast to these shortcomings, the stakeholder feedback on MCDA highlights the potential value of the methodology for decision-making. Therefore, we developed some recommendations for the technical implementation of MCDA in decision-making. Determinants of quality were identified regarding the technical aspects of decision criteria, the scaling of criteria to measure performance, the weighting method and the underlying evidence.

MCDA can bridge criteria and views that are challenging to compare, as shown in the specific bioethical context [49]. Choosing and clustering the right criteria requires active and direct involvement of stakeholders to avoid irrelevant or overlapping criteria [45]. The decision criteria should be technically robust regarding potential overlaps [37]. Every criterion should focus on a single aspect of the decision to avoid double-counting [50]. Double counting means that the same effect impacts more than one criterion in the performance matrix. For example, even though cost-effectiveness was one of six decision criteria in a study, the criterion “age of target group” included economic considerations regarding the working part of society [27]. The implicit consideration of economics in different criteria can bias the results. For health policy decision-making, another important criterion for double counting is the severity of disease. This criterion can impact several other criteria e.g. economics, effectiveness, ethics. Therefore, the explicit structuring of criteria is important. Checking for independent preferences can support the detection of double-counting effects. A deliberative process can increase the legitimacy of MCDA criteria selection [42, 51].

Few studies have suggested the use of separate criteria describing costs and medical effects instead of the mixed criterion cost-effectiveness [36, 37]. This makes decision criteria more transparent and understandable e.g. absolute costs of an intervention as an easy understandable measure. Still, cost-effectiveness is the most important economic criterion in the examined studies which shows a lack of awareness regarding double counting. Several studies used more than one economic criterion indicating that the absolute costs of an intervention should be compared to the impact on the available budget.

The inclusion of economic aspects in decision-making by MCDA can be very different. Most studies compared economic criteria with other criteria in one step. In contrast, some studies considered economic criteria like budget impact in a second step. This procedure can feed a deliberative process [42, 45] or a mathematical model to calculate the reimbursement threshold [35, 41].

In early innovation studies, economic parameters were simple due to the absence of health economic evidence. For example, economics for the prospective usage of photoacoustic imaging for breast cancer diagnosis was assessed by the criteria “price”, “manpower”, “scan time” and “peripheral equipment” [30]. In contrast, a study about new surgery in adolescent idiopathic scoliosis included cost parameters into a decision tree model to estimate total costs [31]. This raises the question whether health economic analysis should be performed in early innovation by taking uncertainty into account or to assess only those simple parameters.

A well-defined scale for every decision criterion should represent the performance of the assessed alternatives. Doing so, judgments on well-defined scales are proportional and consistent with the increase in real world performance. Constructing different levels of performance for decision parameter is not a trivial task [35]. The discriminatory power of MCDA can be decreased if the scaling system and thresholds are not inadequately constructed [45]. Most of the assessed studies used rating scales with five or less categories. However, psychological research showed that scales with more categories (i.e. 10, 101) support users in expressing their feelings. In contrast, rating scales with only a few categories support quick ratings. For most situations, rating scales with seven, nine or ten categories are most appropriate taking all pro and cons into account [52]. Scaling of criteria should take reliability, discriminatory power and preferences of participants into account. A sensitivity analysis can help to determine the uncertainty from scaling effects e.g. whether more scaling levels contribute to a more exact result [35, 41]. This can be used to iteratively improve the MCDA model.

According to the value function approach, direct rating of the decision criteria by participants can improve the awareness in decision-making [34, 36, 37, 39, 40]. This approach reflects the need for explicit statements about the importance of criteria. Doing so, the importance of criteria needs definitions among different dimensions [53]. Firstly, direct weighting provides a relationship between the importance of two criteria (e.g. value of criterion A = 2 vs. value of criterion B = 4). Secondly, trade-off weights between criteria need to be compared taking the ranges on the criteria scales into account. This is covered by the scoring exercise which is required of any MCDA approach that was assessed in this review. Thirdly, the starting point for assessing value trade-offs needs consideration, in particular if preferences are not linear. A linear relationship between performance measure and scale is not always appropriate even if measures are constructed in a linear way [7]. To cover all three dimensions of importance, MCDA methods like PROMETHEE (Preference Ranking Organisation Method for Enrichment Evaluation) enables the clear definition of preference functions [54].

The selection of the MCDA method can affect the uncertainty of decision-making. Different MCDA approaches vary in structuring the decision problem (i.e. criteria selection, weighting, scoring, and calculating). This variety causes structural uncertainty [55]. Each MCDA method has particular advantages for specific phases of the innovation process [29, 34]. For example, methods like DCE are very efficient at the beginning of technology development as they show the conscious and unconscious preferences of stakeholders. If evidence around technologies is generated, direct methods can be used for decision problems about reimbursement [39]. Uncertainty around evidence can lead to wrong decisions such as reimbursement of an ineffective therapy. An advantage of MCDA is that a structured decision framework can make this uncertainty more visible within the structure. The evidence and comments of the stakeholders, involved in the studies, suggest that direct rating methods can better handle this uncertainty due to a better understanding of the methodologies [29, 35, 42]. Additionally, preference elicitation techniques cannot support larger numbers of criteria and scaling levels [25, 28, 35, 42, 44] which again increases the uncertainty within the MCDA structure. The combination of different MCDA methods for different parts of the decision, like criteria weights from DCE and evidence scores from AHP can improve the decision analysis [25]. Combining different approaches requires assessing the transferability of the stepwise results. Handling of MCDA approaches may vary across stakeholder groups [29, 37]. For instance, applying MCDA can be challenging for laypeople and patients [42, 45]. The evidence suggests that the cognitive burden of direct weighting methods is lower in comparison to AHP and DCE [29, 35, 42]. Pragmatic MCDA means to provide simple and flexible approaches which are easy to understand by users with little experience. Doing so, the needs of decision-makers should be carefully balanced with the resource requirements as well as the theoretical requirements of certain MCDA methodologies. Feasibility and flexibility are important aspects to ensure acceptability of these approaches [36, 37, 39, 40].

Awareness about the selection of decision criteria, their importance [56] and the decision perspective are critical parts of MCDA from an ethical viewpoint. Stakeholders have to reflect on their own priorities and rationale to elucidate their decision-making. Patient involvement was limited and reported in only 14 % (3 studies), although they represent the ultimate decision makers on final acceptance of a healthcare intervention. Inhalable insulin is an example for innovation failure because of missing user acceptance in the final phase of the innovation process [57]. A consistent mix of stakeholders across all stages of innovation can ensure that all important needs are addressed. The separation of weighting and scoring by different stakeholders can result in disagreement [34]. Still, combining many stakeholder perspectives helps to identify and possibly resolve differences across perspectives. An essential aspect is to identify the “right” representativeness of a certain stakeholder group [45] as well as the involvement of the “right” stakeholder group. Indeed, the question, which perspective (their own or another e.g. society) participants should take into account, is not trivial [37]. Daniel’s ethical framework of “accountability for reasonableness “ is a widely-agreed rationale regarding fair decision-making [58]. According to this framework, reimbursement decisions should be based on reasons and criteria which reflect society’s values [59]. However, the consensus of experts does not guarantee that choices are representative of a societal viewpoint [38, 44]. MCDA approaches can help make these issues more explicit and thus provide some methodological basis to feed the ongoing debate in society about reimbursement decisions.

Incomplete evidence for the assessed alternatives increases uncertainty around decisions. Pragmatic MCDA bridging HTA methodology with MCDA principles can help clarify these issues in a systematic manner [36, 37, 39, 40]. Still, missing data and a time-consuming workload make existing approaches insufficient [45]. In contrast to mainstream HTA, the evidence in the development of a technology is even more based on expert opinion or adaption of evidence from other technologies [29‐31, 34]. Most reviewed studies related to early innovation lack economic considerations because of missing evidence. Finally, missing evidence can put new technologies at disadvantage in prioritization decisions [44]. Expert judgments could replace clinical evidence but the selection of experts can affect the results [31]. Evidence can also be obtained by early health economic modeling [31, 44]. There is the potential for synergies between evidence generation and MCDA. All MCDA require the involvement of stakeholder groups to some extent. Different stakeholder groups can participate in the MCDA decision process and thus improve the communication of their preferences and perspectives in a systematic way [18, 60]. Values and preferences of stakeholders are essential to integrate different decision criteria and to come up with a decision. For example, AHP supports consensus finding in groups by calculating an inconsistency ratio after the scoring exercise. This can be the basis for a deliberative discussion to refine ratings. In terms of pragmatic MCDA, stakeholder can contribute to the selection of decision criteria and the selection of MCDA techniques to evaluate these criteria. Simultaneously, experts and stakeholders can support the evidence search and synthesis, the validation of health economic models, as well as the MCDA process. Such combination can contributes to a transparent analytic process and results in a more comprehensive understanding of the technology in every phase of innovation. MCDA can facilitate an early dialogue between manufacturers, regulators and HTA agencies about evidence needs. This collaboration can help patients by faster access to treatments as well as manufacturers for more efficient development of new technologies.

This study should be considered in light of its limitations. We only included studies that used economic criteria, a component that was missing in a significant part of the literature. As there is no clear definition of MCDA, we only included studies which mentioned MCDA or a certain described MCDA method. However, other studies may use MCDA methods without explicitly stating it. The results reported in the indexed literature could be systematically different from those presented in the grey literature such as government reports and white papers [61]. Further research should focus on how MCDA can contribute to an efficient innovation process as well as the dynamics of changing conditions regarding new evidence and its impact on decision-making. Development of guidelines on the choice of a particular approach to use at specific phases of the innovation process would provide value for further development in the field. The comparability and the usefulness of the final MCDA outputs also requires some further research. Baltussen suggest using the final result of an MCDA as recommendation, not as formal list of priorities (60). Another line of research is identifying the implementation and administrative workloads of MCDA in comparison to existing approaches, and the overall benefit in decision-making.