Which value aspects are relevant for the evaluation of medical devices? Exploring stakeholders’ views through a Web-Delphi process

This study was developed within the scope of the MEDI-VALUE (“Developing HTA tools to consensualise MEDIcal devices’ VALUE through multicriteria decision analysis”) project [34], a national research project that included as consortium partners the Portuguese national HTA agency (INFARMED) and three leading Portuguese hospitals (Centro Hospitalar Lisboa Norte, Hospital do Espírito Santo de Évora and Instituto Português de Oncologia de Lisboa), and aimed at advancing HTA literature by designing and implementing methods, informed by MCDA, that enable the involvement of a large number of health stakeholders and promote consensus in the structuring and development of sound models for assessing the multidimensional value of medical devices.

To inform HTA processes and to inform the building of evaluation models, this study presents a Web-Delphi process designed (see Fig. 1) for gathering different stakeholders’ opinions on the relevance of distinct aspects for medical devices’ evaluation in terms of their added-value to an alternative comparator. Delphi processes allow to anonymously collect individual opinions in successive rounds presenting, from round 2 onwards, a summary of the opinions given in the previous round to participants [35]. This design allows participants to reflect on their previous opinions and change them, or not, based upon new generated information.

In preparation for this process, the aspects potentially relevant for the evaluation of medical devices were collected from studies using MCDA for medical devices evaluation, identified in the systematic review of Oliveira et al. [2], and by extending the search protocol of that study until October of 2019. From this search, a final list of 34 aspects (described in Additional file 1) was organised, with two MEDI-VALUE experts from INFARMED (the Portuguese HTA regulator) analysing the list completeness, eliminating evident redundancies.

The Web-Delphi was implemented in the Welphi platform [36] and was composed of two rounds. In the first round (12 March-02 May 2020), participants were invited to give their opinion about the relevance of each of the 34 aspects for each of the two types of medical devices, by choosing one level of the four-level qualitative relevance scale presented below. This relevance scale allows not only to identify which aspects are relevant for stakeholders, but also to capture the strength of the relevance, either for the panel or for each stakeholder group. Following the rational of “determinants” [37] and “relevancy” [38] analysis (which is context specific [39]), this information is useful for informing the structuring of a multicriteria value framework, as it helps in screening out non-relevant aspects, and for assisting the building of multicriteria evaluation models, as it differentiates the relevance among aspects. The scale was developed, tested and validated with five health experts from MEDI-VALUE partners.

The 34 aspects and the relevance scale were presented to participants in two separate screens: for ‘Implantable medical devices’ and for ‘In vitro tests based on biomarkers’ (a therapeutic and a diagnostic type of device, respectively). For each type of medical device, participants could provide general comments and, for each aspect individually, they could select the ‘Don’t know/ Don’t want to answer’ option or provide specific comments. Participants could give their opinions for one or both types of medical devices. In the second round (9–31 May 2020), participants had access to similar screens, but then they could access their own previous answers, and the distribution of all first-round answers and comments as feedback, being able to change their answer. After the second round was concluded, participants could access the results and leave comments regarding the results and the process (23 June-14 July 2020). A second study was developed simultaneously, exploring the effect of feeding back the distribution of the answers disaggregated per groups of stakeholders on participants’ opinion change. For this, half of the participants were randomly selected to also have access to such information. A manuscript of this second study is being prepared, and more information can be provided upon request.

Participants’ selection followed a purposive sampling strategy targeting five stakeholders’ groups in Portugal: healthcare professionals (doctors, nurses, pharmacists, technicians), buyers and policymakers, academics, industry, and patients and citizens. MEDI-VALUE partners disseminated the study among their networks, explaining the study's aim and the type of participants being recruited. Specifically, selected stakeholders should: (a) have experience in the use, evaluation, selection or acquisition of medical devices, or having an interest in the topic; (b) be available to participate given the timeline planned; (c) not have any conflict of interest preventing their impartial participation. Through this process, a total of 365 stakeholders were identified and invited to participate. As each participant can have several roles in practice and, thus, could belong to more than one group, participants selected, at the beginning of the Web-Delphi, the stakeholder group in the light of which they would give their opinions.

Analyses were performed in Microsoft Excel and R software to answer three specific research questions: to explore the relevance of each aspect for the evaluation of the two types of medical devices, for the panel and per stakeholders’ groups; to measure the level of agreement within groups; and to explore whether there were aspects with the same relevance for the two types of medical devices.

Table 2 presents the distribution of answers, per type of medical device, of the panel and disaggregated per group of stakeholders. Moreover, it presents the panel majority opinion and the analysis on whether the groups are aligned with the panel, described as the groups majority opinions.

For ‘Implantable medical devices’ six aspects gathered panel majority on ‘Critical’, 16 on ‘Fundamental’, and one on ‘Complementary’. Regarding ‘In vitro tests based on biomarkers’, seven aspects gathered panel majority on ‘Critical’, 13 on ‘Fundamental’ and one on ‘Complementary’. For both types of medical devices, the remaining aspects had no panel majority opinion. These results suggest that the panel considers all 34 aspects as relevant to evaluate the added value of a medical device for any of these types of devices – as there was no aspects gathering majority on ‘Irrelevant’ – and, specifically, that the panel agrees in the relevance of a high number of aspects – 23 and 21 aspects for ‘implantable’ and ‘in vitro’ devices, respectively.

When analysing the answers per groups of stakeholders, we observe some differences. For ‘Implantable medical devices’, for four of the six above-mentioned ‘Critical’ aspects and three of the 11 ‘Fundamental’ aspects, the majority occurred across all stakeholders’ groups, with the remaining aspects presenting one to three groups of stakeholders without the same majority opinion. Identically, for ‘In vitro tests based on biomarkers’, only two of the five ‘Critical’ aspects and two of the 11 ‘Fundamental’ aspects presented the same majority across all groups. Regarding the aspects with no panel majority opinion, most present a majority on one to three groups of stakeholders, with only two aspects (‘Patient-reported outcomes’ and ‘Environmental impact of the production and use of the medical device’) of ‘in vitro’ devices not gathering majority in any group.

In all cases, for the aspects with no groups majority opinions, the answers of the groups tend to be around the same two relevance levels (these two levels are highlighted in light grey in Table 2). Exceptions to this are, for example, ‘Exposure of the healthcare professional to physical or chemical agents’ that presents groups of stakeholders with most answers around ‘Critical’ and ‘Fundamental’ and other groups with most answers around ‘Fundamental’ and ‘Complementary’ (this happening for both types of medical devices, with a panel majority opinion defined only in ‘implantable’ devices).

For ‘Implantable medical devices’ a statistically significant difference (statistical tests results available in Additional file 2) was found between Industry and Patients and citizens in ‘Sensitivity and Specificity’ (p = 0.0449), for which Industry mostly considered ‘Critical’ and Patients and citizens had a dispersed opinion with no majority, and ‘Impact of the disease’ (p = 0.0282), for which Industry group’s opinion gathered no majority and Patients and citizens gathered majority in ‘Fundamental’; between Healthcare professionals and Patients and citizens in ‘Exposure of the healthcare professional to physical or chemical agents’ (p = 0.0405), for which Healthcare professionals gathered no majority and Patients and citizens gathered majority in ‘Fundamental’; between Academics and Patients and citizens in ‘Target population’ (p = 0.0447) and ‘Cost of the medical device (including complementary equipment)’ (p = 0.0266), in which both groups gathered majority on ‘Fundamental’ but Academics gathered opinion among ‘Critical’ and ‘Fundamental’, and Patients and citizens among ‘Fundamental’ and ‘Complementary’; and between Academics and Healthcare professionals in ‘Cost of procedure without the cost of the medical device’ (p = 0.0326), with both groups gathering majority in ‘Fundamental’ but Academics gathering opinion among ‘Critical’ and ‘Fundamental’ and Healthcare professionals among ‘Fundamental’ and ‘Complementary’. For ‘In vitro tests based on biomarkers’, a statistically significant difference was found between Academics and Patients and citizens in ‘Target population’ (p = 0.0370), with both groups gathering majority in ‘Fundamental’ but Academics gathering opinion among ‘Critical’ and ‘Fundamental’ and Patients and citizens among ‘Fundamental’ and ‘Complementary’, and between Academics and Healthcare professionals in ‘Cost of the medical device (including complementary equipment)’ (p = 0.00718), with Academics’ answers gathering no majority and Healthcare professionals gathering majority in ‘Fundamental’. For the remaining aspects, the differences across groups were not statistically significant.

All groups presented a higher Gwet’s AC2 coefficient for ‘implantable’ than for ‘in vitro’ devices (see Additional file 3). Irrespectively of the type of medical device, the agreement increased in round 2, suggesting answers’ convergence. According to both benchmarking methods, Landis and Koch [44] and Gwet’s proposed benchmarking method [41], the calculated Gwet’s coefficients describe a strength of agreement from ‘moderate’ to ‘substantial’ among all stakeholders’ groups. The agreement level assessed first, following Landis and Koch, and after, using Gwet’s proposed benchmarking method, comprised the following differences: for ‘Implantable medical devices’, the agreement changed from ‘substantial’ to ‘moderate’ in the Buyers and policymakers group (in round 1) and for ‘In vitro tests based on biomarkers’, the agreement changed from ‘substantial’ to ‘moderate’ in the Buyers and policymakers group (in round2). Both benchmarking methods resulted in a ‘moderate’ agreement in the Industry group (in both rounds for ‘in vitro’ devices) and in the Buyers and policymakers group (in round 1, for ‘in vitro’ devices). A ‘substantial’ agreement was observed in all the remaining cases.

Similar results were retrieved for both types of medical devices: five aspects were ‘Critical’ and 11 ‘Fundamental’, according to the panel – in three ‘Critical’ and in one ‘Fundamental’, the majority was observed in all stakeholders’ groups. In the remaining 18 aspects, seven aspects did not gather majority in any type of device, 10 gathered majority for only one type and 1 gathered majority on a different relevance level across both types of devices. These results, previously detailed in Table 2, are synthesised in Table 3 together with the common relevance level.

The Delphi panel was composed of distinct stakeholders’ groups. Analysing the distribution of answers of the panel and per groups, results show one of four situations: (1) there is a panel majority opinion with all groups presenting majority on the same relevance level, (2) there is a panel majority opinion but not all groups present majority on the same relevance level, (3) there is no panel majority opinion but some groups present majority on a relevance level, and (4) there is no panel majority opinion nor majority in any group. Considering the meaning of the relevance levels (presented in Overview of the Web‑Delphi process, in the Methods section), these results suggest that participants consider there are aspects that must always be part of the basis of evaluation: that is the case of aspects assigned ‘Fundamental’ or ‘Critical’ by the panel, for instance, ‘User-friendliness for the healthcare professional’ for ‘implantable’ devices and ‘Time between procedure and results’ for ‘in vitro’ devices. Furthermore, participants consider some of these aspects can even preclude the evaluation if there is no data for assessing them – this applies to aspects set as ‘Critical’ –, for instance, ‘Specific features of the medical device’ for ‘implantable’ devices and ‘Sensitivity and Specificity’ for ‘in vitro’ devices. Additionally, participants’ answers suggest that there are ‘Complementary’ aspects, i.e., aspects that can add some value but will not always be part of the basis of evaluation, for instance ‘Environmental impact of the production and use of the medical device’ and ‘Learning curve of the healthcare professional’ for ‘implantable’ and ‘in vitro’ devices, respectively. This can be seen in Table 3 that presents the panel majority opinion and systematizes the groups majorities aligned with the panel. In general, stakeholders’ groups did not present obvious contradictory opinions, as even the aspects with no panel majority opinion gathered most groups answers around the same two consecutive relevance levels (as presented in Table 2). The Kruskal–Wallis test followed by Dunn’s-Bonferroni post hoc method allowed to identify only eight aspects (six in ‘implantable’ and two in ‘in vitro’ devices) with statistically significant differences across groups, but these differences were always observed across only one pair of groups, and the inter-rater reliability calculated with Gwet’s AC2 agreement coefficient showed a strength of agreement from moderate to substantial within each group, suggesting an alignment of the panel and within groups. Despite the general alignment of the groups, the reasons underlying the observed differences may be of interest for further research [46].

This Web-Delphi process collected opinions not only from different stakeholders’ groups but also for two types of devices, a therapeutic and a diagnostic type of device. Comparing the opinions across both types of devices, results show that the panel attributed a common relevance level for 16 aspects, five ‘Critical’ (two agreed by all stakeholders’ groups) and 11 ‘Fundamental’ (one agreed by all groups) (see Table 3). Examples of this are the ‘Clinical efficacy and/or effectiveness’ considered ‘Critical’ (majority in all groups), or the ‘Comfort for the patient’ considered ‘Fundamental’ (not getting majority in all groups but gathering a panel majority opinion). The former aspect is aligned with economic evaluation literature [6] centred into the effectiveness and costs of technologies whereas the latter is not explicitly considered in such methods. Moreover, many other aspects were recognised as relevant by the participants of our study, suggesting the need to formally consider a larger number of aspects in the evaluation of medical devices. This need has been recognised in literature [3, 47], by authors advocating for the use of MCDA in HTA [2], such as the ISPOR (The Professional Society for Health Economics and Outcomes Research) Medical Devices and Diagnostics Special Interest Group [48], by authors developing value framework models for evaluating medical devices, such as in the HTA Core Model from EUnetHTA (European network for Health Technology Assessment) [49], and also by the review on value assessment frameworks of Zhang et al. [22] that covered 19 studies addressing health technologies in general and 38 addressing specific types of health technologies (mainly drugs). Four of the frameworks reported in that review targeted diagnostic or genetic tests and one targeted nondrug health technologies, with evaluation aspects included varying between three and 16 and covering different devices’ features, namely, their medical benefit, the adverse effects, the quality of life and satisfaction of the patient, and the costs. Our study, besides validating this need with a large and diverse panel of stakeholders, adds additional value aspects not identified in the existing frameworks’ literature, e.g., regarding environmental impact and aspects related with devices’ usage by the healthcare professional, such as user-friendliness, the learning curve, the training and the workload.

The list of aspects included in our study tries to be purposefully inclusive, which brings the advantage of being as complete as possible but the disadvantage of entailing potential overlap in some aspects. To evolve towards the construction of a multidimensional framework or of multicriteria models, the identified aspects would require further work and restructuring, eventually combining and clarifying aspects and exploring how to measure them in practice [32] (for e.g., understanding what participants have in mind when considering the sensitivity and specificity of an implantable medical device as relevant). Nonetheless, our work provides important insights to inform such a framework development. In 44 frameworks reviewed by Zhang et al. [22], value aspects were identified through literature review, engagement of stakeholders, or a combination of both, but only four frameworks involved patients or citizens in aspects’ identification. Our study explored a way to collect the wide range and diversity of stakeholders’ perspectives, including patients and citizens, adding to the discussion on how to bring these insights into HTA for standardising and bringing guidance and transparency to the evaluation of medical devices [19, 28, 50] and how to include stakeholders’ views to inform HTA and adoption decisions [23, 26].

Methodologically, through the Web-Delphi it was thus possible, first, to involve a large and heterogeneous group of HTA stakeholders, enabling them to interact and learn with each other by sharing their views and build an agreement about the relevance of most aspects. Second, to draw conclusions about differences in opinion between stakeholders and across types of devices. Third, it has shown in which aspects there is a panel majority opinion. All of this provides input information for additional research on how to develop multidimensional evaluation models and frameworks, and assists in planning future directions of research.

This study shows that it is possible to gather the views of distinct stakeholders’ groups in a structured format, producing results that can be more widely used within HTA processes, as deemed as relevant by several authors [24‐27, 51]. All aspects were considered to some extent relevant, and some aspects gathered the same relevance level irrespectively of the type of medical device under analysis. Accordingly, approaches to assess medical devices value need to consider a broader range of aspects and the specificities of distinct types of devices. Despite the heterogeneity of this type of technologies [12], there seems to be possible to attain some systematization and common standards, so asked in literature [4, 9, 20, 52]. Nevertheless, one should recall that the evaluation may be affected by the context [53], and that ‘Implantable medical devices’ and ‘In vitro tests based on biomarkers’ still comprise diverse devices, which needs to be considered when interpreting results.

Several limitations should be acknowledged in this study. Firstly, this study takes place in Portugal, having only national participants, and thus results can be context- and/or country-dependent. Nevertheless, the list of aspects was based on international peer-reviewed literature which brings useful information to inform the discussion on HTA for medical devices, beyond the considered country and context. Secondly, as the Delphi process is highly dependent on the availability and commitment of participants [54], there was not a balance of participants across stakeholder groups, with the Buyers and policymakers and the Industry having a lower representation. This unbalance is somehow usual in Delphi processes as panels are purposive or convenience samples, not aiming to be representative samples of populations [54]. Furthermore, Delphi literature does not present unequivocal recommendations for the sample size, with studies suggesting ranges from five to more than one thousand participants [54, 55]. To try to mitigate the unbalance as much as possible, invitations and reminders for participation were sent. Thirdly, and still related with the Delphi process, it is important to acknowledge shortcomings of the method, namely the possibility of occurring cognitive biases and other behavioural influences during the process (such as egocentric discounting or the influence of majority positions) due to the freely online interactions among participants, which can also lead to answers not completely clarified by participants [56], or the possibility of information overload due to the high number of aspects to be analysed by the participants, which could become tiresome and cognitively challenging for them [57]. To avoid the occurrence of such shortcomings, not only the panel of participants was heterogeneous but also the aspects were organised, during the validation with experts, so that it would be easier, to the best of their knowledge, to follow the exercise. Additionally, participants could also rate each type of medical device in different time periods by re-accessing the platform, or even answering only one type of medical device if they felt more comfortable. Finally, the initial list of aspects could be biased as it was the result of a literature search followed by the validation by experts of the HTA agency. To overcome this possibility, participants could suggest additional aspects during the process, through the comments option, which was not observed.