Introduction

The introduction of new pharmaceuticals and technologies has caused healthcare costs to steadily rise over recent decades in Europe [1, 2]. This threatens the sustainability of healthcare systems, and forces policymakers and financial stakeholders to make decisions on how to allocate scarce resources. Economic evaluations in which costs and effects of two or more healthcare interventions or innovations are compared can support decision makers in such allocation decisions [3]. Some countries have established cost-effectiveness as a decision criterion to reimburse healthcare interventions, especially for the reimbursement of new pharmaceuticals following their market approval [4].

To ensure the comparability and quality of economic evaluations, several European national agencies developed methodological guidelines on the principles and methods for the design, execution and reporting of economic evaluations over recent decades. In many countries, for example in Belgium, Germany, Norway, Portugal, Poland, the Netherlands, the Slovakia, and Slovenia [4,5,6,7,8,9,10], it is mandatory to prepare an economic evaluation in accordance with the national guidelines when applications are being made for reimbursement of a new healthcare technology.

In recent years, the EU stimulated the development of multidisciplinary partnerships among government agencies, research institutions and health ministries across European countries by funding several health technology assessment (HTA) projects at the European level [11,12,13]. Also, the number of international economic evaluations is growing, for example in situations where health interventions emerge at more or less the same time across different countries, or when it is not feasible to perform an economic evaluation with sufficient power in one country. An example is the European Schizophrenia Outpatient Health Outcomes (SOHO) study, in which the cost utility of treating schizophrenic patients with different types of antipsychotics was determined with use of data from ten countries [14]. Although effectiveness data from cross-country studies are often easily transferable to other settings, costing data are much more context specific. Despite the existence of many national guidelines, there is still little guidance on appropriate costing methods to use when health economic evaluations are conducted in international contexts, which hinders the comparability and transferability of results. Practical difficulties encountered when a cross-country study is performed include variation in the inclusion or exclusion of cost categories, in the classification of cost categories, in the choice of discount rate, and in the valuation of costs [4, 15]. Incomparability of international economic evaluations may result in unnecessary work and expenses, because researchers replicate economic evaluations to resemble their own specific context. Thus, to increase the comparability and transferability of economic evaluations in Europe, it is desirable to have a common set of detailed guidelines for the design and conduct of economic evaluations. The availability of such a set of guidelines will strengthen cross-border HTA collaborations such as those already existing within the European Network for Health Technology Assessment (EUnetHTA), but will also be useful for countries without country-specific guidelines for economic evaluations. Eliciting experts’ opinions on guidelines for economic evaluations will constitute an important step towards developing a common European view on conducting health economic evaluations. Therefore, the aim of this study was to develop cross-European recommendations for the identification, measurement and valuation of resource use and lost productivity for use in cross-European economic evaluations from a societal perspective, with use of a Delphi procedure among European health economic experts. A Delphi procedure was chosen because it is a structured approach to make group-based decisions on topics where strong differences in opinion exist. This method is commonly used to develop costing guidelines and reporting checklists for costing studies [16,17,18,19,20].

Methods

This study is part of the European Identifying Best Practices for Care-Dependent Elderly by Benchmarking Costs and Outcomes of Community Care (IBenC) project. IBenC’s primary aim is to identify best practices of community care delivery systems across Europe by comparing their costs and quality of care outcomes while also developing methods to support the accomplishment of this aim [21].

Study design

A Delphi study with three consecutive, blinded rounds was conducted between December 2014 and January 2015 among European health economic experts. The Delphi study was conducted online to ensure anonymity of the panellists. A steering committee was appointed to force consensus on items that were not agreed on by the panel after the final Delphi round [22,17,24].

Delphi panel

A group of 110 international experts working in the field of health economics at government agencies, universities, research institutes and pharmaceutical agencies from 27 European countries was informed and invited by e-mail to participate in the Delphi study. Experts were selected on the basis of HTA-related publications in peer-reviewed journals, their participation in the development of national guidelines, their participation in other European HTA projects or their involvement with the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) or EUnetHTA. Experts who were unable to participate in a Delphi round, but expressed their interest, were invited again for the subsequent round.

Steering committee

An independent steering committee consisting of four professionals (WH, AW, CD and HN) in the field of health economics decided on items for which no consensus was reached after the third Delphi round. None of them were involved in selecting potential panellists, designing the questionnaires and analysing the results.

Identification and review of existing guidelines

The questionnaire used in the first round of the Delphi study was based on a review of existing European guidelines for designing and conducting economic evaluations or costing studies as part of economic evaluations. To identify existing guidelines, the website of the ISPOR was searched [25]. ISPOR provides an overview of national pharmaceutical guidelines for economic evaluations. This overview is updated regularly on the basis of contacts with professionals in more than 50 countries to ensure its quality and accuracy. For countries for which no guideline was available on the ISPOR website, additional searches though the Internet were performed. Publicly available, English-language national guidelines containing recommendations on the execution of economic evaluations available issued by European government agencies were included. No exclusion criteria were applied, and the publication date was not restricted. Special effort was made to identify the most recent versions of the identified guidelines; webpages of HTA agencies were additionally searched, and authors of guidelines issued before 2003 were contacted for possible updates. A recent update of the Hungarian guideline was not available in English. Therefore, we consulted a health economist from Hungary to identify the most important differences between the two versions of the guideline. Every guideline was reviewed in detail, and information was extracted by two of the authors (LL and JB) independently. Discrepancies were discussed until consensus was reached. A standardized table to synthesize the recommendations was prepared a priori containing relevant issues: perspective; identification of resource use; measurement of resource use; valuation of resource use; discounting of future costs and discount rate used; incremental analysis of costs; sensitivity analysis; modelling; availability of a list with national standard unit costs [3, 26].

Delphi study

The recommendations extracted from the identified guidelines were used to formulate questions for the first Delphi round. In this Delphi study, the starting point was the societal perspective for identifying relevant costs in an economic evaluation, because this is the most comprehensive perspective. Other commonly used perspectives, such as the healthcare and government perspectives, can be derived from this perspective. For each item, panellists were asked to indicate the most appropriate option, or to choose “no expertise” if they felt they had insufficient expertise on a specific topic. To capture recent methodological developments that were not included in the identified guidelines, alternative response options for each question could be provided by panellists. In all rounds, panellists were asked to justify their answers to every question. Together with the questions from the second and third rounds, panellists received a feedback report with the individual and group results from the previous round.

Consensus among the panel was defined a priori as an agreement of 67% or higher to include or exclude a specific item (i.e. a particular perspective, cost category, valuation or assessment method, discounting rate or study design) rated with a dichotomous response option. Agreement of 67% or higher is commonly used in Delphi studies to indicate consensus [27]. Agreement among the panel was calculated for each item separately in every Delphi round.

In the first round, we asked the Delphi panel to indicate for each of the listed perspectives, cost categories, and resource use items whether they should be included in an economic evaluation conducted from a societal perspective. Additionally, the panel was asked to indicate appropriate methods for measuring and valuing resource use and lost productivity, and whether value added taxes (VAT) should be included for all cost categories.

Questions in the second round were developed on the basis of the analysis of the previous responses. Items for which no consensus was reached in the first Delphi round were included once more, accompanied by arguments for and against inclusion reported in the first Delphi round. Alternative perspectives, new resource use items and alternative valuation methods suggested by the panellists in the first Delphi round were put forward for consideration as well. In addition, panellists were asked to rank the listed methods for assessing and valuing resource use and lost productivity with regard to their relevance. Finally, the panellists were asked to indicate which of the listed discounting rates and study design they found appropriate.

To construct the questionnaire for the third Delphi round, rankings on relevance given by the panellists in the second round were converted into relevance scores (a higher score indicates a more relevant method). First, for each respondent, the least relevant method was awarded one point, and the next relevant method was awarded one point more than the previous method. An exception was made for the two methods that were considered most relevant by the panellist. To discriminate better between methods ranked first and second and the other methods, the method placed in the second position received a relevance score twice that of the method in the third position. The method placed in the first position received twice the relevance score of the method in the second position. Subsequently, mean relevance scores for every method were calculated by summation of relevance scores and their division by the number of panellists. The final rankings were based on these mean relevance scores.

On the basis of the relevance scores, the two methods considered most relevant per topic were presented in the third Delphi round. Panellists were asked to choose the method they found most suitable to use in a European economic evaluation from a societal perspective. Also, items for which no consensus was reached in the second Delphi round were addressed again.

Finally, when no consensus was reached on an item, the steering committee was requested to make a final decision. By e-mail, the steering committee was provided with the arguments for and against a specific recommendation given by the panellists so that the members of the steering committee could weight these considerations to reach a final decision. Individual opinions from the members of the steering committee were gathered and used to make a final decision.

Results

Literature review

Eighteen national guidelines were included in the study. These guidelines were published in Austria, the Baltic states (Latvia, Estonia, Lithuania), Belgium, Croatia, Denmark, Finland, France, Germany, Hungary, Ireland, Italy, the Netherlands, Norway, Poland, Portugal, Spain, Sweden and the UK [5,6,7,8,9,10, 28,22,23,24,25,26,27,28,29,30,31,32,40]. For other European countries, no national guideline in English could be identified.

Table 1 provides a structured summary of the main recommendations from the identified guidelines. In short, six of the 18 identified guidelines recommend a societal perspective in the economic evaluation (the guideline from Portugal recommends in addition the use of a third-party payer perspective), six recommend a healthcare perspective and another six recommend both perspectives. Most guidelines stated clearly which costs should be included in a health economic evaluation. All guidelines stressed that all relevant direct healthcare costs should be included for which differences are expected between treatments. The inclusion of social care costs such as those resulting from the use of respite care and supportive care services was recommended in ten guidelines, the inclusion of patient and family costs, including patient out-of-pocket expenses, time costs, informal care costs and travel costs, was recommended in 12 guidelines and the inclusion of lost productivity costs was recommended in 11 guidelines. Five of the 18 guidelines did not describe which sources should preferably be used for the measurement of resource use/costs. There was large variation between the guidelines with regard to the valuation of the resources used. Seven of the 18 guidelines stressed that the valuation method chosen should reflect the opportunity costs (i.e. the value of the forgone benefits because the resource is not available for its best alternative use). In the other guidelines, the underlying principle for valuation was not described. Various valuation methods were recommended in the guidelines, including the use of standard unit costs, tariffs, lowest price, diagnosis-related groups and macro costing.

Table 1 Overview of country-specific recommendations regarding perspective, identification of costs, measurement of costs, valuation of costs, discounting, type of economic evaluation, and availability of a list with country-specific unit costs
Full size table

Delphi panel

Of the 110 invited experts, 26 (24%) participated in one or more rounds, six agreed to participate but did not participate (5%), 48 (44%) did not respond and the remainder (30, 27%) did not wish to participate mainly because of time constraints. Of the 26 invitees who participated, 11 (42%) participated in all three rounds, three (12%) participated in two rounds, and 12 (46%) participated in one round. Each round was completed by at least 16 experts. Background information on the panellists is presented in Table 2.

Table 2 Characteristics of the Delphi panel: country of residence, primary employment and mean number of years of experience in health technology assessment (HTA)
Full size table

Delphi results

Table 3 presents a structured summary of the Delphi results. It includes the agreement (%) per item among the panel in the three Delphi rounds and among the steering committee, the final ranking of the methods for measuring and valuing resource use and lost productivity, and the choice for most suitable method.

Table 3 Results of the Delphi study: perspective, inclusion of cost categories and value added tax (VAT), measurement and valuation of resource use and lost productivity, discounting, type of economic evaluation, and study design
Full size table

The panel reached consensus on 58 of 65 items (89%). The steering committee decided on the seven items without consensus after three rounds. These items comprised the inclusion of four cost items, two measurement methods and one valuation method.

Recommendations

Table 4 gives an overview of the consensus-based recommendations that were developed on the basis of the results of the Delphi study. The results and supporting comments from the panellists are addressed below.

Table 4 Summary of consensus-based recommendations
Full size table

Perspective

The societal perspective is recommended for economic evaluations in a cross-European context (88% agreement). The societal perspective helps to identify cost shifting between sectors. According to the panel, it is likely that relevant costs are missed when a narrower perspective is used because often sectors other than healthcare may incur costs or costs savings as a result of the intervention.

Identification of resource use and lost productivity

It is recommended to take a broad perspective with regard to costs, and to include all cost categories for which differences are expected between treatments. The relevant cost categories according to the Delphi panel are healthcare service costs, intervention costs, patient and family costs, lost productivity costs and future costs.

Healthcare services are services directly related to the prevention, diagnosis, treatment, rehabilitation and nursing care for a particular disorder. Assistance with (instrumental) activities of daily living also falls into this category. The panellists did not reach consensus on the inclusion of costs of complementary therapies (round 1, 50% agreement; round 2, 63% agreement). The arguments provided by the panellists in favour of inclusion were that all related resource use should be included, and that costs of complementary therapies may be significant. A counterargument was that complementary therapists do not represent evidence-based medicine. The steering committee recommended that these costs should be included, since these costs represent resources that were used. Depending on the country’s funding system, complementary therapies may be reimbursed by health insurance and are then considered part of healthcare costs. If the costs are not reimbursed, they have to be categorized as out-of-pocket costs. An additional category that emerged from the Delphi study was the costs of e-health interventions (round 2, 71% agreement).

Intervention costs include all costs related to the implementation of a particular intervention, and should cover both personnel costs [time needed for administration (75% agreement), planning (69% agreement), implementation (67% agreement), and supervision and monitoring (80% agreement)], and the costs of materials needed for implementation of the intervention, including costs of donated items (such as drugs, vaccines, supplies or equipment) (71% agreement). On the inclusion of development and training costs of the intervention, no consensus was reached (round 1, 50% and 60% agreement, respectively; round 2, 40% and 60% agreement, respectively). The panellists’ arguments in favour of inclusion were that all production costs should be included, ad that training costs should be included if training is a prerequisite for the intervention to be effective. However, some argued against inclusion, because a ‘steady state’ should be assumed; the cost of development is usually covered by the price of the intervention. The steering committee recommended that a ‘steady state’ of the intervention should be assumed, implying that costs are estimated for routine implementation of the intervention in daily practice, and thus not to include development costs separately. If training of staff is a prerequisite for adequate execution of the intervention, then training costs represent a true use of resources and should be included. However, if only a one-time initial training is required, it is recommended not to include these costs. Care should be taken that intervention costs are not double counted with healthcare costs.

Patient and family costs include expenses incurred by patients as a consequence of the disorder under study, and costs of informal care. The following cost categories should be taken into account: patient-out-of-pocket expenses such as costs of over-the-counter-medication and costs of assistive aids (89% agreement); patient time costs associated with seeking and receiving care for the disorder under study (78% agreement); the costs travel required by patients (84% agreement); and informal care costs (94% agreement). Informal care costs include costs related to time spent and resources used by informal caregivers that are not compensated (76% agreement) or not partially or fully compensated (94% agreement).

Lost productivity costs are defined as costs related to reduced productivity from paid labour as a direct consequence of the disorder under study. The costs of both absenteeism (i.e. absence from paid and unpaid work) (100% agreement) and presenteeism (i.e. reduced efficiency when present at work) (83% agreement) should be included in an economic evaluation from a societal perspective. Lost productivity costs due to absenteeism from unpaid labour should not be included in an economic evaluation according to the panel (73% agreement). The main argument not to include these costs was the lack of standardized methods to value unpaid labour, leading to unreliable cost estimates and a risk of double counting. However, it was also argued that costs of absenteeism from unpaid labour may be an important cost category in specific patient populations such as elderly people, where the proportion of people performing unpaid labour is higher as compared with the general population.

Future healthcare costs are costs for treatment of disorders occurring in life years gained as a result of the intervention under study. Part of the future healthcare costs is directly related to the intervention, whereas other costs are not (e.g. costs for dementia treatment in added years because of successful cancer treatment). Future healthcare costs related to the intervention should be included according to the panel (100% agreement). A consensus was not reached on the inclusion of healthcare costs unrelated to the intervention (round 1, 53% agreement; round 2, 50% agreement). Panellists in favour of inclusion argued that unrelated future healthcare costs should theoretically be included if an intervention (significantly) prolongs life and if important differences in future costs between interventions are expected. However, others argued against inclusion on the basis that the calculations are difficult as many assumptions are made. The steering committee recommended the inclusion of related and unrelated future healthcare costs if the intervention is expected to result in an extension of life, because it represent a true use of resources.

A summary of relevant cost items per cost category is provided in Appendix 1 in the electronic supplementary material.

Measurement of resource use and lost productivity

The panellists did not agree on the most suitable method to assess healthcare utilization and were divided between patient-based reporting and the use of national insurance fund utilization databases (secondary-level data). Some panellists indicated that these databases are more accurate than patient-level data, whereas others argued that these databases are less precise, may not contain all relevant information and may create problems when data are linked to individual patients. The steering committee opted for patient-based reports as the most preferable method, because not all services are covered in national databases and such databases are not easily available for all countries.

The recommended methods to collect patient-reported data are resource use questionnaires and interviews, activity logs and cost diaries. If patients are incapable of self-reporting, proxy reports are recommended. Patient out-of-pocket expenses are country specific as they depend on the reimbursement level and funding system. Therefore, the patient is considered the most reliable source to obtain these data. Patient time costs are also preferably measured with use of patient reports. In situations where patient reports cannot be used, these costs can also be based on standard time estimates associated with seeking and receiving care. However, researchers should be aware that patient time costs in seeking care may overlap with absenteeism from paid labour, with the risk of double counting. When travel costs are being assessed, it is recommended to use standard distances from the patient’s home to the healthcare provider over patient-reported distances, to avoid random differences between groups (100% agreement). Costs related to time spent and resources used by informal caregivers should be based on self-reports.

No consensus was reached on what is considered the most suitable method to measure absenteeism from paid work (company-registered data or self-reported sick leave). The panellists in favour of using company registered data (50% of the panel) indicated that they are cheaper, easier to collect, more trustful and more credible than self-reported sick leave, whereas opponents (50%) argued that company-registered data are less precise, rarely available, and difficult to obtain in certain situations. The steering committee recommended that absenteeism from paid work should preferably be measured with self-report questionnaires, because self-reported productivity losses can be more accurately attributed to the disorder under study. Also, self-reported productivity losses may be more easily available than reports from a large number of different employers. The recommended method to collect data on presenteeism is to obtain ratings of both the quantity and the quality of work from the patient by standardized questionnaires (73% agreement).

Valuation of resource use and lost productivity

Opportunity costs are recommended to value resource use. Costs used in an economic evaluation should be representative of the country under study. Therefore, country-specific costs are preferred (100% agreement), and European average costs should not be used (71% agreement). Since country-specific costs may differ considerably, resource use rates and costs should be presented separately to facilitate the generalization of study results to other settings.

The preferred proxy measure for the opportunity cost of healthcare services is country-specific standard unit costs, when available (77% agreement). Standard unit costs include all costs related to the provision of a particular service. However, standard unit cost estimates should not be used for patient out-of-pocket expenses because of large variations between patients (75% agreement). The panellists did not agree on the most suitable valuation method for the opportunity costs of patient time and informal care (shadow prices or national average wages of unskilled labour sex/age specific, 54% and 46%, respectively). The arguments in favour of the use of national averages wages were the availability and “otherwise disease affecting highly skilled or paid people have an advantage”, whereas the counterargument was that patients are not unskilled. The steering committee recommended the use of sex- and age-specific average wages of unskilled labour for the valuation of patient time and informal care, although the committee recognizes this may underestimate the true opportunity costs associated with informal care. Use of shadow prices may be more flexible since they can be adapted to the specific informal care situation. With regard to costs of traveling by public transport, tariffs should be used (86% agreement). Tariffs are expected to be closely related to market prices, and are therefore considered to resemble opportunity costs adequately. For travel by car, it is recommended to use standard costs per kilometre/mile, including costs for petrol, maintenance, depreciation, and taxes (86% agreement). Lost productivity should be valued with age- and sex-specific national average wages (93% agreement). For the valuation of absenteeism from paid work, the friction cost approach is preferred over the human capital approach because the latter is expected to lead to overestimation of productivity losses (73% agreement). With the friction cost approach, the period over which the production loss is calculated is limited to the friction period (i.e. the time that an employer needs to replace a sick employee). It should be taken into account that the length of the friction period depends on the local economic situation and that friction periods are not available for most European countries. Therefore, countries should try to determine the friction period for their country or, when this is not feasible, perform a sensitivity analysis in which productivity losses are valued with use of the human capital approach.

Inclusion of VAT

VAT should preferably be included in the societal costs (79% agreement). VAT are indirect taxes on the domestic consumption of goods and services. Some goods are zero rated such as essential drugs and medical devices. Although VAT is a transfer from the individual to the government, it can be seen as part of the true costs of healthcare services according to the panel.

Discounting

Country-specific discounting rates should be used in the reference case (80% agreement). It is recommended to perform sensitivity analyses with the lowest and highest European discounting rates.

Study design

The choice of a model-based approach or a trial-based approach should depend on the research question according to the panellists. The study should be model based when the intervention effects are expected in the long term, when it is not possible to include all relevant treatment alternatives in one study, or when the incidence of the clinical end point is low. In other situations, a trial-based approach suffices.

Discussion

This Delphi study aimed to develop cross-European recommendations for identifying, measuring and valuing resource use and productivity losses in economic evaluations. The use of a societal perspective is recommended for economic evaluations in a cross-European context because this perspective provides a broad view of the impact of healthcare interventions on society as a whole. Resource use is preferably measured by means of patient-reported measures and valued with use of country-specific standardized costs. Lost productivity is preferably measured by means of self-report and valued with use of the friction cost approach with age- and sex-specific national average wages.

The Delphi design used allowed us to elicit the opinion of a heterogeneous international panel of experts with extensive expertise in economic evaluations within a relatively short time. By inviting the panel to suggest more suitable, alternative options in addition to the ones prespecified, we expected to capture recent developments in this field as well. Consensus was reached relatively easily for items regarding identification of cost categories. It was more difficult to reach consensus for items regarding assessment and valuation of resource use and lost productivity: in Delphi round 1, no consensus was reached for 21 the 52 valuation methods (40%) that were listed in the questionnaire. This is in line with previous studies that tried to harmonize existing national guidelines [4, 15].

Recently, EUnetHTA developed a guideline with a framework for the method of economic evaluations based on common denominators in existing national guidelines from 25 European countries. The recommendations from our study on how to measure and value resource use and lost productivity in economic evaluations complement the EUnetHTA guideline since it does not extensively cover these topics. Also, the recommendations on topics discussed in both guidelines are in line with each other. Together, the EUnetHTA guideline and the recommendations described in this article are expected to support researchers, healthcare professionals and policymakers when they are conducting and appraising economic evaluations.

Considerations

Because of its comprehensiveness, the societal perspective is recommended as the reference case. However, in some countries (e.g. Belgium, Italy, Germany, and the UK) decision making is solely based on the healthcare perspective. To provide relevant information for these specific countries, an additional analysis from a healthcare perspective is required. If indicated, additional sensitivity analyses can be performed from other relevant perspectives [e.g. health insurance–public funds or healthcare payer(s)]. From a societal perspective, prices should reflect societal opportunity costs. However, in some countries such prices are unavailable or may be closer to costs from the healthcare perspective, whereas other countries, such as the Netherlands, attempt to approach societal opportunity costs by providing standard costs. The Delphi panel was not asked to decide between discounted and undiscounted costs. However, to facilitate the transfer of the results to other settings, we suggest that both discounted and undiscounted costs are presented.

The lack of agreement among the panellists on the method to assess healthcare utilization may suggest that this is one area that it may not be possible to standardize, since national insurance fund databases differ considerably. On the basis of our Delphi study, the use of patient-based reports is recommended, but we are aware that may be infeasible in some economic evaluations, especially those that use a modelling approach.

Because of the lack of standardized methods to measure absenteeism from unpaid labour, these costs should not be included in an economic evaluation according to the panel. However, in some populations this may be an important cost category (e.g. elderly people). Therefore, further research is needed to develop appropriate methods to validly assess costs due to decreased productivity in unpaid work.

Among the panel and in the literature, strong differences of opinion exist with regard to the inclusion of unrelated future healthcare costs [41]. Inclusion or exclusion of these costs may have important distributional consequences for life-saving or life-extending interventions [42]. Thus, ignoring these costs may lead to suboptimal decisions [41]. Therefore, the steering committee decided to include these costs. Practical objections against inclusion can be tackled, at least for some part, as methods to accurately estimate unrelated future healthcare costs are being developed increasingly [42], although these methods are currently available for only a few EU counties. Further improvements in this area are needed.

Limitations and strengths

Some limitations of the current study need to be acknowledged. First, the Delphi study had a lower response rate than anticipated; only one quarter of the invited experts participated in at least one of the Delphi rounds. This may have led to selection bias. However, since the panellists were distributed across Europe, the results likely reflect the variety in opinions and methods applied across Europe. Moreover, we considered the number of panellists to be sufficient, since the literature generally recommends 10–18 experts for a Delphi panel [16]. Second, the decisions of the steering committee might be dominated by the Dutch perspective, because all members were from the Netherlands. However, the steering committee members took into account the panellists’ arguments when making decisions, and cross-national expertise on funding systems was available in the steering committee. Finally, the recommendations are broadly formulated. This means that potentially relevant items in economic evaluations among specific patient populations are not included in this article. However, this can also be considered a strength of the study since this makes the guidelines applicable to a broad range of situations.

Other strengths are the structured and systematic approach used to identify agreement and disagreement in the methods among the panel and to generate consensus about the recommendations presented in this article. By developing recommendations based on existing guidelines and opinions of experts from all over Europe, we expect that the recommendations can be validly used across Europe. Also, the inclusion of generally accepted methods for economic evaluation greatly enhances the feasibility of complying with the recommendations formulated in this articles.

Conclusion

We developed consensus-based recommendations for the identification, measurement and valuation of healthcare utilization and lost productivity that are applicable for European economic evaluations, despite differences between national guidelines. We think that these recommendations are generally acceptable because the recommendations were developed on the basis of a Delphi study among European experts in this field, and consensus was reached for most items within three rounds. Application of these recommendations will improve the comparability of economic evaluations conducted in Europe. Although pricing and reimbursement decisions will remain the full competence of national authorities, the recommendations will help decision makers better interpret the results of a study conducted elsewhere and facilitate the exchange of results between European countries. Moreover, it will allow countries to build on each other’s expertise and make cross-country evaluations easier to perform.