A framework for assessing Health Economic Evaluation (HEE) quality appraisal instruments

The framework consists of 36 items organized within 7 dimensions, each of which captures a specific domain of quality: Dimension A (“purpose and scope”, items A1-A5), Dimension B (“stakeholder involvement”, items B1-B3), Dimension C (“rigor of development process/validity”, items C1-C8c), Dimension D (“reliability/ reproducibility”, items D1-D4d), Dimension E (“clarity of presentation”, items E1-4), Dimension F (“applicability”, items F1-F5), Dimension G (“evaluation”, items G1-G2). The quality assessment framework is presented in Table 2.

To test the established framework and to demonstrate the shortcomings of existing HEE quality appraisal instruments, four well-known, often-cited, and widely-used HEE quality appraisal instruments, which have been subject to relatively more scrutiny than most others, were selected: the Quality of Health Economic Studies (QHES) instrument developed by Chiou et al. [21], the British Medical Journal (BMJ) guidelines for economic submissions established by Drummond and Jefferson on behalf of the BMJ Economic Evaluation Working Party [22], the Consensus on Health Economic Criteria (CHEC) list devised by Evers et al. [23], and the Good Practice Guidelines for Decision-Analytic Modeling accomplished by Philips et al. [24].

The BMJ list, the CHEC list, and the Philips list were chosen because they are recommended by the Cochrane Handbook for Systematic Reviews of Interventions [25] for critical appraisal of the methodological quality of health economic evaluation studies. In the chapter related to systematic reviews of economic evaluations, the Centre for Reviews and Dissemination’s guidance for undertaking reviews in health care [26] also refers to the BMJ list and the Philips list as instruments to assess the quality of economic evaluations. The QHES instrument was selected because it is an example of a quality scoring system which has been the subject of controversy in the literature.

As can be seen in Table 3, the BMJ list and the QHES instrument are broader and more comprehensive than the other two quality appraisal instruments: this is because the Philips list is solely designed for model-based economic evaluations and the CHEC list is only intended for undertaking systematic reviews of trial-based economic evaluations. In Table 4, the main characteristics of the HEE quality appraisal instruments are provided.

The experiences of applying the quality assessment framework to the HEE quality appraisal instruments are presented in Table 5.

The quality dimension of “purpose and scope” (dimension A) is fulfilled by all HEE quality appraisal instruments, even though not all items are explicitly described. In terms of the quality dimension of “stakeholder involvement” (dimension B), only Evers et al. (CHEC list, [23]) do not declare whether they have any conflicts of interest. Furthermore, at none of the different HEE quality appraisal instrument development stages were all the key professionals (e.g., economists, clinicians, epidemiologists, and statisticians) involved. However, all the quality appraisal instruments used methods to ensure that the perspectives of the target audience informed the development process (e.g., by participation on the development group, or by external review of drafts of the appraisal instruments). The main differences in the quality of the four HEE quality appraisal instruments relate to the quality dimension of “rigor of development process/validity” (dimension C). Especially with the BMJ list, the different stages of the development process are not reported. Moreover, as methodology advances, the date for updating the appraisal instrument should be stated. Only Philips et al. [24] refer to the need for periodic updates, but when these should take place remains unclear. The lack of formal validity is one of the main limitations of all four quality appraisal instruments. Only the QHES instrument was formally validated in terms of construct validity [21]. Other limitations refer to the quality dimension of “reliability/reproducibility” (dimension D). As mentioned before, the development process of the BMJ list is not documented. All four HEE quality appraisal instruments were published in peer-reviewed journals and, hence, external reviewers were involved in appraisal instrument development during the generic review process. However, in order to develop the BMJ list and the Philips list, additional external experts were convened to discuss drafts of these quality appraisal instruments. In addition, provided that external reviewers should not have been involved in developing the appraisal instrument, the QHES instrument and the CHEC list have not been externally reviewed before their publication (certainly, except for the review process). For the appraisal instrument to be effective with regard to reliability and reproducibility, it also needs to be piloted/pretested among the target audience. Only the QHES instrument was pretested among 60 experts in the field of health economics, who evaluated the methodological quality of three health economic analyses, first on a visual analog scale, and then using the grading system. In respect of the quality dimension of “clarity of presentation” (dimension E), the appraisal instruments established by Philips et al. [24] and the grading system developed by Chiou et al. (QHES instrument, [21]) do not provide specific and unambiguous items in those cases where more than one question refers to the same criterion, resulting in ambivalent assessments. Additionally, on account of the rather technical nature of the questions provided by Philips et al. [24], these are only suitable for specialist readers with expertise in the field of decision-analytic modelling and with knowledge of the disease area. Philips et al. [24] state that without that knowledge, it is a complex matter to decide whether all structural assumptions are justified, or whether all feasible and practical options have been evaluated. Further, because of the problem with the interpretation of the term “justified” or “appropriate”, it might be difficult to use these four quality appraisal instruments in a straightforward manner. Some items/questions are highly dependent on the judgment of the respective user and thus have an unavoidable element of subjectivity. For example, based on a comparison of three instruments for appraising the quality of health economic evaluations (BMJ list, CHEC list, and QHES instrument) Gerkens et al. [27] found that the reviewer has a greater influence on the results of the quality assessment than the appraisal instrument itself. Another problem within this dimension concerns the operationalization of the questions. Chiou et al. (QHES instrument, [21]) and Evers et al. (CHEC list, [23]) provide questions in a yes/no format, but in specific circumstances some questions may not be applicable to the study context. Because of the weighting of the criteria, only the key items of the QHES instrument can be easily identified. Regarding the quality dimension of “applicability” (dimension F), all appraisal instruments provide a standard reporting format and present items of methodological quality and transparency. All but the QHES instrument give detailed assessment instructions (to a greater or lesser extent) and all appraisal instruments except for the BMJ list specify their strengths and limitations. The QHES instrument is the only quality appraisal instrument to use a quality score. Concerning the quality dimension of “evaluation” (dimension G), all but the QHES instrument describe the evaluation methods. However, none of the instruments describes the extent of adherence by the target audience.