Issues Related to the Frequency of Exploratory Analyses by Evidence Review Groups in the NICE Single Technology Appraisal Process

The UK National Institute for Health and Care Excellence (NICE) Single Technology Appraisal (STA) process is usually undertaken for a technology for a single indication and includes the production of a submission by the manufacturer or sponsor of the technology. NICE commissions independent Evidence Review Groups (ERGs) to critically appraise company submissions as part of the STA process. In their critique of the evidence submitted by companies, the ERGs often undertake exploratory analyses to explore uncertainties around the company’s model and their implications for decision making. The number and type of exploratory analyses undertaken varies between appraisals. The ERG reports are a central component of the evidence considered by the NICE Technology Appraisal Committees (AC) in forming their recommendations. The findings of the committee are used to produce the Appraisal Consultation Document (ACD) and, after further considerations and a consultation period, a Final Appraisal Determination (FAD) is produced, which results in NICE guidance. The STA process is outlined in detail in NICE’s Guide to the Process of Technology Appraisal [1]. The company is expected to follow the decision-analytic approaches as described in the Guide to the Methods of Technology Appraisal [2], and the submission is expected to contain an evaluation of the clinical effectiveness and cost effectiveness of the technology. It is the responsibility of the ERG to determine what additional analyses are required and to undertake them, although they may be constrained by the availability of data, the model structure, or time. The ERG may also identify and correct technical or programming errors. This critical appraisal of the company submission, and any additional work as a consequence of this critical appraisal, forms the basis of the ERG’s report. NICE’s remit to the ERGs is not overly prescriptive, allowing ERGs to use their acknowledged expertise and judgement in the methods used to critically appraise company submissions. This is appropriate due to the wide variation in complexity and quality of the company submissions received.

There are currently nine ERGs:

An additional ERG, the West Midlands Health Technology Assessment Collaboration, undertook STAs during the period 2005–2010. ERGs are assigned to work on STAs based on availability and lack of conflicts for specific topics.

Some assessment of ERG report production process has been undertaken. For example, Wong et al. [3] assessed approaches used by ERGs to critically appraise search strategies within company submissions. Previous research has highlighted issues with company submissions that are particularly challenging to the ERGs [4]. Carroll et al. [5] suggested that company STA submissions could be improved if attention were paid to transparency in the reporting, conduct and justification of the review, and modelling processes and analyses, as well as greater robustness in the choice of data in the model and closer adherence to the scope or decision problem. Kaltenthaler et al. [6] also recommended the need for clear and transparent reporting of company submissions, and for a clear and concise rationale for the synthesis of clinical data, the development of economic models and the assumptions used to develop models.

The aim of this study was to note the number of exploratory analyses undertaken by the ERGs within the NICE STA process and to assess whether their frequency might be explained by variables such as disease area or a company’s base-case incremental cost-effectiveness ratio (ICER). For the purpose of this research, an exploratory analysis was defined as any additional analysis that generated an ICER and was included in the ERG report section titled ‘Exploratory and sensitivity analyses undertaken by the ERG’ (most commonly reported in Sect. 6 of the suggested ERG report template). This is an underresearched area and this study was commissioned by NICE to develop understanding of this aspect of the STA process. This research is of interest to all key stakeholders in the STA process, including the ERGs, pharmaceutical companies, AC members and NICE. The full research report forms part of the National Institute for Health Research Health Technology Assessment (NIHR HTA) monograph series [7]. The objectives of the study presented here were twofold.

This was a joint project undertaken by researchers from the University of Sheffield and the University of York, all with considerable experience of the NICE STA process. The 100 most recently completed STAs (2009–2014) for which guidance has been published were selected for inclusion in the analysis. A list of the STAs included in this analysis is provided in Appendix 1. Data were extracted from over 400 separate documents associated with these STAs, including ERG reports (unredacted versions, including confidential information used by the ACs), the first ACD issued and the final FAD issued.

A data extraction tool to extract relevant data to address the project objectives was developed and piloted to ensure usability and to standardise extraction [7]. The following items were included in the data extraction form:

All data extractions were double-checked by a second researcher, and a narrative synthesis [8] of the extracted data was performed, summarising key data through text and tables to highlight any potentially important patterns or relationships in the data. The mean and median numbers of exploratory analyses per ERG report were calculated. The mean number of analyses was used to provide a simple binary variable with which to test some assumptions about relationships between the number of exploratory analyses and variables such as disease area and ICER. It was considered a priori that the disease area and an estimated cost-effectiveness ratio of £20,000 per quality-adjusted life-year (QALY) were the key variables potentially most likely to predict the incidence and frequency of exploratory analyses. This was due to the known impact of disease area on other elements of STAs [9] and the perceived importance of the £20,000 per QALY ICER for NICE decision making [10]. An assessment was also made to identify any changes in the number of exploratory analyses undertaken over time, and whether the number and type of exploratory analyses differed by the ERG undertaking the critical appraisal. A more in-depth investigation of the types of exploratory analyses undertaken, and their role and impact on NICE decision making, is provided elsewhere [7].

The key data used in the synthesis were then reduced to whether an STA conducted more or less than the overall mean number of exploratory analyses, and whether just ‘one or more’ exploratory analyses were explicitly cited as having an influence on a recommendation (defined as being mentioned in the ACD or FAD). These arbitrary selections were made as a means of making the most of the data to address the objectives of the project and are explained more fully elsewhere [7, 11].

In this analysis, the vast majority (93%) of ERG reports reported one or more exploratory analyses, with a mean of 8.5 exploratory analyses per report for the 93 reports where exploratory analyses were undertaken. Of the 93 ERG reports with at least one exploratory analysis, a total of 40 (43%) included eight or more such analyses. The likelihood of an ERG performing eight or more exploratory analyses does not appear to be affected by the disease area covered by the STA or the company’s base-case ICER. In previous research, Barham [9] analysed data from the first 18 STAs undertaken by NICE and found 56% of these to cover cancer topics. In an analysis of the first 4 years of the NICE STA process, 48% of STAs were undertaken in cancer topics [14], slightly higher than reported in this study. This shows that the percentage of STAs that are cancer topics may be decreasing over time.

The proportion of ERG reports with eight or more analyses appears to be relatively stable over time and not related to any developments in the process between 2009 and 2014. Although there have been changes to the NICE technology appraisal process and methods guides during this time period, these appear to have had little effect on the number of exploratory analyses undertaken by the ERGs. As shown in the example of the analysis of ScHARR data, there does not appear to be a time-dependent trend as the number of analyses undertaken by the ERG appears to be stable with respect to time. Additional analyses of the data, both from other ERGs and from pharmaceutical companies, show similar results (data not shown).

The number of exploratory analyses varied by ERG. For the 10 ERGs undertaking STAs, the mean number of exploratory analyses per report ranged from 2.3 (West Midlands) to 11.4 (ScHARR). The reasons for this variation are unclear and potentially complex, and will include such factors as how thoroughly the company has explored the uncertainties and plausible alternative scenarios. It should be noted that no regression analyses were performed to explore the relationship between the mean number of analyses per report and variables such as ERG, disease area or year due to the limitations of the data because there was only one independent variable in any category (cancer in disease area) that exceeded 20 in number, with the majority being very small numbers, and with the result being that any such analysis would be underpowered. Other potentially influential factors, such as the complexity and perceived quality of company submissions, were also not explored or analysed. Therefore, this finding should be interpreted with caution and more research is needed to determine the relationship between variables such as ERG and number of exploratory analyses undertaken, which may also vary according to the skills, experience and judgements of the ERGs and that of individuals within the ERGs. This issue was not explored within this study.

Only seven of the STAs included in this analysis had no exploratory analyses undertaken by the ERGs, principally due to serious flaws identified by the ERG in the company model, which would have rendered analyses irrelevant. In only one of the STAs were no exploratory analyses undertaken by the ERG because the ERG was satisfied with the model and analyses presented by the company. The exploratory analyses undertaken by the ERGs serve various functions: to correct errors and violations within the company’s model; to address uncertainties in the evidence base; and explore a range of plausible scenarios, and therefore support NICE AC decision making. The exploratory analyses undertaken by ERGs frequently influence both ACD and FAD recommendations [11].

This analysis is a good reflection of current practice as the most recent 100 STAs were included. The data extraction tool was extensively piloted and the double-checking of all key data across the 100 STAs by at least two experienced cost-effectiveness modellers reduced the likelihood of inconsistency and inaccuracy in the data. The use of narrative synthesis was principally descriptive, and reduced the likelihood of overstating relationships in the data. A reductive approach was taken to managing data that might be affected by interpretation or by poor reporting in the original documents. However, there are some limitations to this research. The descriptions of analyses undertaken were often highly specific to a particular STA and could be inconsistent across ERG reports, and thus difficult to interpret and categorise. In addition, small inconsistencies might have affected the data extraction, which may be due to several people being involved. A number of other factors not considered in this study may have influenced the number of exploratory analyses undertaken by the ERGs but this was beyond the remit of this research.

This analysis of over 400 documents provides an overview of some of the principal factors potentially affecting the number of exploratory analyses undertaken by the ERGs. The wider study reported that four types of exploratory analysis were conducted in relation to companies’ models: fixing errors; addressing violations; addressing matters of judgement; and the provision of a new, ERG-preferred base-case. Ninety-three of the 100 ERG reports contained at least one of these analyses, and the most frequently reported type of analysis related to the category ‘matters of judgement’. The results of the wider study also suggest that these additional analyses undertaken by ERGs were highly influential in the policy and decision-making process [7, 11]. More in-depth analysis is needed to understand how ERGs make decisions regarding the exploratory analyses to be undertaken. More research is also needed to fully understand the types of exploratory analyses most useful to ACs in their decision making.

There is no clear pattern to the presence or frequency of exploratory analyses; they do not appear to be predicted by the disease area covered by the STA, the time the STA took place, or the company’s base-case ICER. In addition, there does appear to be a pattern in the mean number of analyses conducted by individual ERGs, but more research is needed to understand this relationship.