Common drug review recommendations for orphan drugs in Canada: basis of recommendations and comparison with similar reviews in Quebec, Australia, Scotland and New Zealand

A list of orphan drugs was obtained from the Orphanet web site [27]. Orphanet is an online reference source for information on rare diseases and orphan drugs and the web site provides a comprehensive list of drugs granted an orphan designation in Europe [27]. The Orphanet portal was initially a component of the Canadian government initiatives designed to support research and approval of these drugs [2]. The Federal Drug Administration (FDA) list of drugs with orphan designations in the USA was also used to augment the list of drugs included in the study [28]. A list of the orphan drugs included in the study, and their indications, is provided in Table S1 in the Additional file 1.

Publicly available CDR recommendations were reviewed for each identified orphan drug (review period was January 2004 to October 2017) and the following information was extracted from the Recommendation and Reasons file [29]: date of CDR recommendation, generic and brand name of drug, drug indication, type of recommendation, basis of decision for recommendation, incremental cost-utility ratio (ICUR) (if provided) and whether or not recommendation was based on a request for reconsideration [29]. The latter is a process where participating drug plans or the manufacturer can issue a request for reconsideration following an initially negative draft CDR recommendation; this initiates an additional CDR review and issuance of a final recommendation [18]. Only final CDR recommendations and the date of the decision were included in the analysis; if a drug had more than one submission for the same indication, only the result of the most recent submission was recorded. In most cases, ICUR values recorded were those provided by the manufacturer; exceptions to this rule when CDR determined the ICUR are described in table footnotes. If the ICUR was reported as confidential with a CDR comment that the ratio exceeded acceptable values, it was assumed that the ICUR was >$100,000/quality adjusted life-year (QALY).

CDR recommendation categories have changed over time and the current options (March 2016) are: reimburse; reimburse with clinical criteria and/or conditions; do not reimburse [25]. However, over the timeframe of the study, CDR recommendation options were: do not list; do not list at the submitted price; list with clinical criteria and/or conditions; list [18]. Nevertheless, for this analysis recommendations were recorded only as list or do not list and whether the listing recommendation was conditional on a reduction in price.

The basis of decisions for recommendations was categorized in a similar manner to that defined in a prior study assessing all CDR recommendations [19]. In the majority of CDR reviews, reasons for recommendation are listed and Rocchi et al. [19] hypothesized that they were listed in hierarchical level of importance for at least the first two factors (see Supplemental Digital Content in reference [19]). On this basis, a primary reason for recommendation was defined, based on the first two reasons listed, in the following manner:

An identical definition was used in the current study to describe the basis of the decision for each orphan drug recommendation.

To quantify how CDR recommendations for orphan drugs compare with other public payers, the reimbursement recommendations for the same drugs and indications were reviewed in Scotland, Australia, New Zealand and Quebec.

In Scotland drug submissions are reviewed by the Scottish Medicines Consortium (SMC) whose task, similar to that of the National Institute for Health and Care Excellence (NICE) in England, is to provide advice to National Health Service (NHS) boards across Scotland on the clinical and cost-effectiveness of newly licensed medicines [30]. SMC has adapted the definition of orphan drugs provided by the European Medicines Agency (EMA) and has recently defined procedures specifically for the evaluation of end-of-life drugs and drugs used to treat very rare conditions [31]. As of May 2014, pharmaceutical companies can request that SMC convenes a Patient and Clinical Engagement (PACE) group for drugs in this class allowing patient groups and clinicians a stronger voice in decision making [31].

In Australia, drug-funding decisions are made through the Pharmaceutical Benefits Advisory Committee (PBAC) [32]. Although PBAC recognizes the importance of the treatment of rare diseases (defined as a disease with a prevalence of ≤ 2000 individuals in Australia), there are no specific evaluation procedures for drugs in this category [32]. However, patients can get access to orphan drugs through the Life Saving Drugs Program (LSDP), which provides funding for drugs that may have been rejected by PBAC because of an unacceptable cost-effectiveness ratio. Drugs eligible for the LSDP are defined by a number of criteria including disease rarity, impact of the disease on patient life-expectancy and current treatment alternatives [33‐35].

Drug funding decisions in New Zealand are made through the Pharmaceutical Management Agency (PHARMAC) with input from the Pharmacology and Therapeutics Advisory Committee (PTAC) [36, 37]. Historically there were no specific policies regarding access to orphan drugs in New Zealand. However, in 2014 PHARMAC reviewed their procedures for funding orphan drugs and introduced a trial contestable fund and bidding process for suppliers to encourage more competitive pricing [38, 39]. To support this process PHARMAC has allocated $25 million over 5 years for orphan drug funding [40].

Currently there is no specific policy regarding reimbursement of orphan drugs in Quebec, but INESSS has initiated a review of how rare diseases are managed in other countries, which led to a recommendation that similar policies be adapted in Quebec and that collaboration through participation in Orphanet be prioritized [41, 42].

To compare CDR recommendations for the selected orphan drugs, web sites with information on listings for the same drugs and indications in Scotland [43], Australia [44], New Zealand [40, 45] and Quebec [46, 47] were searched. If a specific drug had been evaluated in other jurisdictions, decisions were recorded as either not listed or listed; in the case of the latter, the specific listing criteria were not recorded. For Australia, drugs that were rejected and deferred by PBAC were included in the not listed category. Decisions for New Zealand include drugs approved for funding as a result of a test of a commercial process aimed at improving patient access to drugs for rare diseases [40].

Recommendation rates for CDR and the additional jurisdictions were reported as number and percent of total reviews with a positive recommendation. Fisher’s exact test was used to test for differences in CDR recommendation rate based on clinical, clinical and price and price parameters. It was also used to test recommendation rate difference in submissions with and without an ICUR. Concordance in recommendations between CDR and the other jurisdictions was expressed as the proportion of reviews with identical decisions (to list or not to list). The level of agreement between jurisdictions was determined using a kappa analysis. In this analysis a kappa value of ≤0.2 indicates a poor level of agreement while values ranging from 0.21–0.40 to 0.81–1.0 define levels of fair to very good agreement, respectively [48].

Using criteria previously defined to identify primary reasons for CDR recommendations [19], the majority of CDR orphan drug recommendations were based on the strength of clinical parameters alone but the subsequent positive recommendation rate was only 45.5% versus 86.7% for recommendations incorporating both clinical and price parameters (p = 0.0012). Therefore, there was a low probability of a positive recommendation when CDR decisions were driven by clinical outcomes and where drug cost was not a major factor in the evaluation. The relevance of clinical outcomes in driving CDR recommendations has been confirmed by Janoudi et al. [49] who in a recent review of 63 CDR orphan drug submissions (2004 to 2015) reported that the major reason for a negative reimbursement recommendation was lack of clinical effectiveness (38.5% of negative recommendations vs. only 7.7% for high cost-effectiveness/high cost). Clinical uncertainty has also been identified as a significant predictor of a do not list recommendation in all CDR reviews [19]. For most orphan drugs there is a degree of uncertainty regarding efficacy, effectiveness and safety [50]. Small potential patient numbers for clinical trials, reliance on surrogate outcomes, clinical trial periods that may not reflect disease duration, heterogeneity in patient response and unclear comparator therapies all contribute to this clinical uncertainty and compromise HTA assessments [50]. However, the March 2016 CADTH framework acknowledges that there may be limitations in clinical data for orphan drugs and that normal review procedures may have to be modified to address these limitations [25]. Although the present study presents historical data indicating that clinical uncertainty was a significant factor in rejecting orphan drugs for reimbursement, the data also suggest that the guidance provided in the revised CADTH framework has led to changes in how CDR reviews orphan drugs [25]. This is most apparent in the dramatic increase in positive recommendations in 2016 (86.7% in 2016 vs. 48.3% between 2004 and 2011). The overall recommendation rate over the complete assessment period (2004 to 2016) was 63.2%, a rate elevated by the more frequent listing recommendations in 2016. This compares with a CDR orphan drug recommendation rate of 55.4% (between 2004 and 2015) estimated from data reported by Janoudi et al. [49] when their data was expressed in the same way as in the present study (i.e., only including the most recent submission in the analysis for drugs with re-submissions). Overall CDR positive recommendation rates have been estimated at 52% (for the period 2003 to 2009) indicating that at least prior to 2016, orphan drugs were recommended for listing at a similar rate to other drugs [19].

A recent change in how CDR reviews orphan drugs was also apparent in CDR drug review comments. Whereas prior to the release of the 2016 framework, uncertain clinical data would have increased the probability of a negative recommendation, that same uncertain clinical data may now be more acceptable with criteria defining evidence development over time, particularly if the drug meets an unmet clinical need. As shown in the post-framework reviews described here (Table 3), elements of the CADTH March 2016 framework [25] were apparent, including recommendations for starting and stopping rules, real-world evidence development and characteristics of the care setting. However, almost all the positive recommendations in 2016 for drugs with high ICURs were conditional on a substantial reduction in price because CDR did not consider the drugs cost-effective at the submitted price. In addition, most recommendations came with clinical criteria and/or conditions defining drug use. As a result of the new framework, CDR appears more willing to accept clinical uncertainty through future real-world evidence development but has applied substantial cost reduction conditions for reimbursement.

Despite these changes, it is currently unclear how this new approach to recommendations for orphan drugs will impact patient access in Canada. It has been suggested [20, 51] that CDR recommendations with conditional price adjustment may guide subsequent pCPA pricing negotiations but since the pCPA price negotiations are confidential it is difficult to determine the final impact of such CDR recommendations. The new 2016 CADTH framework may actually delay patient access since the evidence summarized here suggests that the time taken for the CDR review has only resulted in a conditional cost recommendation for most reviewed drugs, the application of which is directly under the control of the drug plan (i.e. negotiation of a reduction in the drug price and adjustment of drug plan budgets to accommodate use of the drug). In addition, in some cases provincial Ministries of Health have decided to review orphan drug clinical data for themselves, and have made independent decisions on listing [52‐54]. This further suggests that the CDR (or pCODR) review of orphan drugs may be somewhat redundant, only delaying patient access to the drugs at the drug plan level where the final decision on listing is ultimately made.

Although this study is based on CDR recommendations, there is no legal requirement for the different public health plans to accept the CDR recommendation, although a positive CDR recommendation is a strong predictor of subsequent provincial listing. For example, a recent study found that for drugs with a positive CDR recommendation, 93% were subsequently available in Ontario [52]. However, for orphan drugs there is variable agreement between CDR decisions and subsequent provincial listing. Allen et al. [53] in a comprehensive review of 174 medicine-indication pairs in CDR reports (2009 to 2015) found 78.9%, 81.1% and 78.8% agreement between CDR recommendations and listing decisions in Alberta, British Columbia and Ontario, respectively, for all drugs. However, analysis of the Allen et al. [53] data for orphan drugs (22 of the drugs were identified as orphan drugs) indicated only 59.1% and 63.6% agreement with CDR recommendations in Alberta and Ontario, respectively, but 86.4% agreement in British Columbia; only 36.4% of the drugs had the same recommended listing in all three provinces. In the same way, a negative CDR recommendation does not necessarily preclude provincial listing and in Ontario 50% of all drugs with a negative CDR recommendation were subsequently available in the province [52]. Of the 22 orphan drugs identified in the Allen et al. [53] study, 45.5% had a negative CDR recommendation, but despite this 60.0% were listed in Ontario. This heterogeneity in provincial listing decisions for orphan drugs in Canada has also been noted by Menon et al. [54] and emphasizes the unequitable national access to this drug class.

The CDR orphan drug recommendation rate (63.2% overall) was similar to the rates determined in other jurisdictions, based on percent concordance, except for Australia where patients have greater access to orphan drugs, aided, in part, by the LSDP program [33]. However, the kappa analysis indicated little evidence of consistency in recommendations. Variation in HTA agency decisions has been noted by others and likely reflects varying interpretation of the validity and relevance of pharmacoeconomic evidence, local drug prices, variation in patient populations meeting clinical treatment criteria, effectiveness of potential market comparators and different interpretation of the same clinical trial data [17, 55]. The urgent issue facing many HTA agencies is how to apply cost and clinical criteria, initially defined for non-orphan drugs, to drugs for rare diseases so that patients can have equitable and affordable access to treatment. In many countries the solution is one of conditional orphan drug reimbursement where the “condition” can be based on a variety of factors including a drug price reduction, temporary reimbursement with clinical evidence development, higher willingness-to-pay thresholds, creation of special orphan drug funds or risk-sharing agreements [56, 57]. However, there is no consensus as to the most effective approach and the HTA environment is currently fragmented in terms of the review and reimbursement of orphan drugs, as shown by the poor agreement in listing decisions shown in this study [56, 57]. With regard to Canada, the latest trends at CADTH suggest that orphan drug manufacturers should be prepared for real-world evidence studies which may be required to demonstrate the clinical value of an orphan drug in real clinical practice.

Concordance of CDR and INESSS orphan drug recommendations, where drugs were evaluated within very similar healthcare systems, was 69.1% (kappa coefficient 0.3307), emphasizing the potential for variable interpretation of presumably the same country-specific clinical and economic data. This level of concordance between INESSS and CDR for orphan drugs is similar to the degree of concordance (64.0%) recently reported for all non-cancer drugs [58].

There are a number of limitations in the present study. The detailed analysis of CDR recommendations relied only on publically available information; any confidential information included in submissions was not available for this review. In addition, the reasons for recommendation were based on assumptions on the hierarchical level of importance in the listing of reasons [19]. There are limitations in applying a structured process to define a primary reason for a recommendation to a fluid CDR review procedure incorporating multiple factors. Nevertheless, in the absence of other methods to calibrate CDR review, this procedure does provide a tool that can be used to define the CDR recommendation process and may be particularly effective in comparative analyses. Given the new CADTH framework, and the recognition of the special characteristics of orphan drugs, the positive recommendation rate for decisions based on clinical parameters alone will likely increase from the 45.5% observed here because CADTH has defined procedures to address clinical uncertainty. In fact this may be already happening; data from all 68 indications showed that 48.5% of CDR recommendations were based on clinical parameters alone with a 45.5% recommendation rate, reflecting clinical uncertainty. In 2016 (15 submissions), 53.3% were assessed on the basis of clinical parameters alone but 75.0% of the recommendations were positive, albeit with conditional price reductions, suggesting more confidence in clinical data.

Finally, assessing an absolute positive recommendation rate for orphan drugs can be subjective, depending on which drugs were included in the review and the criteria used for their inclusion. In addition, only four HTA agencies were included for comparison which may have limited the generalizability of the results.