An Evaluation of “Drug Ineffective” Postmarketing Reports in Drug Safety Surveillance

Adverse event (AE) reporting has been a central part of the US FDA’s postmarketing drug safety surveillance for nearly 50 years [1]. While population-based databases have become additional sources of postmarketing safety information, especially in the past two decades, individual case safety reports (ICSRs) of AEs remain an important source of postmarketing safety information [2, 3]. Each data source has well-characterized strengths and limitations. A major limitation in postmarketing ICSRs is the frequent lack of sufficient detail to allow an independent reviewer to make a reasonable assessment of the potential relationship between a drug and a reported AE [4]. For example, lack of accurate product identification and relevant clinical details limit the inferences that can be made from ICSRs, especially when there are alternative potential explanations for the reported AEs.

The most commonly reported AE, based on frequency of Medical Dictionary for Regulatory Activities (MedDRA) preferred terms (PTs), in the US FDA Adverse Event Reporting System (FAERS) database is “drug ineffective” (DI) [5]. These DI reports in FAERS have not been assessed systematically for quality and inferential value from a pharmacovigilance perspective. The objective of this study is to describe the DI reports in FAERS and provide data to support recommendations on how to best evaluate these reports.

FAERS is an electronic database that currently contains over 14 million ICSRs describing AEs or medication errors. Approximately 95% of FAERS reports are submitted to the FDA by drug manufacturers, whereas 5% (called “direct reports”) are submitted directly to the FDA [5]. Any member of the public (e.g., consumers, healthcare professionals [HCPs]) can report an AE to the FDA or the manufacturer. When the manufacturer receives a report, they must in turn report the AE to the FDA in accordance with regulatory requirements. Reports in FAERS may contain narrative free text describing an AE, a list of products suspected in that event, and information identifying the reporter. In addition to the narrative description of the AE, reporters may supply additional information, such as past medical history, laboratory data, the names of suspect drug manufacturers, or the event outcome.

We searched FAERS for all reports received by the FDA between 1 September 2012 and 31 August 2016. The retrieved reports were stratified by those coded with and without the MedDRA PT “drug ineffective” for comparison. We summarized the following report characteristics: report type (i.e., manufacturer or direct), patient and reporter attributes, reporter country, reported outcomes, primary suspect product’s application type (i.e., new drug application [NDA], abbreviated new drug application [ANDA], biologics license application [BLA]), and concurrently reported PTs. The primary suspect product’s application type was determined from a structured field provided by the submitting manufacturer. The most frequently reported suspect products, by the product’s active ingredient, were compared between the DI and non-DI report groups. A product’s active ingredient was defined as the ingredient that has a pharmacological effect of a medicinal product reported by drug manufacturers. Additionally, we identified the active ingredients with the highest proportions of DI reports for products with at least 1000 total reports in the study period.

Following the high-level characterization of DI and non-DI reports, we conducted a manual evaluation to assess the availability of informative data elements in the report narratives and to classify reports by their potential utility in the assessment of the relationship between a drug and the AE (of drug ineffectiveness). This assessment was made to determine whether the report was “useful” from a pharmacovigilance perspective. We defined “useful” as reports containing the necessary information that would prompt a reviewer to consider further action, which in most cases would be obtaining additional information. For this study, a “useful” report contains criteria 1 and 2 and at least one of the other four criteria, as listed in Table 1. An assessment of causality was not conducted in the determination of a “useful” report. All FAERS reports were stratified by initial FDA received dates (September 2012–August 2013, September 2013–August 2014, September 2014–August 2015, September 2015–August 2016). Random sampling was conducted from DI reports, reflecting the proportions of reports in the periods. We restricted our manual evaluation to US reports given the limited reporting requirements for reports that manufacturers receive from outside the USA. A sample size of 552 was determined for manual evaluation assuming a prevalence rate of DI reports with potential utility of 10% and a precision of 2.5%. The prevalence of usefulness was estimated by piloting a review of 120 reports.

Data collected during the manual evaluation was aimed at determining whether information relevant to the evaluation of drug ineffectiveness was documented in the ICSR with respect to the suspected ineffective product. These included (1) classification of the product as brand (innovator version) or generic; (2) description of medication switching; (3) action taken with the product suspected to be ineffective; (4) presence of the suspected product’s batch or lot number; and (5) concurrently reported PTs. In this evaluation, a product was considered “brand” if an innovator’s product name was used or if an active ingredient was described as brand in the narrative. A product was classified as “generic” if a non-innovator product was specified or if an active ingredient was described as generic in the narrative. If a manufacturer name was provided in the description, the manufacturer (and the suspect product) was identified as brand or generic using the FDA’s list of Approved Drug Products with Therapeutic Equivalence Evaluations (commonly known as the Orange Book) [6]. Reports describing the use of both brand and generic suspect products were classified as “multiple”. Medication switching was defined as a switch from one medicinal product to another with the same active ingredient, the same dosage form, and the same route of administration. The switching patterns included (1) a patient taking drug A and drug A worked, the patient switched to drug B then DI occurred; and (2) a patient experienced DI with drug A, then switched to drug B and drug B worked. A switch may include brand to generic, generic to brand, or generic to generic switching. Reports describing ineffectiveness with a different batch or lot of the same product were excluded from the medication switching definition. PTs reported in addition to DI were further classified into those describing product quality issues or any AE other than a product quality issue. Reports were determined to contain additional PTs associated with product quality issues if additional PTs included at least one within the MedDRA high-level group term (HLGT) “product quality, supply, distribution, manufacturing and quality systems issues”. All included reports were evaluated independently by two reviewers. Differences in evaluation between the two reviewers were discussed as a group for consensus. The cases determined to be “useful” were further compared with the remainder of the sample. We summarized the resulting data using descriptive statistics.

During the study period, 3.8 million reports were entered into FAERS, of which 247,513 (6.4%) reports were coded with the PT “drug ineffective”. The overall characteristics of DI and non-DI reports are presented in Table 2. Compared with reports without DI, DI was more likely to be reported by consumers (69.8 vs. 48.1%) and less likely to have a serious outcome reported (26.2 vs. 56.3%). Most DI reports (88%) were from the USA. While age distributions were relatively similar between the DI and non-DI report groups (median age 57 years [interquartile range {IQR} 43–67] and 58 years [IQR 43–69], respectively), DI reports were more often missing the patient age (49.5 vs. 38.7%). In total, 62% of DI reports included additional coded PTs. The most frequently co-reported PTs in the DI reports were “product quality issue” (4.2%), “pain” (3.8%), and “fatigue” (3.2%). Of the non-DI reports, the most frequently reported PTs were “death” (4.5%), “nausea” (4.1%), and “fatigue” (3.9%).

Table 3 shows the top 20 suspect product’s active ingredients with the highest number of reports for the DI and non-DI groups. Adalimumab and etanercept were the top two products during the study period in both the DI and the non-DI groups. Relative to the non-DI reports, many of the most frequently reported suspect product’s active ingredients in the DI reports were those used for symptomatic management (e.g., pain: naproxen, pregabalin, gabapentin, buprenorphine, ibuprofen; allergies/asthma: loratadine, fexofenadine, fluticasone/salmeterol, albuterol) or where an effect is expected shortly after exposure (e.g., onabotulinumtoxinA, sildenafil, insulin lispro). Suspect products’ active ingredients with the highest number of DI reports in FAERS were adalimumab (8.0%), etanercept (5.7%), naproxen (2.4%), pregabalin (2.3%), and onabotulinumtoxinA (2.2%). In all, 100 active ingredients accounted for 80% of all DI reports during the study period. The proportion of DI reports within each active ingredient was not proportional across suspect products. For example, DI reports consisted of 41.2% of all reports received with onabotulinumtoxinA, but only 6.8% of etanercept reports given the large total number of etanercept reports. The products with the highest proportions of DI reports are provided in Table 4.

The results of the manual evaluation of 552 reports are presented in Table 5. We could identify the product related to the DI in the narrative field in 95.3% of the sampled reports, of which 75.2% described a brand product. The most frequently identified products in the sampled reports included adalimumab (8.5%), etanercept (5.4%), and naproxen (3.4%). DI associated with medication switching was reported in 6.2%. Ineffectiveness when using a different batch or lot of the same product was described in 1.3% of DI reports. The suspect product was continued in 14.1% of reports. The suspect product’s batch or lot number was reported in 17.2%. Of the 552 reports, 59.2% were coded with additional PTs, the most frequent of which were “fatigue” (4.9%), “product quality issue” (4.5%), and “off-label use” (3.3%).

We determined that 43 of the 552 reports (7.8%) met our definition of “useful”. Among the 43 “useful” reports meeting criteria 1 and 2, a total of 20 reports (46.5%) met an additional two or more criteria, and 12 reports (27.9%) met an additional three or more criteria. Characteristics (including the criteria used to classify reports as “useful”) of the 43 reports determined to be “useful” are in Table 6. The suspect products identified from the narrative field were generic in 51.2% (vs. 7.6% overall sample DI reports). The most frequently reported products were fentanyl (9.3%), alprazolam (7.0%), and adalimumab (4.7%). Medication switching was reported in 44.2% (vs. 6.2% overall sample DI reports). The suspect product was discontinued in 37.2% (vs. 29.7% overall sample DI reports). A higher proportion of “useful” reports provided a batch or lot number (39.5 vs. 17.2% overall sample DI reports). The majority (83.7%) of the reports were coded with additional PTs beyond “drug ineffective” (vs. 59% overall sample DI reports); “product quality issue” (23.3%), “product substitution issue” (18.6%), and “feeling abnormal” (11.6%) were the three most frequently reported PTs. Of the 43 “useful” reports, 44.2% included PTs related to product quality issues within the MedDRA HLGT “product quality, supply, distribution, manufacturing, and quality systems issues”.

DI is the most frequently reported AE in the FAERS database. Our analysis demonstrated that, in contrast to all other reports in the FAERS database, the majority of DI cases did not report a serious outcome and were more likely to be reported by consumers. These reports were primarily from the USA. The US Code of Federal Regulations requires responsible parties (i.e., applicants, manufacturers, packers, or distributors) to submit adverse experiences associated with drugs or therapeutic biologic products to the FDA, including experiences associated with a failure to produce an expected pharmacologic action. In addition, responsible parties are not required to submit non-serious AE reports from foreign marketing experience [7].

The suspect products with the highest number of DI reports during the study period are used primarily for the management of symptomatic conditions (e.g., adalimumab, etanercept, naproxen, loratadine), suggesting that consumers have self-awareness of worsening or no improvement of their own subjective experiences. Similarly, the suspect product with the highest proportion of DI reports within each active ingredient (docosanol) is used primarily for symptom management. Poitras et al. [8] conducted similar research using the Canadian Vigilance Database and also identified high proportions of DI reports (ranging from 14.6 to 20% annually using the standardized MedDRA query “lack of efficacy/effectiveness”). The most frequently reported drug classes in lack-of-efficacy reports were tumor necrosis factor-α inhibitors and proton pump inhibitors [8].

Our manual evaluation revealed that many of the reports lacked the clinical details needed to distinguish the reported DI from disease progression. Interpretation of DI reports is complicated by the variable efficacy of approved products. Evidentiary standards for drug approval are that “substantial evidence that the drug will have the effect it purports of is presented to have under the conditions of use prescribed, recommended, or suggested in proposed labeling thereof” [9]. This does not translate to effectiveness in every patient who takes the drug, as evidenced by the efficacy of some of the most frequently reported products reporting DI (e.g., subcutaneous adalimumab 40 mg weekly achieved an American College of Rheumatology 20 response rate in 53% of patients with rheumatoid arthritis, 74% of patients demonstrated a clinical response from subcutaneous etanercept 0.4 mg/kg twice weekly for polyarticular juvenile idiopathic arthritis, and 81% of patients responded to intradermal onabotulinumtoxinA 50 units for > 50% decrease in axillary sweat production) [10‐12]. In addition, consumer expectations regarding efficacy may differ from that of HCPs [13, 14]. For example, patients’ high expectations may be shaped by general optimism, attitudes, and advertisements or other media representations. Although HCPs may generally anticipate that a medication will be effective, their expectations may be refined by considering individual patient characteristics such as age and disease severity [15]. This discrepancy of efficacy expectations between consumers and HCPs may be an additional reason relatively more DI reports were submitted by consumers.

Our ability to attribute ineffectiveness to product quality issues relies at a minimum on accurate identification of the product in question; however, prior studies have suggested that reliance on a product’s name or the reporting manufacturer’s name alone may result in misclassification [4]. Nonetheless, ICSRs of AEs remain an important source of postmarketing safety information. Safety issues relating to ineffectiveness have been evaluated by the FDA using FAERS data. One example includes an analysis of DI reports for two generic methylphenidate extended-release (ER) products. These generic ER products were formulated to be administered once daily. In this case, the types and quality of DI reports received by the FDA were determined to be “useful” because they contained specific details describing the failure of therapeutic effect during the latter part of the day. The reports also provided specific information that identified the manufacturer. These reports suggested the generic ER products may not produce the same therapeutic effects for patients as the innovator product. This prompted an FDA investigation that resulted in regulatory actions [16].

The low yield of potentially useful DI reports in contrast to the high volume of DI reports received annually highlights the need to develop strategies to efficiently identify which reports are more likely to contain useful information. In the “useful” reports, generic products tend to be reported as a suspect product more frequently and were often accompanied with the PT “product quality issue”. The information about medication switching or batch or lot change can be useful especially if the patient responded to a prior batch or lot, but this must be considered in the context of disease progression. DI reports without enough information to determine the suspect product in the narrative field or coded solely with the PT “drug ineffective” may have limited utility. Opportunities may exist to use natural language processing or other tools to identify features in the narratives that may be indicative of a report’s utility. Additional studies are needed to identify other characteristics or surveillance algorithms that may be useful for prioritizing review of DI reports.

Our study has certain limitations. We did not capture all potential reports describing drug ineffectiveness because we limited our search to the PT “drug ineffective”. We selected this PT because it was the most frequently reported MedDRA PT within the high-level term (HLT) “therapeutic and nontherapeutic responses”, and the HLT encompasses concepts broader than ineffectiveness. Other reports describing the concept of ineffectiveness would not have been captured by relevant terms included in the HLT because they may be only coded with event-specific PTs. For example, a report of ineffectiveness may describe increases in blood pressures after switching anti-hypertensive drugs but may be coded with the PT “blood pressure increased” rather than “drug ineffective”. While we determined the sample size needed to accurately estimate the proportion of DI reports considered “useful”, our resulting small sample of “useful” reports limits the generalizability of the specific characteristics within the subset. Nonetheless, this sample captured findings such as the increased availability of lot or batch information relative to the overall population of DI reports. Furthermore, our definition of useful was based on the expertise of reviewers with pharmacovigilance experience, which may limit reproducibility.

Many DI reports involved non-serious AEs and were reported by consumers. The most frequently reported products were used primarily for symptomatic management. Although DI was the most commonly reported AE during the study period, the yield from “useful” reports was low. The “useful” reports were often related to product quality issues and mentioned medication switching or batch or lot information; consequently, these characteristics can be important elements in useful report identification. Additional studies to identify strategies for the efficient identification of reports more likely to contain useful information are needed.