Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department

A reimbursement claim database enables large cohorts to be set up, providing comprehensive data at a relatively low cost [1]. Their use in pharmacoepidemiology has considerably increased in recent years [2]. In France, numerous studies have been conducted using the French National Health database (SNDS, previously known as SNIIRAM) [3‐7]. The French hospital database (PMSI), part of the SNDS, provides a discharge diagnosis (ICD-10 codes) for all patients admitted to hospital in France. It is considered that hospital-based data, and discharge codes in particular, can be used as valuable sources of information to define patient populations, assess comorbidities [8] or the severity of disease, determine patient outcomes [3] and drug effectiveness [4], and detect adverse events, including major bleeding [5‐7].

Major bleeding is the most feared serious adverse reaction when using antithrombotic agents. Estimating the occurrence of major bleeding events is therefore a key issue. Patients presenting major bleeding are mostly referred to hospital, which therefore makes hospital-based data useful. Nonetheless, caution is needed regarding the accuracy of codes in hospital-based data for major bleeding event identification: for instance when assigning primary or secondary discharge diagnoses, a focus on the reimbursement of the care delivered could hide the real reason for admission; coding inaccuracies or inconsistencies can occur across care sites, and bleeding events that are not coded could be overlooked; hence a validated algorithm is crucial.

Emergency wards are obviously optimal settings to observe and report serious adverse drug reactions for drugs prescribed in the community. With regard to major bleeding events, the validation of a hospital-based data algorithm could gain from a confrontation with medical charts from emergency wards.

First, we observed overall low sensitivity (65%); a third of the major bleeding events identified by our standard procedure were not detected by the ICD-10-based algorithm (false negatives). Second, we highlighted differential sensitivity and specificity across types of bleeding, with the highest values for intracranial hemorrhage. One hundred fifty-five cases out of 637 (24%) were false positives, with a large majority being non-serious, involving mostly “other” bleeding and GI bleeding events.

Major bleeding is an adverse event common to all types of antithrombotic drugs. An automated approach to identifying major bleeding in real time would be particularly useful. Indeed it would enable continuous monitoring and early signal detection in the area of pharmacovigilance. However, accuracy in coding for major bleeding is required to yield trustworthy results on the basis of hospital databases. To date, validation studies on ICD-10 code-based algorithms are scarce and focused on positive predictive values: only one study [9] identified major bleeding from emergency ward discharges using 35 ICD-10 codes for ICH or GI bleeding; a random sample was independently reviewed by two trained chart reviewers to validate the diagnosis, but no criteria to define major bleeding were applied, except for the location; the analysis showed an overall good positive predictive value of 88% (95%CI 83 to 91), with better estimates for ICH (90%) than for upper GI bleeding (74%). It can be noted that other forms of major bleeding events were not studied. These results based on emergency ward discharges were similar to others from studies using hospital discharge records (ICD-10 codes): Kokotailo et al. reported a positive predictive value of 98 and 91% for ICH and subarachnoid hemorrhage respectively [10]. Cunningham et al. showed that an algorithm identifying bleeding-related hospitalizations from the primary discharge diagnosis had a positive predictive value of between 89 and 99% in distinguishing specific bleeding sites [11]. The accuracy of upper GI bleeding codes in one Dutch administrative database using the ICD-10 coding system showed a positive predictive value of 77% [12]. In an assessment of four different coding systems in different European countries, it was concluded that positive predictive value is associated not only with the code itself, but is also with the way the code is used [12]; France was not part of this study. In France, main discharge diagnoses may not reflect the motive for referral, but rather what most impacts hospital resources; indeed the PMSI database has a primarily financial objective, not an epidemiological point of view. Lastly, Delate et al. recommended a manual chart review to validate warfarin-related bleeding events from administrative data [13]. Ruigomez et al. also advocated additional information to prevent misclassification as regards major GI or urogenital bleeding events [14]. Our findings are in line with these previous results and have highlighted differential measures of accuracy between ICH and GI. When evaluating anticoagulant safety profiles, it would be wise to perform separate analyses according to these outcomes. It is well known that non-differential misclassification biases the risk towards the null, but it can be noted that NOAC trials reported a lower risk of ICH but no significant decrease of GI bleeding; hence differential positive predictive values could be problematic when evaluating overall safety profiles.

The reported low sensitivity is a concern. This result might reflect a discrepancy between the motive for referral and the discharge coding (false negatives); it is worth noting that the PMSI does not collect emergency ward data; in any case, emergency ward data is thought to be unreliable with considerable inconsistency as a result of a lack of standardization. Only primary hospital discharge diagnoses were used in the index test to define major bleeding [5]. The consequence when using this algorithm (index test) will be an underestimation of the incidence of major bleeding. It has already been observed that when the code is listed as the most likely diagnosis or the admission diagnosis, a true bleeding event has occurred 96% of the time [15].

False positives from the algorithm (index test) were mostly non serious gastrointestinal bleeding or non serious other bleeding events. The point here is whether the algorithm catches in-hospital serious bleeding. This is important to consider when the question focuses on serious bleeding as a reason for hospital referral, which means bleeding potentially related to drug delivered on an ambulatory basis. Including in the analysis in-hospital serious bleeding might biased the results because at this time patients may not be exposed to the drug they were prescribed before hospital entry.

Our study has several strengths. Our sample size is by far the largest among studies testing the validity of ICD codes. We reviewed a consecutive set of charts irrespective of the ICD-10 coding allocated. Using negative controls, we calculated the sensitivity. In contrast, previous studies have been published without negative controls and have only reported positive predictive values. The medical chart review was blinded to the discharge diagnoses.

Our study also has several limitations. Firstly, the study was carried out by one medical expert reviewer in a single center. However, the reviewer followed objective criteria to determine the presence of major bleeding. On the other hand, inter-rater variability related to different level of expertise would have been an issue. Secondly, our definition for major bleeding was conservative, and it is likely that we underestimated the numbers of certain major bleeding events, especially with respect to the ISTH definition [16], which includes a drop in hemoglobin level of 20 g L^− 1. The PMSI database does not include laboratory results. In addition, to take account of a drop in hemoglobin level, a reference level is required, which is not straightforward. Our definition applied only to hospital-based care. Therefore, data from individuals who do not seek medical attention or who are only seen in outpatient clinics or surgeries was not captured. For major bleeding, we thought this would not lead to a substantial bias, except for bleeding-related sudden death. Thirdly, we used a two-step approach as the standard procedure; an extensive chart review was simply unrealistic, because of the expense involved in reviewing so many charts. A previous pilot study has shown good sensitivity and specificity for the first automated step. Of course, medical review of all charts would have been the true reference standard.

To conclude, the external validity of bleeding diagnostic codes has not been previously assessed in the French PMSI database. To the best of our knowledge, this is the first report. Our results showed overall low sensitivity, and, interestingly, different measures of accuracy across bleeding types. The results therefore provide support for specific data collection and a medical validation approach rather than an ICD-10-based algorithm for assessing the incidence of major bleeding.