Description and validation of a spectrum score method to measure antimicrobial de-escalation in healthcare associated pneumonia from electronic medical records data

The approach to assigning a numerical spectrum score to antimicrobial regimens has recently been summarized [7]. First, national VA Corporate Data Warehouse (CDW) susceptibility data for organisms and antimicrobials tested during years 2008–2012 were used to estimate antimicrobial susceptibility if available. The VA CDW includes culture and susceptibility results for 7 million veterans who receive care in 152 medical centers. Percent susceptibility was calculated for individual antimicrobial-organism pairs utilizing one isolate per patient per year. Next, remaining organism-antimicrobial pairs were categorized as possessing no intrinsic activity (NA), no VA susceptibility data available (ND), or further confirmation of susceptibility estimates required. NA classifications were based upon current FDA approved prescribing information, Clinical Laboratory Standards Institute (CLSI) standards for susceptibility testing, and tertiary references [9-42]. Current CLSI reporting recommendations were applied preferentially. For example, the CLSI-based susceptibility of Staphylococcus aureus to oxacillin was used to populate the susceptibility values for other β-lactams (except ceftaroline) with activity against Staphylococcus aureus. Organism-antimicrobial pairs without approved susceptibility methods, but suppressed results, were handled individually; where possible findings were cross-referenced with susceptibility results identified through literature sources.

Delphi panelists indicated that antimicrobials covering organisms with high potential for developing resistance should receive extra weight in the spectrum score [7]. Consistent with panelist preferences, ordinal susceptibility values for organism-antimicrobial pairs involving Staphylococcus aureus, Enterococcus faecium, Escherichia coli, Klebsiella spp., and Acinetobacter spp. were multiplied by a factor of 1.25, and Pseudomonas aeruginosa was multiplied by a factor of 1.75. Domains for each organism’s weighted or un-weighted scores were added to create a composite spectrum score for each antimicrobial on a 0–60 spectrum of activity scale.

To account for overlapping coverage in combination regimens, the following approach was taken. First, if none of the regimen antimicrobials possessed activity against a species, a zero was assigned to the organism-combination regimen pair. If one of the regimen antimicrobials possessed activity against a species, the active antimicrobial susceptibility was used to populate the organism combination regimen pair. For other regimens where a combination of antimicrobials possessed activity against a species, the proportion of organisms susceptible to a multi-antimicrobial regimen was estimated as one minus the joint probability of resistance to all regimen antimicrobials, assuming that antibiotic susceptibilities were independent [7]. Spectrum scores were then computed identically to individual antimicrobial regimens. Table 1 illustrates the steps involved in generating spectrum score values for an individual and combination antimicrobial regimens.

Inpatient antimicrobial administration within the VA is documented utilizing Bar Code Medication Administration (BCMA) technology. For each administered dose of antimicrobial, data regarding the dose and route of administration are recorded electronically with a time stamp [43]. Antimicrobial use data was obtained for a VA-wide cohort (years 2008–2012) of inpatient admissions with HCAP [8,44]. Each systemically administered antimicrobial the patient received during each calendar day of admission was extracted from the CDW [45]. As combination antimicrobial therapy was commonly administered, data were “smoothed” to prevent over and under-calculation of spectrum scores on days when different antimicrobials were simultaneously being initiated or discontinued. Smoothing was accomplished by inferring the intent to treat from antimicrobial administration data. Skip days, where antibiotics that were administered less frequently than once daily, were filled when between adjacent calendar days where the same antibiotic was administered (e.g., vancomycin administered every 48 hours). Skip days were assigned the same antibiotic regimen as the adjacent calendar days. To avoid double counting antibiotics when an antibiotic regimen was switched, an antibiotic day was not counted when there was a change in a multidrug regimen from the current, previous, and next days, the antibiotic was given in the past two days but not in the next three, and the day did not fall on the beginning or end of an antibiotic treatment course. For example, switching from cefepime and vancomycin to levofloxacin and clindamycin on day 4 would appear as a four-drug regimen unless these rules are invoked, in which case only the latter regimen would be counted (Table 2).

Refinement and validation of the spectrum score method was performed in three stages. First, 300 vignettes were created based on daily antimicrobial regimens obtained from a random sample of patients who met HCAP criteria [8,44]. Vignettes included antimicrobials administered on each calendar day of hospitalization and microbiology findings obtained in the first two days. Three antimicrobial stewards [mean (SD) 17.3 (0.7) years of experience] who were unfamiliar with the spectrum score method reviewed the vignettes. The stewards ranked each case on a 7-point Likert scale: de-escalation (score >4), no meaningful change in therapy (score =4), or escalation (score of < 4). Spectrum scores were determined for regimens administered on calendar days 2 and 4 of hospitalization; day 4 scores were subtracted from day 2 scores resulting in a spectrum score ∆ for each vignette. A positive spectrum score ∆ indicated de-escalation. Sensitivity and specificity of the sign of spectrum score ∆ to predict antimicrobial steward de-escalation opinion (i.e., reference standard) were calculated. To determine if varying the weight for coverage of resistant organisms impacted test characteristics of the method, spectrum score variants with differing weights (no weight, 1.25, 1.5, 1.75, 2.0, 3.0) for coverage of these organisms were calculated for the cases, and then compared to expert opinion in a similar manner.

Based upon observations in the initial exercise, 40 pairs of vignettes were modified to include the administration route for each antimicrobial. Each pair of vignettes contained identical or almost identical antimicrobial regimens with the same spectrum; however, in one case the regimen on day 4 contained one or more antimicrobials administered orally (PO) whereas in the other case all antimicrobials were administered intravenously (IV) (i.e. amoxicillin versus ampicillin). These cases were evaluated by the same experts and compared to the spectrum score ∆ as described. Mean paired difference in Likert scores between the IV and PO cases was calculated for pairs in which the PO case had 50-100% PO therapy and for pairs with >0% but less than 50% PO therapy on day 4. A linear regression model relating mean Likert and spectrum score Δ for vignettes with 100% IV therapy on day 2 and day 4 was used to express the means of paired difference in Likert in spectrum score scale (paired Likert means divided by the slope). The mean values converted to the spectrum score scale provided additional credits to the spectrum score Δ to account for vignettes with minimal conversion to PO therapy (>0% but <50%) and greater or full conversion to PO therapy (50%-100%) on day 4. Finally, 100 vignettes were selected from the original 300 HCAP cases and the route of antimicrobial administration data was added. These vignettes were assessed for de-escalation status by three new antimicrobial stewards unfamiliar with the spectrum score method [mean (SD) 14.0 (5.2) years of experience] using the same 7-point Likert scale. Mean Likert scores were compared to the spectrum score ∆ credited for IV to PO conversion. Only vignettes with 100% IV therapy on day 2 were eligible for the credit. Test characteristics of the spectrum score method to predict antimicrobial de-escalation were performed as above. Table 2 illustrates application of the spectrum score method to measure de-escalation in a theoretical case, and Figure 1 summarizes the process for development, refinement, and validation of the spectrum score method. A procedure that can be used to apply the spectrum score method is available in Additional files 1 and 2.

This research complies with all Federal guidelines and VA policies relative to Human Subjects and research. The VA Puget Sound Healthcare System, University of Utah, and Idaho State University Human Subjects Committees approved the study and its conduct was in compliance with the Helsinki Declaration.

Spectrum scores were determined for 14 organism domains which included 19 bacterial species and 27 categories of antimicrobials. Table 3 illustrates the ordinal susceptibility scores for organism-antimicrobial pairs. There were 513 possible organism-antimicrobial pairs for consideration of which 53% had sufficient VA susceptibility data, 6% were determined by applying CLSI-based rules, 28% were identified as possessing NA, and 15% required further confirmation or determination of susceptibility results with external sources. Suitable references could not be identified in a 3% of cases and domain scores were assigned by investigator opinion.

Findings for the initial validation exercise have recently been overviewed [7]. Briefly, there were 142 distinct antimicrobial regimens (79% combination regimens) administered in the 300 HCAP patient cases. Day 2 and 4 spectrum scores in the vignettes ranged from a minimum of 4.0 to a maximum of 60.0. The sensitivity and specificity of the spectrum score method to identify antimicrobial de-escalation events as determined by antimicrobial stewards was 86.3 and 96.0%, respectively. Adjustment of weights assigned for coverage of intrinsically resistant organisms did not improve the sensitivity and specificity of the spectrum score method beyond the weighting proposed by the Delphi panel, and spectrum score ∆ was more predictive of de-escalation events with application of these weights (P = 0.05, difference in area under the curve between weighted and un-weighted receiver operator curves). Upon inspection, it appeared that in select vignettes stewards inferred that a patient received a PO antimicrobial on day 4 even though the route of antimicrobial administration was not stated (e.g., ceftriaxone administered on day 2 followed by cefpodoxime administered on day 4). Despite many of these regimens possessing similar spectrum scores (i.e., spectrum score for both ceftriaxone or cefpodoxime was 25.25) , experts scored these vignettes higher if regimens contained antimicrobials with PO dosage forms available compared to regimens with IV antibiotics on day 4 (e.g., ceftriaxone on day 2 and day 4).

In the second vignette exercise, the mean (± SD) Likert score for paired vignettes containing identical or nearly identical antimicrobials with the same spectrum but differing routes of administration by day 4 was 5.0 (1.5) and 4.6 (1.5) for PO and IV cases (P = 0.002), respectively. The mean (± SD) paired difference in Likert scores was 0.44 (0.50) for pairs where the percentage of PO therapy in the PO case was 50-100% by day 4 (P < 0.001) and 0.24 (0.68) for pairs where the percentage of PO therapy was > 0% but < 50% in the PO case by day 4 (P = 0.12). Based on the linear relationship between mean Likert score and weighted spectrum score ∆ (slope = 0.072, P < 0.001) for vignettes with 100% IV therapy on day 2 and day 4, the paired mean difference in Likert score for vignettes with 50-100% PO therapy on day 4 was equivalent to approximately 6 spectrum score points, and was equivalent to approximately 3 spectrum score points for >0% but <50% PO therapy on day 4. Adding the two credits to spectrum score ∆s for applicable regimens improved detection of de-escalation events in the set of paired vignettes as classified by stewardship experts.

Taken together the vignette exercises suggest that inclusion of the route of administration was an important consideration in the assessment of de-escalation events, and that IV to PO conversion was viewed favorably when assessing de-escalation events. Table 4 summarizes the three refinement and validation exercise findings.

There were five discordant cases between the spectrum score with PO credits and reference method. Tetracyclines, which were only included in five vignettes, were involved in three of the discordant cases. Reviewers scored regimens in favor of de-escalation when tetracyclines were included on day 4 compared to the spectrum score method. Another discordant case involved the addition of a duplicative spectrum PO antibiotic to an IV regimen on day 4, which was classified an escalation by reviewers and a de-escalation by the spectrum score method due to the PO credit awarded. In the final discordant case, one of four antimicrobials was administered PO on day 2, but the antimicrobials remained identical on day 4 with two antimicrobials administered by the PO route. Experts scored this case as a de-escalation; however the case was not eligible for the credit as not all antimicrobials were administered IV on day 2.