Development and validation of a bedside risk score for MRSA among patients hospitalized with complicated skin and skin structure infections

This retrospective cohort study used data collected from 62 hospitals in the Health Facts electronic medical record database (Cerner Corporation, Kansas City, MO). Health Facts is a database built from hospitals’ comprehensive clinical records including pharmacy, laboratory, admission, and billing information from all affiliated patient care locations. All admissions, medication orders, and laboratory orders and collections are labeled with date and time. Hospital billing/registration and encounter data are integrated with clinical information relating to the drugs dispensed and the results of diagnostic laboratory testing. Cerner Corporation has established Health Insurance Portability and Accountability Act (HIPAA)–compliant operating policies and procedures using statistical methods to establish de-identification.

Included patients were at least 18 years of age, hospitalized between April 2005 and March 2009, and met our diagnostic criteria in one of two ways: 1) Admission had a primary diagnosis from Additional file 1: Appendix A OR 2) Admission had a secondary diagnosis from Additional file 1: Appendix A, a primary diagnosis of sepsis (International Classification of Diseases, Clinical Modification [ICD-9-CM] codes 020.2, 038.xx, 790.7, 995.91, or 995.92), and no other source of infection documented by ICD-9-CM code. Patients were also required to have both 1) collection of a blood or skin culture that was positive for bacteria and 2) treatment with any intravenous (IV) antibiotic within 48 h of presentation. The positive index culture had to contain at least one organism that was not considered to be a skin contaminant. Cultures of organisms considered to be skin contaminants (Corynebacterium/diphtheroids, Staphylococcus epidermidis or other coagulase-negative Staphylococcus sp., Propionibacterium, Streptococcus viridians, Aerococcus spp, Bacillis spp [except B. anthracis]) were excluded from the cohort. Other exclusions were pregnancy or complications of childbirth, major trauma, and inpatient admission within 30 days with primary or secondary diagnosis of cSSSI (Additional file 1: Appendix A).

Microbiology results were prioritized to reduce the likelihood of colonizing microorganisms, though included patients could have polymicrobial infection. For patients with multiple culture results within the 48-hour timeframe, cultures from a skin source or venipuncture were prioritized over blood cultures drawn from a line, and certain sources of skin cultures with a high risk of contamination such as “swabs” were excluded. Lower extremity infections were identified using the 4th digit of the ICD-9-CM diagnosis code and/or the anatomic location of the skin culture, if present. Healthcare-associated (HCA) infection was defined based on the presence of at least one of the following factors: 1) admission from a chronic care facility, or 2) hospitalization within the previous 180 days or recent outpatient surgery, or 3) hemodialysis within 90 days or end-stage renal disease (ESRD) diagnosis code, or 4) chronic dependence on mechanical ventilation. Impaired immune function, based on specific ICD-9-CM codes and/or medication use, was examined separately. Chronic comorbidities present during admission or 12 months prior were identified by ICD-9-CM codes; baseline (within 48 hours of admission) laboratory variables were also recorded. Organ dysfunction measures at or before the time of the index culture were designed to be consistent with a Sepsis-related Organ Failure Assessment (SOFA) score equal to 2 or greater [13]. These included cardiovascular, respiratory, hepatic, hematologic, and renal dysfunction. Unadjusted hospital mortality served as the primary endpoint, while length of stay (LOS) among survivors was the secondary outcome.

Based on statistical and clinical considerations, we built and explored the usefulness of a simple summary score to identify infection with MRSA vs. a non-MRSA organism.

Our aim was to develop a simple summary score to be used at the bedside to serve as a clinical decision tool to distinguish patients with MRSA from those with an infection with another pathogen (non-MRSA). To accomplish this, we first compared patients admitted with MRSA to those with non-MRSA with regard to baseline demographic, clinical and hospital characteristics. To test for differences between groups, we used t-tests for continuous variables and the chi square test or Fisher’s exact test for categorical variables. When constructing the summary scores, potential covariates were included in the models if they reached significance at alpha=0.2 in the univariate analyses, or if a priori they were determined to have clinical relevance (Additional file 1: Appendix B) [14]. Variables were included only if they were present on admission or could be known within the first 24 hours of admission. The rationale for the 24-hour window was that inappropriately targeted empiric antimicrobial coverage within this time frame is associated with worsened outcomes [5‐11]. In the selection of laboratory variables, we only considered tests for which at least 95% of values were obtained within 24 hours of admission.

A split sample approach was used, whereby 80% of the patients were randomly selected for model development and the remaining 20% were used for model validation. To identify the most predictive variables of MRSA vs. non-MRSA infection, we constructed a series of backward stepwise logistic regression models on bootstrapped with replacement (simulated) datasets from the development set and created a frequency table indicating the number of times the eligible predictors entered these stepwise regression models. The variables appearing most frequently were then considered for inclusion in the final summary score [15]. The score was designed a priori to rely on no more than 5 predictors for ease of use in clinical practice (which compelled us to exclude highly predictive but rare variables from the final model). The final model’s predictors were included in a logistic regression model and the coefficients were fully standardized to a unit variance. The predictors in the summary score were then assigned integer-valued weights proportional to these standardized coefficients.

We calculated the prevalence of MRSA infection at every score value in the development cohort and compared these results to the prevalence in the validation cohort to assess calibration. A simple, overall classification rate is often misleading when the overall prevalence of a condition is either quite high or low. To better assess the predictive ability of the models, we calculated specificities and sensitivities at various threshold levels of the score(s) to derive likelihood ratios. With these statistics, we constructed Fagan’s nomograms to show a patient’s posterior probability of having MRSA if the score was above or below a particular threshold (i.e., being above a threshold level was analogous to a positive “test”) [16].

To augment usefulness at the bedside and to benchmark the score against the HCA risk factors, an additional assessment of score characteristics was done calculating sensitivity, specificity, positive and negative predictive values for each threshold score along the scale. These characteristics were then compared to the performance of the HCA definition as a way to discriminate the risk for MRSA from that for non-MRSA infection.

Among the 7,183 patients hospitalized with cSSSI, MRSA was cultured in 2,387 (33.2%), with the remainder (n=4,796) infected with non-MRSA (Table 1). Compared to patients with non-MRSA, those with MRSA were younger, less likely to be Caucasian, more likely to be on the medical service, and had lower comorbidity burden. Additionally, fewer of them had any risk factors for HCA infection (MRSA 32.6%, non-MRSA 36.8%, p<0.001) or a history of immune suppression (MRSA 22.9%, non-MRSA 28.9%, p<0.001).

Unadjusted hospital mortality was lower in the MRSA (3.0%) than the non-MRSA (4.7%) group, p<0.001. Similarly, the median hospital LOS was shorter in the MRSA (4.2 days) than in the non-MRSA (5.7 days) group, p <0.001.

The MRSA score was constructed on a random sample of 80% of the patients (development set, N=5,736) and validated on the remaining 20% (validation set, N=1,447). Factors differentiating MRSA infection from other organisms, based on the logistic regression, resulted in a score ranging from 0 to 8 points (Table 2). Those factors and their corresponding points were: age (range 0–3 points), African-American race (1 point), no evidence of diabetes mellitus (1 point), cancer (1 point) or renal dysfunction (1 point), and prior history of cardiac dysrhythmia (1 point). Increasing scores corresponded well to increased likelihood of MRSA infection in both development and validation cohorts (Figure 1).

Table 3 presents the prevalence of actual MRSA isolates among patients with a particular score. While approximately 20% of all cases of MRSA had the score of 0–1, among those scoring 7–8 nearly 2/3 had the organism. Between these extremes, the score exhibited a consistent dose–response relationship.

We additionally examined the test characteristics of various score thresholds for detecting MRSA, and compared those to the HCA definition (Table 4). A MRSA score of 5 or higher turned out to be superior to the HCA classification in all characteristics, while that of 4 or higher was superior on all metrics except specificity. Using the threshold of 5 as an example for the score, if a clinician is most interested in excluding the probability of MRSA, the negative predictive value (the probability of scoring below 5 given that there was no MRSA present) was 72.3%. This represents a 12% relative improvement over the negative predictive value of the HCA definition (64.3%).

A Bayesian analysis was performed via Fagan’s nomogram on the validation set (Figure 2). The threshold classification results are based on the scores of 3 (T-3), 5 (T-5) and 7 (T-7). Using the pre-test probability of MRSA of 33% (the overall prevalence in our cohort), at a threshold score of 3 or higher the posterior probability of having MRSA is 34.5%. Similarly, for the score of 5 it becomes 43.0%, and for the score of 7 it is 48.2%. Alternatively, if the score is below 3, the posterior probability of having MRSA diminishes to less than 25%.