Prediction of severe community-acquired pneumonia: a systematic review and meta-analysis

We systematically searched Medline, Embase, and the Cochrane Controlled Trials registry by using the following key words: community-acquired pneumonia AND (decision tree OR clinical prediction rule OR clinical prediction score OR clinical decision rule OR clinical decision score OR management studies OR outcome studies OR ICU admission OR ICU need OR invasive management OR severity assessment). The search was performed for articles in English, French, Italian, Spanish, and German languages and limited to articles with an abstract and completed on the first of March 2012. To ensure a comprehensive literature search, we examined reference lists from retrieved articles and reference literature (guidelines and systematic reviews) and questioned experts in CAP for possible missing studies.

Eligible studies were prospective or retrospective studies evaluating clinical prediction rules in adult immunocompetent patients with CAP to predict the need for ICU admission, intensive treatment, or early mortality (< 14 days). The evaluation had to be performed during the first 24 hours after hospital admission. Studies addressing specific patient subgroups based on etiology or age were excluded. A prediction rule was defined as the combination of two or more clinical or biologic markers. Four investigators (CM, NG, SC, and OG) evaluated studies for possible inclusion. All studies were evaluated independently by at least two investigators. Nonrelevant studies were excluded based on title and abstract. For potentially relevant studies, the full text was obtained, and two investigators (CM, NG) independently assessed study eligibility and extracted the data on study design, patient characteristics, and outcomes. Disagreements were resolved by consensus or by discussion with a third reviewer (AP)

We used modified quality criteria based on the guidelines for assessing quality in prognostic studies [14]. Two investigators (CM, NG) assessed study quality independently. Each of six items was scored from 0 to 2. Studies with a total quality score between 11 and 12 were considered "good," between 9 to 10, "moderate," and 8 or less was considered "poor."

For each score, the diagnostic performances (sensitivity, specificity, likelihood ratios, and diagnostic odds ratios) to predict different definitions of SCAP (ICU admission, early death, or intensive treatment) at the usual cut-off were pooled by using the method of the inverse of the variance. Random effects were systematically introduced [15]. Heterogeneity was measured by the I-square index [16] and tested with the Cochran test. Potential heterogeneity factors were explored by subgroup analyses for the Pneumonia Severity Index (PSI) (only score with more than 10 studies). A sensitivity analysis was conducted to check the robustness of the pooled sensitivities and specificities by removing each study, one by one. The R package "meta: Meta-analysis with R, version 1.6-1" was used for these analyses. For the PSI and CURB-65 (Confusion, Urea, Respiratory Rate, Blood pressure, Age > 65 years) scores, a summary ROC curve was assessed by the approach proposed by Moses et al.[17]. As several sensitivities and specificities were reported in the studies at different cut-offs, we used a linear mixed model with a correlation structure to take the dependence of the measures into account. The 95% confidence intervals of the areas under the curves were obtained by bootstrap. This analysis was performed with S-plus 8.0 for Windows. The significance level was 0.05 for all analyses. Forest plots of the Sensibility and Specificity were used for the graphic display of the results.

We identified 11 main severity scores based on 20 variables. Components of the main severity scores are illustrated in Figure 2. Sufficient data were available to perform a meta-analysis on eight scores: PSI, CURB-65, CRB-65 (Confusion, Respiratory Rate, Blood pressure, Age > 65), CURB (Confusion, Urea, Respiratory rate, Blood pressure), ATS 2001 criteria, ATS/Infectious Disease Society of America (IDSA) 2007 criteria, SCAP score (Severe Community-Acquired Pneumonia), and SMART-COP (Systolic Blood pressure, Multilobar infiltrate, Albumin, Respiratory Rate, Tachycardia, Confusion, low Oxygen, low PH) (Table 2) Score definitions are included in the Additional file 1. Forrest plots for specificity and sensitivity of the eight scores for the prediction of ICU admission are provided in Additional file 2.

Renaud et al.[12] proposed a prediction rule (REA-ICU index) based on 11 predictors (see Additional File 1). This rule was derived to predict early ICU admission (day 1 to day 3), excluding patients with an obvious ICU indication at admission and patients with therapeutic limitations (not to be resuscitated, NTBR order) and validated on four North American and European prospective multicenter cohorts including 6,560 patients. AUC was 0.81 (CI, 0.78 to 0.83) on the overall population.

An abbreviated version of the PSI was tested on an administrative database [30], and some authors proposed to modify the CURB rule to include an oxygenation parameter [25] (CORB). These scores have not been validated in other cohorts.

We completed quality assessment according to the criteria developed by Hayden et al.[14].

Nine studies were considered good (scores ≥ 11); 23, moderate (9 to 10); and four poor (≤ 8) quality (see Additional file 3). Important differences were observed in terms of design, populations, and ICU admission rates. Twenty-nine studies included only hospitalized patients, and seven included both hospitalized and ambulatory patients. Patients with therapeutic limitations (NTBR order) were excluded in 15 of 36 studies, and patients with an obvious indication for ICU, in three [12, 42, 47]. ICU admission rate varied from 3.3% [26] to 23.9% [45], and receipt of intensive treatment in the ICU was highly variable (16% to 100% [25, 33]). When multiple measures of the same variable were available, as for vital signs, the measure introduced into the prediction rule was the first available one in four studies, the most abnormal one in five, and not specified in the majority of studies.

We performed an analysis to identify sources of heterogeneity for the only score (PSI) evaluated in at least 10 studies. Analyses did not reveal any significant factor of heterogeneity for sensitivity. Three factors influenced specificity: inclusion of NTBR patients decreased specificity; studies of good quality or high prevalence (≥ 15%) showed higher specificity. Nevertheless, the NTBR factor is highly correlated with the quality of the study (the NTBR patients were more frequently excluded in high-quality studies) and the prevalence (NTBR patients were more frequently excluded in studies with a high prevalence). NTBR exclusion is suspected to be the main factor explaining the heterogeneity observed in the specificity. The association between specificity and prevalence or quality may be caused by the correlation with NTBR exclusion.

Sensitivity analysis showed that the conclusions of the meta-analyses were robust and not caused by a single study. Sensitivity and specificity did not vary by more than 10%, except for CURB score: the study by Ewig [32] has a lower sensitivity and higher specificity than do other studies, but only four studies were included in this analysis.

Our review incorporates the most recent published studies and updates the systematic review by Chalmers et al.[60]. We also used somewhat different inclusion criteria and data analysis. We did not include subgroups of patients based on age or pathogen, leading to the noninclusion of three studies [61‐63]. Among the 40 studies included in the two systematic reviews, 24 were included in both, 12 in our work only, and four in the study by Chalmers et al. only. We decided against aggregating different definitions of SCAP, in an attempt to limit heterogeneity, and computed the performances of the prediction rules for the different definitions of SCAP. We included large recent studies [49, 50] and could estimate the pooled performance of the more-recent prediction rules, SMART-COP and SCAP score. This allows direct comparison between recent prediction rules. Although included studies differed partially between our systematic review and that by Chalmers et al., our final results are very similar, mutually strengthening their validity.

An important limitation of systematic reviews is their dependence on the quality of the included studies. Although the majority of included studies were considered of good or moderate quality, several pitfalls remain in the prediction of severe CAP. First, as already discussed, two biases are highly prevalent in these studies: inclusion in the studied population of patients not at risk for ICU admission (patients with therapeutic limitations); and use as a predictor of a surrogate of the outcome (use of mechanical ventilation and vasopressors, which are universally delivered only in an intensive or intermediate care unit).

Second, no universally accepted definition exists of severe CAP. The most frequently used proxy, ICU admission, is heavily influenced by ICU beds availability, local ICU admission policy, or subjectivity of the ICU specialist's evaluation. Use of a subjective decision such as ICU admission as a gold standard might lead to circular reasoning, because a perfect rule would be the one fitting usual practice. However, alternative definitions of SCAP, such as receipt of intensive treatment, do not seem to modify importantly operative characteristics of the prediction rules [23].

Third, definition of a "false positive" ICU admission is unclear: some patients might benefit from ICU admission even if they are not receiving vasopressors or mechanical ventilation (for example, through better fluid resuscitation).

Fourth, some of the studied rules have been fully incorporated in expert society recommendations. This might lead to contamination of ICU admission practices, further leading to an overestimation of their accuracy.

Finally, major heterogeneity was present among included studies, limiting the validity of the meta-analysis.