Paradoxical ventilator associated pneumonia incidences among selective digestive decontamination studies versus other studies of mechanically ventilated patients: benchmarking the evidence base

There are four objectives here: First, to derive a VAP-IP benchmark and prediction range derived from observational (benchmark) groups. Second, to summarize VAP-IP separately for the control and intervention groups from studies of two broad approaches to VAP prevention that have been included in systematic reviews; studies of SDD versus studies of non-anti-microbial methods of VAP prevention. Third, to assess the dispersion among the group specific VAP-IP of control groups and intervention groups versus the VAP-IP benchmark using caterpillar plots. Finally, to assess the impact of group level factors as possible explanatory variables toward the group specific VAP-IP in meta-regression models that include both the benchmark and the prevention study groups.

This analysis is limited to component groups from studies of patients receiving mechanical ventilation as abstracted in nine published reviews (four non-systematic and five systematic) of VAP incidence and specific VAP prevention methods [1‐9]. The unit of analysis here is the component patient group, whether observational (benchmark) [1‐5], or control or intervention groups from studies of various methods of VAP prevention [6‐9].

The inclusion criterion for this analysis was a study of adult patients receiving prolonged mechanical ventilation in intensive care units (ICUs) for which VAP-IP and denominator data had been abstracted in one of the nine reviews [1‐9]. The exclusion criteria as specified in the Cochrane review [6] are applied to achieve harmonization across the studies obtained from all nine reviews. That is; studies based on specific pre-selected types of patients (patients undergoing elective esophageal resection, cardiac or gastric surgery, liver transplant or suffering from acute liver failure), studies of non-ICU populations, populations for which the proportion receiving MV for >24 hours was <50% and studies for which VAP-IP data were not available. Also, studies of pediatric populations, and studies published before 1984 do not appear among the studies abstracted in the review of Liberati et al. [6] and these study types are also excluded.

The benchmark groups are those groups of observational studies as abstracted in one of five reviews of VAP incidence [1‐5]. Any intervention study abstracted in one of these five reviews of VAP-IP incidence was not used in the derivation of the benchmark.

The component groups of studies of non-antimicrobial methods of VAP prevention are as abstracted in one of three systematic reviews of various methods of gastric acid suppression [7], open versus closed methods of tracheal suction [8], or passive versus active humidification [9] as methods of VAP prevention. In the gastric acid studies, the interventions studied were those that might suppress gastric acid (for example, ranitidine or antacid treatment) versus interventions that did not (for example, no treatment or sucralfate) [7]. The designation of control and intervention groups were as indicated in the systematic reviews of open (control) versus closed (intervention) methods of tracheal suction [8] and passive (HH, control) versus active (HME, intervention) humidification [9]. The component groups from the studies of SDD are as abstracted in the Cochrane review [6].

The primary outcome is the VAP-IP, which is the incidence of ventilator associated pneumonia per 100 patients. The VAP-IP and its denominator were taken for all component groups as abstracted in the review documents in which they appeared.

Additional information abstracted directly from the original publication was whether the mode of VAP diagnosis required bronchoscopic sampling versus tracheal sampling methods, whether <90% of patients received at least 24 hours of mechanical ventilation, and the proportion of patients admitted to the ICU for trauma. The scoring of study quality was also abstracted from each systematic review. However, each systematic review used different quality scoring systems and scoring was not used in the non-systematic reviews. The indicator of highest study quality in this analysis was whether the study received a majority score in the source systematic review. Data were extrapolated from tables and figures if not available in the text. Care was taken to stratify patient groups appearing across more than one publication.

A caterpillar plot is a forest plot-like display of group specific odds and 95% confidence intervals with the studies listed in rank order of increasing event rate. This display reveals both the overall symmetry of the individual group results and their deviation from the overall mean. This display shows the impact of group size with the larger groups, having greater precision, expected to deviate less from the summary or benchmark.

The VAP-IP data were converted to logits for analysis as follows; if D represents the denominator, N represents the numerator, and R represents the proportion (N/D) of the VAP-IP, the logit(VAP-IP) is log(N/(D-N)) and its variance is 1/(D*R*(1-R)) [15, 16]. This variance formula was used to calculate the group specific 95% confidence intervals. Using these calculated logits and logit variances, the metan command [17] in STATA (release 11.0, STATA Corp., College Station, TX, USA) generates summary logits by a random effects method together with the standard errors (SE) and tau², which are measures of within and between group variances, respectively, and the associated 95% CI's. The metan command also generates the caterpillar plots of the group specific logits and 95% CI's.

The VAP-IP benchmark was derived as the mean logit VAP-IP and 95% confidence interval derived together with a 95% prediction interval. The later is calculated using the metan command as mean ± 1.96 * (SE² + tau²)^0.5 [17]. In each of the caterpillar plots, both the overall VAP-IP mean derived from the groups in the plot and the 95% prediction interval derived from VAP-IP benchmark range are displayed.

To test the stability of the benchmark, five replicate derivations of the VAP-IP benchmark were derived using the VAP-IP data abstracted from the four non-systematic and one systematic reviews individually [1‐5].

The calculated logits and logit variances were used with the metareg command [18] in STATA (release 11.0, STATA Corp.) to perform meta-regression models that incorporate group level factors as predictors. There are six meta-regression models of logit VAP-IP including the benchmark groups with either the control (models 1 to 3) or the intervention (models 4 to 6) groups of the prevention studies. Models 1 and 4 include group membership (benchmark, SDD study or non-antimicrobial method study), as the only predictors. Models 2 and 5 include three additional group level properties as predictor variables; whether <90% of patients in the group received >24 hours of MV, whether the mode of diagnosis of VAP required bronchoscopic sampling and the proportion of trauma admissions to the ICU. Models 3 and 6 replicate models 2 and 5 but are limited to those studies that had received majority quality scores in the source systematic reviews. Regression coefficients were compared using the lincom (linear combination) post-estimation command in STATA.

Meta-regressions models 2 and 4 were repeated after exclusion of studies for which the proportion of patients receiving >24 hours of mechanical ventilation was <90% or unknown. Also, meta-regressions models 3 and 6 were repeated with component groups from 19 studies of SDD that had received a quality score of one out of two included.

Three meta-regression models were performed as described in the methods to evaluate several group level properties as predictors of the group specific logit VAP-IP's of the control (Table 4) and intervention (Table 5) groups versus the benchmark groups.

For the control groups versus the benchmark groups (Table 4; meta-regression models 1 to 3), membership of a control group of an SDD study was a consistently positive predictor. For the intervention groups versus the benchmark groups (Table 5; meta-regression models 4 to 6), membership of an intervention group of an SDD study was a negative predictor of logit VAP-IP but not consistently significant.

In comparing these factors in the meta-regression models, membership of a control group of an SDD study differed significantly versus membership of a control group of a non-antibiotic study in model 1 (P < 0.001), model 2 (P < 0.001) and model 3 (P = 0.003). By contrast, membership of an intervention group of an SDD study did not differ significantly versus membership of an intervention group of a non-antibiotic study as a predictor in model 4 (P = 0.7), model 5 (P = 0.6) or model 6 (P = 0.3).

Meta-regressions models 2 and 4 were repeated after exclusion of studies for which the proportion of patients receiving >24 hours of mechanical ventilation was <90% or unknown. Also, meta-regressions models 3 and 6 were repeated with component groups from 19 studies of SDD that had received a quality score of one out of two included. With both of these re-analyses, the findings were replicated (data not shown).

There are several limitations of this analysis. The random effects method of analysis presumes that the groups in each summation are representative of a 'random' selection of an undefinable super-population of groups. The VAP-IP benchmark derived here may only be representative of patient groups as found within systematic reviews.

Only nine reviews were used in this analysis and other smaller reviews have not been included. However, other reviews applicable to this patient group can be tested against the 95% prediction range derived from the benchmark groups here. For example, systematic reviews of kinetic bed [136] therapy and topical chlohexidene [137] as methods for the prevention of VAP had identified 10 and 7 studies respectively. Of the 17 studies identified in these two systematic reviews, only two studies, one from each systematic review, had a control group with a VAP-IP above 47.3%, the upper 95% limit of the prediction range derived from the benchmark groups here, whereas three had an intervention group VAP-IP below the lower 95% limit of the prediction range derived from the benchmark groups.

Also, studies of SDD which had a non-concurrent design have not been included in the meta-regression. This would help to test the postulated contextual effect of SDD. Among the SDD studies included here, three have a third non-concurrent control group arm in addition to the two concurrent arms [101, 104, 131]. The VAP-IP of all three of these non-concurrent control groups is less than 24% [12].

A further limitation was that the number of group level factors that could be explored was limited by those that were readily available and identified for all the groups in the analysis. Origin from a European country and year of publication were tested and found not to be significant in preliminary analyses (data not shown). Other factors such as the prevalence of antibiotic use have not been explored beyond that accounted for by duplex study design. Also the duration of mechanical ventilation has not been considered beyond the group average, which appeared to be similar across the strata (Table 1). The appropriate investigation of these factors would require patient level data to control for the possible influence of ecological bias [138].

There are three possible interpretations of these paradoxical findings. First, publication and citation bias need to be considered. Deriving the benchmark from four non-systematic and one systematic reviews was done to test the stability of the benchmark estimate (Table 2). In replications of the benchmark range from these five reviews separately this varies by no more than five percentage points.

Given that 11 of the 33 control groups from studies of SDD are above the upper limit of the 95% prediction range of the benchmark where only 2.5% of the distribution would be predicted to be found, this could be taken to indicate a deficit of 407 groups below the upper limit of the 95% prediction range of the benchmark {407 = (11 * 97.5/2.5)- 22)}. This estimate corresponds to an earlier test for publication bias using a funnel plot method which indicated a deficit of >500 'inlier' groups with VAP-IP < 45% from studies of SDD that had been unpublished or were otherwise 'missing' [12].

A second interpretation is that the possible impacts of unmeasured and unknown patient level risk factors for VAP-IP have not been evaluated in this analysis. However, for such a risk factor to account for the discrepancies between the VAP-IP of the control groups of SDD studies versus the benchmark groups is unlikely. Such putative risk factors would need to be consistently strong across the range of studies and yet have a profoundly uneven distribution between the SDD studies versus other studies. This is in contrast to the inconsistent strength and direction of the known VAP risk factors [2, 3].

For example, duration of mechanical ventilation is the strongest patient level risk factor for VAP with increases of approximately 2 per 100 patients per day of ventilation during the second week of ventilation [139]. The discrepancies in VAP-IP noted here between the control groups of SDD studies versus the benchmark would equate to a difference in mean duration of ventilation across all groups of 6.8 days.

A third interpretation is a possible contextual effect of SDD. The possibility of contextual effects due to cross colonization and infection within the ICU environment resulting from SDD use as was postulated in the original study of SDD [10] needs to be considered [140]. SDD is known to alter the colonization among recipients [141, 142]. However, identifying cross colonization within a single study is difficult. A major limitation toward testing this postulate is that colonization pressure [143] and cross colonization, two crucial intermediary steps, have not been measured in any of these studies.