Systematic review and meta-analysis of the value of initial biomarkers in predicting adverse outcome in febrile neutropenic episodes in children and young people with cancer

With multi-modality therapies, children with malignancy have an excellent chance of survival, with overall rates approaching 75% [1]. Deaths are largely due to their disease, but around 16% of deaths are from complications of therapy [2, 3]. This proportion depends on the underlying malignancy, and the risk of death from infection remains high in some groups, for example, acute myeloid leukemia [4]. Robust risk stratification, which reliably predicted those children at high risk of complications, could target more aggressive management, where children at very low risk of having a significant infection could be treated with reduced intensity and/or duration of hospitalized antibiotic therapy [5]. There are a wide range of differing approaches to this risk stratification, largely built on simple clinical data [6‐8], demonstrating only moderate discriminatory ability.

The ability of specific serum biomarkers to predict adverse consequences in patients with febrile neutropenia has been explored, for example, C-reactive protein (CRP), pro-calcitonin (PCT), interleukin-6 (IL6) or interleukin-8 (IL8) [9‐12]. These studies have been small in the numbers of patients and episodes and the researchers could not reach definitive conclusions. Drawing these reports together and synthesizing their results should improve our understanding of their clinical usefulness.

Although systematic reviews have been conducted previously in adults [13] and non- immunocompromised children [14, 15], their results are difficult to compare. There are data to suggest children and adults with neutropenic fever vary in the nature of the infections which afflict them [16], implying any review needs to take into account the specific population under study.

This review aimed to identify, critically appraise and synthesize information on the use of biomarkers at initial evaluation for the prediction of the outcome of febrile neutropenic episodes in children/young adults and to highlight important problems in the current methods used in such analyses.

The review was conducted in accordance with "Systematic reviews: CRD's guidance for undertaking reviews in health care" [17] and registered on the HTA Registry of Systematic Reviews: CRD32009100485. It sought studies which evaluated the diagnostic ability of serum biomarkers of inflammation/infection in children or young people aged 0 to 18 years of age, taken at the onset (within 12 hours) of an episode of febrile neutropenia. Both prospective and retrospective cohorts were included, but those using a case-control approach were excluded as these have been previously shown to exaggerate diagnostic accuracy estimates [18].

Three hundred, sixty-eight articles were initially reviewed, and 72 retrieved for more detailed examination. Twenty-five articles provided quantitative outcome data in the form required for the review (see Additional file 3). The included studies included 2,089 patients and over 3,585 episodes, assessing 14 different markers of inflammation or infection (see Table 1). The study outcomes were grouped into: bacteremia, invasive fungal infection, significant/documented bacterial infection, sepsis and death. The population in the studies varied, with most being a mixture of hematological and solid malignancies, and very little data from stem cell transplant recipients (see Table 2 for further detail). Thirteen of these contributed to 1 or more meta-analyses while the remaining 12 studies did not provide data which could be included in any meta-analysis. (see Figure 1). Three biomarkers and 2 outcomes could be included in the meta-analysis: 11 studies provided data on CRP [9, 26‐35] and documented infection. Four studies provided data on PCT [28, 29, 31, 33] and documented infection. Four provided data on IL6 [31, 36‐38] and documented infection or gram negative bacteremia.

This systematic review of the predictive value of serum markers of inflammation and infection in children presenting with febrile neutropenia found 25 studies reporting 14 different markers. Of these, CRP, PCT, IL6 and IL8 were most commonly examined. The finding of a diverse range of potentially useful markers, but such little consistency across studies, is unfortunately common in such research [40], and may reflect the relative lack of coordination in supportive care studies.

The studies presented similar challenges in reporting, methodology and analysis. Reporting if the test was interpreted 'blind' to the results of the outcome analysis, and vice-versa, was very poorly reported. Many studies failed to assess if the marker had supplementary value above the simple admission data collected by clinicians at every encounter: age, malignancy, temperature, vital statistics and blood count. Analysis of the data was frequently undertaken by episode, with no account of multiple admissions for the same patient. Such an analysis ignores the variation which may be expected from genetic polymorphisms for the production of the biomarker under investigation [39], or in individual genetic susceptibility to infection [41, 42]. The biomarker cut-off values reported were frequently derived from the dataset to which they were then applied, which is likely to produce significant overestimations of accuracy [43]. The data were sometimes presented as mean and standard deviation estimates, from which measures of test accuracy were derived. Although this may raise concerns because of the assumption of a normal distribution, there is some empiric justification for this procedure [44].

Quantitative meta-analysis using three approaches demonstrated how the commonly used, simple techniques may fail to reflect inconsistencies in the whole data set and so produce misleadingly precise results. The example of this review is important to recall when appraising other reviews where inconsistencies may not have been as extensively investigated.

The analysis undertaken using only the most commonly reported cut-off in a restricted number of studies produced excessively precise results which did not reflect the uncertainty of the whole data set, and so should be rejected. A similar problem was found with the use of data points with different thresholds to produce a hierarchical summary receiver operator curve (HSROC). The HSROC modelled by these techniques does not take into account the actual value of the thresholds. This is frequently reasonable: it is impossible to quantify the thresholds used by different radiologists to call a radiograph 'positive' for pneumonia. In cases where the values are known, an ordered relationship should be possible to determine, flowing from high to low cut-offs from left to right on the curve. This ordered relationship did not hold true for analyses of CRP and PCT and so should call into question analyses in other studies which do not assess whether thresholds vary according to the implicit structure of the model.

A previously developed [8] technique to undertake the ordered pooling of all the results was used to attempt to overcome these difficulties of only selective use of the data, and of incorrect relationships between test thresholds. This approach failed to produce meaningful results for the ability of PCT and CRP to identify patients who developed a documented infection, reflecting the inconsistencies and great heterogeneity of the data.

Some of the observed heterogeneity may be due to differences in measurement between apparently similar outcomes. While bacteremia is likely to be similarly reported across the studies, the diagnosis of a soft-tissue infection may vary between clinicians and centers. Very few studies reported in detail the exact definitions of the outcomes they reported. Further variation may have been introduced by the varying definitions of fever and neutropenia. In this review, 20 different combinations of criteria were used to define febrile neutropenia. These data could not be directly assessed to explore their relationship with the diagnostic value of the biomarkers, but as the depth of neutropenia and peak, and duration of temperature may affect the generation of biomarkers, the variation may further account for some of the heterogeneity. Additionally, although the assay techniques used in the studies were reported to be similar, there was no calibration of assays across the various studies. Other differences in the populations studied, such as the nature of the malignancies, recent surgical interventions and duration of therapy, may also add heterogeneity to interpreting markers which are themselves affected by a malignant disease. A more prosaic reason for heterogeneity may be publication bias: the tendency for reports demonstrating good predictive value to be published than those showing poor discrimination [45‐47].

In order to interpret the information from this review in a clinically meaningful way, both the estimates of predictive effectiveness and the uncertainty that surrounds these estimates need to be taken into account. CRP has been most extensively studied in this setting; it is a ubiquitous test and the only one which has been shown to add to the predictive ability of clinically-based decision rules [26, 34]. These studies chose two differing cut-offs (> 50 mg/dl [26] or > 90 mg/dl [34]). It is at best only moderately discriminatory in the setting of detecting documented infection (Sensitivity 0.65; 95% CI 0.41 to 0.84, Specificity 0.73; 95% CI 0.63 to 0.82), which is in keeping with estimates drawn from its value in the detection of serious bacterial infection in non-neutropenic children [48], and may be a significant overestimation of its value. The clinical role of CRP as a screening tool may be limited, however, if another biomarker is shown to be a more discriminatory test.

Data from this review and meta-analytic comparisons of CRP and PCT in the non-neutropenic population [49] are suggestive of the improved predictive value of PCT over CRP. This has a strong pathophysiological basis, as PCT levels are reported to rise within 3 to 4 hours in response to infection as compared with the 24 to 48 hours required for CRP [33]. However, the data for the improved predictive value of PCT are quite varied (see Additional file 3 and previously published reviews [13]). This may be related to the degree of neutropenia, as reports from the post-transplant setting have shown disappointingly poor discrimination [50], or this again may be due to small studies and publication bias [47, 51]. Based on the data from this review, procalcitonin cannot yet be recommended for use in routine clinical practice

Similar pathophysiological claims for improved predictive ability can be advanced for IL6 and IL8 [52]. In this review, IL6 level shows potential to be a better discriminator than CRP of those children who will develop a serious infectious complication. IL8 also appears to have moderate discriminatory ability and has been used in combination with clinical data in a small pilot study to withhold antibiotics to a highly select group of patients with febrile neutropenia [53]. Both of these cytokines show promise, and should be subject to further investigation.

Given the very limited data available for other potential biomarkers of infection in the setting of pediatric febrile neutropenia identified by this review, no strong clinical conclusions for their use can be reached without further studies.

These conclusions are drawn from an extensive and detailed systematic review of the available evidence using advanced techniques of meta-analysis, supplemented by rational clinical and pathophysiological reasoning. It should be clearly understood that they are uncertain and unstable, as only small amounts of new data may substantially alter these findings.

This review demonstrates flaws in our current understanding of the value of biomarkers in the prediction of adverse outcomes from episodes of febrile neutropenia, but also provides us with clear opportunities for development. All further investigation should estimate the additional value of biomarker measurements, beyond the discrimination already achieved by clinical variables. This should take into account key features of the treatment, for example, stem-cell transplantation and any clinically defined risk stratification already undertaken.

This includes the use of individual patient data (IPD) meta-analysis, which should allow the effective added-value of markers to be measured when the best clinical data have been taken into account in differing sub-groups. Such a venture is in progress [54]. The biomarkers IL6, IL8 and PCT appear promising, and should certainly be subject to new primary studies investigating more thoroughly the prediction of significant infectious morbidity, which includes both clearly defined infections and the sepsis syndrome, across a variety of clinical settings. By developing harmonized definitions of outcomes for such studies, greater confidence could be placed upon their results. The new SIOP Supportive Care group is ideally placed to lead on such a venture, and allow pediatric oncology/hematology to once more push the boundaries of international, collaborative clinical research.