An exploration of mortality risk factors in non-severe pneumonia in children using clinical data from Kenya

The reporting of this observational study follows the Strengthening of reporting of observational studies in epidemiology (STROBE) statement [10], which is a set of recommendations for the reporting of observational studies in epidemiology (cohort, case-control studies and cross-sectional studies) [10].

This study was approved by the Scientific and Ethics Review Unit of the Kenya Medical Research Institute (KEMRI). Additionally, it was approved by the Ministry of Health, with the Medical Superintendents of participant hospitals giving consent for participation. Individual consent for access to de-identified patient data was not required.

This study was a retrospective cohort study at 14 public hospitals in Kenya with each having at least 1000 annual paediatric admissions, purposefully selected to represent two main regional groupings based on high or low malaria prevalence. The study was embedded within a collaborative health information network developed to help improve outcomes of care, accelerate knowledge discovery and advance cross-domain development of digital architecture in support of research in a low-income setting. This Clinical Information Network (CIN) is described in detail elsewhere [11].

All paediatric inpatient children admitted to the selected hospitals from 1 February 2014 to 28 February 2016, aged 2–59 months who had non-severe pneumonia at admission were eligible for inclusion in this study. This was determined from clinician diagnosis and clinical signs documented in patient records. To avoid confusion, we have used the term pneumonia to refer to children with a documented clinical diagnosis of pneumonia, and the terms non-severe and severe pneumonia to refer to those for whom WHO, under the 2013 revised definitions, recommends outpatient and inpatient care, respectively. The ideally diagnosed and managed population consisted of patients with a clinician-assigned admission diagnosis of non-severe pneumonia, who, as based on WHO guidelines, had clinical signs supporting this diagnosis and were treated with penicillin monotherapy. We excluded children born before the introduction of the pneumococcal conjugate vaccine to the national childhood immunisation schedule in January 2011; thus, the study population included children who were born after the introduction of both the pneumococcal and Haemophilus influenzae type B (Hib) conjugate vaccines (introduced nationally in 2001). Comprehensive data collected for these admissions comprised clinical, investigation and treatment data focused on admission and discharge events, with up to 350 variables per patient encounter collected. These variables span different disease conditions. A detailed description of the methods of data collection and analysis is reported elsewhere [11]. For our study, 37 variables were used: 18 variables were used in pneumonia classification criteria and in identifying the analysis population, and 19 variables were used for subsequent statistical analysis — 6 of which were interaction terms (variables used to test whether the effect of one independent variable differs depending on the level of another independent variable). In brief, hospitals implemented two data collection tools (a paediatric admission record and a discharge form) with one clerical assistant posted to each hospital to collect data from the medical records and laboratory reports. Data collection was conducted as soon as possible after discharge through abstracting data from inpatient paper records into a non-propriety electronic tool, Research Electronic Data Capture (REDCap) [12]. Data quality reports were generated by R scripts [13] based on validation rules and metadata pulled from REDCap’s application programming interface. These reports were fed back to the hospitals to improve the quality of clinical data used in this research. We have reported in detail elsewhere the process by which we established a clinical information network in Kenya, the multiple unique challenges we faced including the development of new data collection procedures and new methods to implement the provision of accurate reporting to hospitals [11].

Our prognostic models focused on paediatric inpatient hospital mortality, described by a binary variable (dead or alive). Predictors were identified a priori guided by clinical expert opinion and literature review. We selected variables posited to be associated with mortality and which could also be widely ascertained in low-resource clinical settings. To denote nutritional status, we used recorded weight and age to retrospectively compute weight-for-age Z-scores (WAZ) using WHO child growth standards [14], as data for these two variables were complete for the majority of patients studied. This resulted in the following predictors being selected, covering demographics and clinical characteristics: Age < 12 months (binary), Sex-Female (binary), Respiratory rate ≥ 70 breaths/min (binary), Temperature ≥ 39 °C (binary), Weight-for-age Z-score (ordinal — 3 levels), Dehydration status (ordinal — 3 levels), Pallor (ordinal — 3 levels), Malaria status (ordinal — 3 levels), Presence of ≥ 1 comorbidity (binary), Hospital in malaria endemic area (binary), Acute nutrition status (binary). Table 1 provides description of levels of the ordinal variables. Severity of pneumonia was categorised based on documented WHO clinical criteria (non-severe vs severe) [15]. Dummy binary variables were created for all levels in ordinal predictors. All predictors were assessed at the time of admission.

Patients with non-severe pneumonia with a diagnosis of either (1) severe dehydration or (2) severe malaria were recoded to severe pneumonia, since either of these diagnoses would render the respective patients’ ineligible for outpatient care. All cases of severe pneumonia were excluded from the analysis. Pneumonia cases with additional admission diagnosis of meningitis, acute malnutrition and shock were also excluded from the study sample; these conditions follow alternative management protocols under the clinical guidelines [16].

Data manipulation and statistical analyses were performed using R software [13] employing the caret package [17]. Categorical data were tabulated and summarised as proportions, while continuous variables were reported with medians and interquartile ranges as appropriate. To evaluate differences in the risk profile between the two groups of inpatient mortality outcomes, and identify predictors that substantively account for these differences, an adjusted multivariable logistic regression model was used.

In previous research, logistic regression modelling has been used to look at risk factors in pneumonia (Ambrose Agweyu, et al., Appropriateness of clinical severity classification of new World Health Organization (WHO) childhood pneumonia guidance: a multi-hospital retrospective cohort study. The Lancet Global Health, under review). However, due to the violation of assumptions of independence of predictors (e.g. age variable would be collinear with Weight-for-age Z-score variable, etc.) and the limited understanding of the relationship of the predictors with mortality in non-severe pneumonia, this approach is susceptible to incorrect inferences about relationships between explanatory and response variables. Additionally, apart from model coefficients and significance tests, logistic regression models offer limited guidance on feature selection from model outputs that can guide future intervention design. Feature selection is defined and explained further in Additional file 2: Table S1 and in published reports [18, 19]. Therefore, the magnitude of coefficients included in the traditional adjusted logistic regression models might not be good indicators of clinical value of features, since they do not incorporate clinical consequences involved in targeting those features [20, 21].

To address these challenges in generating decision-analytic solutions from logistic models, machine learning techniques were used. These techniques were also explored to test whether, given the available data, models using complex adaptive techniques perform better in determining the inpatient mortality risk associated with non-severe pneumonia given the choice of predictors. The use of these techniques would also provide implicit feature selection as part of the model output, in addition to allowing us to evaluate whether there was consistency of findings given the different model choice. The machine learning models used in analysis were partial least squares - discriminant analysis (PLS-DA) [22], random forests (RFs) [23], support vector machines (SVMs) [24] and elastic nets [25]. Brief descriptions of these models are given in Additional file 2: Table S2. Detailed descriptions of the models are provided in the referenced works. Here we offer an introduction to the techniques used, which may be less familiar.

Model validation was checked by employing a 10-fold internal cross validation on two thirds of the data. The remaining one third of the data was used as the validation set. This is further explained in Additional file 2: Table S1. Variable importance scores, which would guide feature selection, were generated to identify predictor contribution to classification, with higher scores considered more relevant in classification. Detailed explanations of variable importance estimation for the models included in the analysis are reported elsewhere [26]. The selection of critical parameters for each of these modelling techniques was auto-determined by the R caret train function by choosing the tuning parameters that produced the highest values of receiver operating characteristic (ROC) curves where a grid search cross-validation was applied. These parameters are provided in Additional file 2: Table S2.

To evaluate the clinical impact of implementing the models in practice as part of screening algorithms, we performed decision curve analysis, evaluating how different threshold probabilities vary the false-positive and false-negative rate expressed in terms of net benefit [27]. The unit of net benefit is true positives, and the details of its calculation are extensively reported elsewhere [20]. When carrying out a head-to-head comparison of different prediction models on the same population, the interpretation is straightforward — at each clinically relevant probability threshold, the model that has the highest net benefit is preferred. Models are also compared to the extreme choices of designating admitting all and no patients at high risk of inpatient mortality.

Across the machine learning techniques used, Age < 12 months, Respiratory rate ≥ 70 breaths/min, Presence of comorbidities, Female sex and Very low WAZ consistently featured among the topmost five features explaining variability in inpatient mortality (see Table 3). It is noteworthy that the sizes of the odds ratio from Table 2 do not correspond with the order of feature importance in Table 3 below — a common assumption made when generating risk scores from logistic models. The reason is that the estimation of the contribution of each variable to the logistic model is determined by its corresponding t statistic absolute value. Detailed explanations of variable importance estimation for each of the models included in the analysis are reported elsewhere [26] and demonstrated in the analysis codes provided in Additional file 4.

The performance as represented by AUC score was consistent across the selected models (Fig. 2), ranging from 0.725 to 0.796. These models were all found to be moderately discriminative for inpatient mortality risk. Using the full dataset with imputation in modelling, logistic and random forest models demonstrated much higher prediction accuracy (>0.9) than the other models but had low sensitivity (see Additional file 2: Table S4).

Overall, while all models had a higher AUC score than traditional logistic regression models (Fig. 3), from DeLong’s significance test, these differences were not statistically significant apart from the random forest model (see Additional file 5). However, the PLS-DA model demonstrated the highest sensitivity for inpatient mortality when using imputation and was least influenced when only complete cases were used. PLS-DA was therefore used for subsequent sensitivity analysis — this is consistent with the pragmatic perspective emphasising sensitivity over specificity across pneumonia to allow for optimisation of public health benefits [36]. For purposes of feature selection, the average feature rank across all models in Table 3 was used as a guide. From the table, Respiratory rate ≥ 70 breaths/min, Age < 12 months and Very low WAZ were the three features targeted for subsequent decision analysis due to their high average ranks.

To evaluate the clinical impact of using the selected features in decisions to admit non-severe pneumonia patients in routine practice as part of screening algorithms, we performed decision curve analysis. The results of this analysis are presented in Fig. 3. The current decision to admit is based on whether the patient has a diagnosis of severe pneumonia — which is associated with a high mortality risk. We included an overlay of a model of severe pneumonia to help compare the net benefit of the decision to admit using the top three clinical and patient features identified compared to the current strategy (admit only severe pneumonia). The models that performed best are those to the extreme right of the figure.

Those to the right of the current admission criteria (severe pneumonia) are modelled on non-severe pneumonia patients based on WHO criteria. The threshold probability associated with the highest predicted inpatient mortality risk varied from 2 to 14%. At predicted probability thresholds between 2 and 7%, severe pneumonia model’s net benefit was greater than all other models and greater than strategies labelling all patients at high risk (grey line) or none at high risk (black line). For predicted probability thresholds between 7 and 14%, the net benefit of a model for non-severe pneumonia patients less than 12 months old with respiratory rate ≥ 70 breaths/min and very low WAZ was greater than that for all other models and greater than strategies labelling all patients at high risk (grey line) or none at high risk (black line). This is illustrated in Fig. 3. In summary, for a probability threshold of inpatient mortality between 7 and 14%, there is a net benefit for admitting non-severe cases of pneumonia involving children less than 12 months old with respiratory rate ≥ 70 breaths/min and with very low WAZ.

Given that the PLS-DA model had an AUC score higher than the logistic model (Fig. 2) and a sensitivity greater than all other models when considering its performance in both imputed and complete case analysis (see Additional file 2: Table S4), we used it for comparison of variable importance across the alternative criteria for pneumonia classification. This is guided by evidence from published studies showing AUC to be statistically consistent and a more discriminating measure than accuracy [37]. From Fig. 4, when WHO guidelines are used to determine the severity of pneumonia cases, the predictors that best explain the variance of inpatient mortality outcome among children with non-severe pneumonia are age, respiratory rate, comorbidities and very low WAZ.

This slightly differs from clinician admission diagnosis results where the presence of comorbidity is ranked higher than respiratory rate. In the non-severe pneumonia sub-population defined by prescription of penicillin monotherapy at admission criteria and also in the ideally diagnosed and managed sub-population, WAZ and age do not rank highly as risk factors. This might be indicative that in these sub-populations, very low WAZ and age explained very little variation in inpatient mortality.

From the results comparing whether risk factors were strongly linked to the pneumonia classification criteria, clinician classification criteria had the best performance in predicting inpatient mortality outcomes (see Additional file 2: Table S5). All pneumonia classification criteria, apart from that of the ideal population, had moderately acceptable performance with AUC > 0.7 (Additional file 2: Table S5). However, a comparison of the WHO classification criteria to each of the other pneumonia classification criteria revealed no statistically significant differences in AUC values (see Additional file 5). This is suggestive that differences in pneumonia classification criteria might not have a statistically significant impact in determining risk factors for this population.

This study investigated the clinical and patient attributes that best discriminated the risk of inpatient mortality in children with non-severe pneumonia aged 2–59 months and the net clinical benefit of targeting these features using routine hospital data from a clinical information network running in 14 hospitals across Kenya. Several machine learning techniques were applied for discriminating these risks to test whether, given the available data, models using complex adaptive techniques performed better or provided robust ways to make decisions from model outputs, and also to see if there was consistency of findings across the alternative model choices.

Our analysis revealed age, respiratory rate and very low WAZ to be moderately discriminative of inpatient mortality in non-severe pneumonia cases across all models. Looking at the clinical consequences associated with features identified, the decision to admit these patients based on the characteristics identified was predicted to have a substantively higher net benefit on inpatient mortality outcomes for risk probability thresholds between 7 and 14% compared to using the existing WHO criteria for severe pneumonia only.

Other studies conducted in similar epidemiological and geographical contexts have found comorbidities to be significant predictors of mortality in pneumonia patients, with odds being three times that of non-comorbid patients [38‐42]. However, the study populations have consisted of both severe and non-severe pneumonia cases considered as a single population [38] or have omitted patients with any comorbidity [43].

Methodologically, our approach goes beyond the use of effect sizes for the determination of risk factors coupled with expert opinion [44]. In addition to traditional logistic regression models, we also applied robust machine learning techniques posited to improve modelling results by offering the ability to rank the importance of clinical signs and patient characteristics as risk factors for pneumonia mortality and also evaluate the consequence of using them in clinical decision making. These approaches have previously been used to predict risk of pneumonia and mortality, although not using a similar patient cohort or in a similar geographic context [45, 46].

While traditional statistical modelling techniques are simpler to implement and offer easier interpretation of results, machine learning techniques — which are adaptive to the datasets they are applied to and tend to perform better given their complex estimation procedures — are growing in popularity. However, these machine learning approaches produce results that are often regarded as difficult to interpret and operationalise due to their “black-box” nature [47]. It is becoming more difficult to decide which modelling approach to use, given the growing volume and veracity of clinical and epidemiological data. This is evident in our findings, where in predicting inpatient mortality, all models had a higher AUC and sensitivity compared to the traditional logistic regression model, with the exception of random forest’s sensitivity performance. Our approach was consistent with the pragmatic perspective adopted by WHO that emphasises sensitivity over specificity across pneumonia and other case definitions in this region in order to optimise public health benefits [36]. This ought to be — and was — factored into our choice of model.

The WHO guidelines recommend oxygen therapy for children with respiratory rates ≥ 70 breaths/min [48]. While this guidance implicitly suggests the need for admission care, this clinical sign is not listed among the classification criteria for severe pneumonia. In this study, respiratory rate ≥ 70 breaths/min was independently associated with increased inpatient mortality, and there was a net benefit of clinicians’ decision to admit non-severe pneumonia cases with tachypnoea at this threshold. The introduction of point-of-care diagnostic tools to objectively assess respiratory rates in routine care settings may mitigate the challenges of reliable measurement and the potential for misclassification of pneumonia severity on the basis of this sign [49].

The presence of pallor is associated with high risk of inpatient mortality in children with severe pneumonia [50] and, from our findings, in non-severe pneumonia also. This might be attributed to similarities in clinical presentation for malaria and severe pneumonia in children [51, 52], presenting challenges in discrimination of the two where confirmatory microbiological tools are unavailable. The findings of this study demonstrating increased risk of inpatient mortality in children presenting with any comorbidities are particularly relevant in SSA where comorbidities in non-severe pneumonia are high [53]. The adoption of treatment guidelines from controlled studies that fail to factor in common local comorbidities in children may be inappropriate in high mortality settings [54, 55].

Among demographic characteristics, female sex and age younger than 12 months were independently associated with increased odds of inpatient mortality. These risk factors were also highlighted in a recent review of 77 studies conducted in low- and middle-income countries [56]. While the risk associated with young age may be attributed to the limited capacity of the developing immune system to withstand severe infections [57], higher mortality among girls is less clear. Gender inequities in care seeking have been posited to play a role [58, 59]; however, the evidence to support this theory remains weak, warranting further study.

Despite having excluded children with documented admission diagnoses of severe acute malnutrition from the study population, analyses based on WAZ computed using available data on weight and age revealed increased mortality among children with low (–2 to –3SD) and very low (< –3SD) WAZ. These findings, consistent with the results of a large systematic review, challenge a recent technical update to the WHO guidelines for the management of severe acute malnutrition, which now recommends the use of weight-for-height Z-score and mid-upper arm circumference in place of WAZ for the diagnosis of severe malnutrition [60].

The large sample size drawn from the hospitals across the country resulted in precise estimates that are representative of the population of children hospitalised with pneumonia in a majority of hospitals in Kenya. The 2-year data collection period further strengthened representativeness by eliminating seasonal bias — an important consideration in studies on acute respiratory infections in children [61].

Our data were limited by the clinical information network’s reliance on documented diagnoses in the absence of diagnostic tests. However, the approach adopted for data collection remains the only realistic data source that reflects how routine paediatric care is delivered at scale in Kenya. To mitigate this limitation, considerable efforts to improve data quality have been made, including follow-up in the laboratory to confirm whether there was evidence that investigations were performed and their results. Additionally, the data used reflect the routine practice of many clinicians in the Kenyan setting, thereby increasing the generalisability of our findings.

We restricted our analysis to children who were born during the period after the introduction of the pneumococcal vaccine to provide an understanding of the risk factors for pneumonia mortality in the current era. A consequence of this was the exclusion of older children who had not received the vaccine in the early period of the study and a relatively larger proportion of younger children with a higher risk of death.

Finally, the study sites were part of an ongoing implementation science project designed to improve quality of documentation practices and utilisation of data to improve care of children in district-level hospitals in Kenya. The population studied may therefore not be representative of children presenting to lower level health facilities or in the community where the influence of limited staff and resources and care-seeking behaviour may all influence clinical presentation and prognosis of pneumonia.