Development and validation of a prognostic score during tuberculosis treatment

Tuberculosis (TB) is a major cause of death worldwide [1]. Estimates from the World Health Organisation (WHO) suggest that about 16% of the 9 million individuals who developed a TB in 2013 died of the disease, with 95% of these deaths occurring in low- and middle-income countries [2]. Determinants of death during treatment for TB have been largely characterised as recently reviewed by Wait and co-workers [3]. These determinants in developing countries with high endemicity for TB and high prevalence of HIV infection comprise essentially the co-infection TB/HIV, severe HIV related immune-depression, smear negative pulmonary tuberculosis (PTB−), under-nutrition and low socio-economic status [3]. In developed countries, determinants of mortality during TB treatment are mostly non-infectious comorbidities, smear positive pulmonary TB (PTB+), alcohol and drugs abuse [3].

In 2013, Horita and co-workers derived a risk score to predict mortality in Japanese aged on average 65 years, receiving in-hospital care for smear positive pulmonary tuberculosis [4]. However, their cohort did not include patients with HIV infection, and did not account for other clinical forms of tuberculosis (PTB− and extra-pulmonary TB). Furthermore, we are not aware of a prognostic score integrating all major clinical forms of tuberculosis, and that has used routinely collected variables in national TB programs. Healthcare practitioners need simple and accurate tools to identify at the start of treatment, those patients with TB who are at high risk of death during treatment, and whose outcome can be modified through tailored monitoring and management. Accordingly, the aim of the present study was to develop and validate a simple score for predicting the risk of death during follow-up in patients starting TB treatment in areas of high endemicity for TB, and high prevalence of HIV infection.

Data analysis used the R statistical software V.3.1.1 (10-07-2014) (The R Foundation for Statistical Computing, Vienna, Austria) and ‘rms’ and ‘pROC’ packages. Data are summarised as count and percentages for categorical variables and mean (or median) and standard deviation (or 25th–75th percentiles) for continuous variables. Groups’ comparisons used chi square test for categorical variables and Student’s t-test and Mann-Whitney U test for continuous variables. Logistic regressions models were used to derived the prediction models and were implemented with the lrm function, which fits binary logistic regression models using maximum likelihood estimation. Candidate variables were first tested for inclusion in univariable models, then the significant ones based on a p-value < 0.20 were all entered into the same multivariable models, and backward stepwise eliminations procedures used to retain the variables in the final models as recommended by Collett [7]. The linearity of the effects of continuous predictors was examined using restricted cubic splines. The performance of the derived model was tested on the development sample and in bootstrap resampling, which was based on 2000 replications. In the bootstrap internal validation based on 2000 replications, the original dataset was resampled at random with replacement 2000 times, with each random sample being of the same size (number of participants) as the original dataset. For each sample, the logistic regressions were fitted, the new beta coefficients estimated and a new equation derived using predictors in the final model. This new model was then tested on the dataset from which it was developed and as well as on the original dataset, and the differences in the performance on the derivation and original samples, were averaged across the 2000 replication to derive the optimism. Model’s discrimination was assessed by calculating the c-statistic in the derivation sample, and after correction for optimism in bootstrap internal validation. Model’s calibration was assessed graphically through calibration plots, and by computing the Hosmer and Lemeshow statistics (based on n-2 degrees of freedom) [8].

To facilitate the uptake of the model in clinical setting, a graphical calculator (nomogram) and a risk score table were constructed by summing up the weighted point-score from each predictor to arrive at a total score which can then be used to determine if the patient is at high risk. For this purpose, receiver-operating characteristic (ROC) curves were used to derive the optimal cut-off point score, applying the Youden’s index method [9]. Performance measures including the sensitivity, specificity, positive and negative predictive values where then estimated at this optimal threshold as well as other selected thresholds.

Based on the optimal point-score of 46.5 in the derivation sample, the four levels of risk described in Table 5 were established. Class I corresponds to low risk and class IV to critical risk. As expected death rate increased across risk classes, being 1.6, 6.8, 15 and 36.5% respectively for Class I, II, III and IV.

We have derived a model to predict fatal outcomes in patients starting treatment for tuberculosis in settings of high endemicity for tuberculosis and HIV infection. The derived model used easily assessable variables in routine clinical practice. These predictors were age, clinical form of tuberculosis, body mass index and status for HIV infection. The model developed and the derived point-scoring system with these predictors had near perfect calibration and good discrimination, with only marginal variation in bootstrap internal validation.

Predictors retained in our final model were essentially those previously known to be associated with mortality risk in tuberculosis [10‐21]. In both developed and developing countries, advancing age for instance has been widely reported to be associated with higher mortality risk in patients with tuberculosis. Possible reasons for this association include late diagnosis of TB at the advanced stage of the disease in older people, unfavourable living conditions, and comorbidities. Lower BMI is a potent marker of under-nutrition, a well-known determinant of mortality risk in patients with tuberculosis. Various markers of under-nutrition have been associated with excess mortality in patients with TB, including low BMI or body weight and hypoalbuminemia [4], and a composite score of under-nutrition comprising low BMI, hypoalbuminemia, hypocholesterolemia, and lymphopenia [22].

Available studies have also linked extra-pulmonary TB with higher risk of mortality [17], with the magnitude of such risk being two to five times higher when compared with mortality from smear positive TB. Likewise, smear negative TB is associated with two to three times higher mortality rates, relative to smear positive TB [17]. Co-infection with HIV has also been established as a powerful determinant of death in patients with TB, with the magnitude ranging from two to nearly eight folds when compared to HIV-free patients [13, 14, 17, 23‐27].

We did not test the radiographic predictors. It should however be recalled that radiographic features of TB tend to be correlated with indicators of TB severity such as low BMI [28, 29]. Therefore, loss of discriminatory information subsequent to not including radiographic predictors in our model, if any, is likely marginal.

The final model developed in the current study was based on four predictors that are routinely assessed in most tuberculosis control programs in high endemicity areas. Interestingly, data on those predictors can be accurately collected event by lay people, meaning that the model will be applicable even in remote and underequipped primary care settings. This model had excellent calibration and good discrimination both in the derivation sample and in bootstrap internal validation. The observed discriminatory power of the model (AUC ≥ 0.80) suggest that any drop in this performance property of the model when applied to other settings, will need to be sizable to push the model below the unacceptable zone (AUC < 0.70); which is very unlikely. However, due to the anticipated difference in death incidence across settings, it is an expectation that the model may require some adjustment where appropriate in order to maintain optimal calibration properties [30]. There are practical examples to support that this adjustment is easily achieved through simple intercept recalibration [31].

Our receiver operating characteristic curves analysis identified the absolute risk threshold of around 5% as estimated by the derived model, to be the optimal threshold to identify at the start of treatments, those patients with TB who are at increased risk of death during follow-up. In TB patients scoring above this threshold, active investigation and timely management of modifiable co-factors know to be associated with mortality risk has a potential to improve survival among patients starting TB treatment. Such co-factors include and are not restricted to under-nutrition, anaemia and opportunistic diseases. Considering that TB tend to predominantly affect socially deprived people [32, 33], early nutritional support should be an integral component of TB treatment programs. Similarly, early introduction of antiretroviral therapy has been shown to reduce the risk of death by about 68% in TB patients co-infected with HIV [34].

A sizable proportion of potentially eligible participants had missing data on the outcome of candidate predictors of interest and was excluded. While data imputation technics could be used to fill the missingness and possibly using data from all eligible participants, implementing some steps of predictive models development on imputed data is not straightforward [35]. However, the similarity of profiles between excluded participants and those in the final analytic sample suggest that the derived estimates are likely unbiased. By using bootstrap resampling for internal validation, we were able to use all participants with valid data for model development, therefore achieving a very high number of outcomes per candidate variable tested, and accordingly a very good statistical power [35]. In the absence of individual level data on height, the adjusted BMI was used as predictor in our model. By constraining the height to assume the same gender-specific value in all participants, we have possibly reduced to some extent the variability in the true BMI across participants, which in turn could lead to non-optimal extraction of predictive information from BMI. However the overall good discriminatory power achieved using this imperfect BMI suggests that further gain from using the true BMI will likely be marginal. The non-appreciable difference in discriminatory powers between our final model and a model accounting for possible non-linearity of the effect of age and BMI through restricted cubic splines, confirms that essential discriminatory information from the two predictors are captured by the linear forms. This supports our choice of a simplified final model assuming a linear functional form of the two predictors.

We have developed a simple risk score based on four routinely collected variables in TB program, to estimate the chance of death in patients commencing TB treatment in high endemicity areas for TB and HIV infection. This model had good performance in the derivation sample and during internal validation using bootstrap resampling methods. Successful application of this model in routine TB care settings can assist the identification of a group of patients whose risk of fatal outcome can potentially be modify through identification and correction of co-factors of mortality in TB. To promote the uptake of the model in routine setting, we have developed a graphical calculator in the form of a nomogram as well as a point scoring system to allow risk estimation of the model without manipulating complex formula. The model is hereafter called CABI, reflecting the four variables in the models: C for ‘clinical form of tuberculosis’, A for ‘age’, B for ‘body mass index’ and I for ‘status for HIV infection’. Just like any newly developed model, results from this study are just preliminary findings. CABI will require independent external validation to establish the performance both in the study setting and in other settings including African, Latin America and Asian populations [36], before recommendation for uptake in routine clinical setting.