Development of diagnostic algorithm using machine learning for distinguishing between active tuberculosis and latent tuberculosis infection

The current study was carried out from January 2018 to January 2022. The study participants in discovery cohort were recruited at Tongji Hospital (the largest tertiary hospital in central China with 5500 beds). The study participants in validation cohort were enrolled at Sino-French New City Hospital (a branch hospital of Tongji Hospital with 1600 beds). Participants in two cohorts were included based on positive T-SPOT results. Participants were classified as ATB patients and LTBI individuals on the grounds of clinical and laboratory evaluation. ATB was diagnosed by positive Mtb culture and/or GeneXpert MTB/RIF for the allocated samples including bronchoalveolar lavage fluid and sputum. LTBI was defined by positive T-SPOT result without symptomatic, radiological or microbiological evidences of ATB as well as the history of TB. Specially, the symptoms compatible of ATB in the current study included prolonged cough, chest pain, fever, and night sweats. Patients with the following condition were excluded from the study: (1) having anti-TB treatment within 1 month prior to the enrollment; (2) being younger than 18 years old. This study was approved by the ethics committee of Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology.

Heparin anticoagulated peripheral blood was collected for T-SPOT assay (Oxford Immunotec, Oxford, UK). Briefly, the isolated peripheral blood mononuclear cells (PBMCs) (2.5 × 10⁵) were added to 96-well plates precoated with anti-IFN-γ antibody. Four wells were prepared for the test: medium, early secreted antigenic target 6 (ESAT-6), and culture filtrate protein 10 (CFP-10), phytohemagglutinin (PHA). Plates were incubated for 16–20 h at 37 °C with 5% CO₂ and developed using an anti-IFN-γ antibody conjugate and substrate to detect the presence of secreted IFN-γ. Spot-forming cell (SFC) in each well was counted by ELISPOT reader (CTL Analyzers, Cleveland, OH, USA). The result was regarded as positive when ESAT-6 minus medium or CFP-10 minus medium ≥ 6. The result was regarded as negative if both ESAT-6 minus medium and CFP-10 minus medium ≤ 5. The result was considered as undetermined when the spot number in PHA well was < 20 or spot number in medium well was > 10.

Heparinized peripheral blood was collected for the measurement of lymphocyte subset number and lymphocyte IFN-γ secretion ability. The numbers of CD4⁺ T cells, CD8⁺ T cells, NK cells, and B cells were determined by using TruCOUNT tubes and BD lymphocyte subset reagent kit (BD Biosciences, San Jose, CA, USA). A volume of 50 µL peripheral blood was labeled with antibody cocktail for 20 min in room temperature. After adding 450 µL of FACS lysing solution, samples were analyzed with FACSCanto flow cytometer. TruCOUNT beads were gated based on side scatter and fluorescence intensity. CD3⁺CD4⁺CD8⁻ and CD3⁺CD4⁻CD8⁺ cells were respectively defined as CD4⁺ T cells and CD8⁺ T cells. CD16⁺CD56⁺ cells and CD19⁺ cells in CD3⁻ cells were respectively defined as NK cells and B cells. Lymphocyte IFN-γ secretion ability detection was performed under phorbol-12-myristate-13-acetate/Ionomycin/ionomycin (PMA/Ionomycin) stimulation as described in previous study [22]. The procedure was as the following: (1) 100 µL peripheral blood was diluted with 400 µL of IMDM medium (Gibco, Grand Island, NY, USA); (2) the diluted peripheral blood was incubated in the presence of Leukocyte Activation Cocktail (Becton Dickinson GolgiPlug™) for 4 h; (3) the cells were labeled with antibodies including anti-CD45, anti-CD3, anti-CD4, anti-CD8, and anti-CD56 for 20 min at room temperature; (4) the cells were fixed and permeabilized; (5) the cells were stained with intracellular anti-IFN-γ antibody; and (6) the cells were analyzed with FACSCanto flow cytometer. The percentages of IFN-γ⁺ cells in cell subsets were defined as IFN-γ secretion ability of them. Specially, the percentage of IFN-γ⁺ cells in CD3⁺CD4⁺CD8⁻ cells was regarded as CD4⁺ T cell IFN-γ secretion ability; the percentage of IFN-γ⁺ cells in CD3⁺CD4⁻CD8⁺ cells was regarded as CD8⁺ T cell IFN-γ secretion ability; the percentage of IFN-γ⁺ cells in CD3⁻CD56⁺ cells was regarded as NK cell IFN-γ secretion ability.

Diagnostic models were established using machine learning by the R package “mlr3” and related packages. Multiple data acquired from study participants in discovery cohort was randomly divided at a 3:1 ratio. The large one (3/4) was utilized for modelling (training set), whereas the small one (1/4) was applied as test set. The models established in discovery cohort were further verified using an independent cohort (validation set). Machine learning learners used were generated using R packages “mlr3”, “mlr3learners”, and “mlr3extralearners”. The probability ranging between 0 and 1 for ATB diagnosis for each case was obtained by the prediction of the model. The performance of models was evaluated by measures involved in R package “mlr3”. The importance of indicators in the contribution to the model was also evaluated.

Continuous variables were represented as mean ± standard deviation (SD) or medians. Categorical variables were expressed as number (%). Student’s t test and Mann–Whitney U test were applied for the comparison of continuous variables. Chi-square test and Fisher’s exact test were used for the comparison of categorical variables. P < 0.05 represented that statistical difference existed. Cor linear regression was performed to evaluate whether there is a linear correlation between various indicators. Tree-leaf clustering, principal components analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and uniform manifold approximation and projection (UMAP) were utilized to visualize the differentiation of multiple results. Receiver operating characteristic (ROC) curves were created to evaluate the performance of various indicators and models for discriminating ATB from LTBI. Area under the ROC curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR), as well as accuracy, together with their 95% confidence intervals (CI), were calculated. The comparison between AUCs was achieved by DeLong’s test [23]. The tools involved in data analysis and graphing throughout the study included R 4.0.2 program (R Core Team), GraphPad Prism Software 6.0 (GraphPad Software, Inc, San Diego, CA, USA), Java (TM) SE Development Kit 11.0.14 (Oracle), SPSS Software 25.0 (Social Sciences Inc, Chicago, Illinois, USA), and MedCalc 11.6 (MedCalc, Mariakerke, Belgium).

A total of 468 patients with ATB and 424 individuals with LTBI were recruited in discovery cohort, while 125 patients with ATB and 138 individuals with LTBI were enrolled in validation cohort (Table 1). There is a preponderance of male cases in both ATB group and LTBI group. Diabetes mellitus is the major underlying disease in both two groups. There was no significant difference in the age and sex distribution between ATB group and LTBI group in both discovery and validation cohorts.

Most indicators showed significant differences between ATB patients and LTBI individuals. It was observed that the levels of ESAT-6 SFC, CFP-10 SFC, WBC, NEUT, RDW_CV, RDW_SD, GLB, TG, K, APTT, PT, FIB, D_D, ESR, and HsCRP were significantly higher in ATB patients than those in LTBI individuals (Fig. 1A). On the contrary, the levels of CD4 cell number, CD8 cell number, NK cell number, B cell number, CD4 cell IFN-γ secretion, CD8 cell IFN-γ secretion, NK cell IFN-γ secretion, LYMPH, EO, BASO, RBC, HGB, HCT, ALB, T_CHOL, Cl, Ca, Na, and TT were significantly lower in ATB patients than those in LTBI individuals (Fig. 1A). There was no statistical difference in the levels of MONO, PLT, P_LCR, PCT, PDW, TP, P, and Mg between ATB patients and LTBI individuals. The capability of individual indicator to distinguish ATB patients from LTBI individuals was determined using ROC curve analysis. It was found that the AUCs of 8 indicators were more than 0.7, while the AUCs of the remaining 34 indicators were under 0.7 (Fig. 1B, C). Specially, CFP-10 SFC, HsCRP, ESAT-6 SFC, D_D, ESR, CD4 cell IFN-γ secretion, CD4 cell number, and HGB were the most accurate biomarkers in differentiating ATB from LTBI (Fig. 1B, C).

Given the fact that the combination of various biomarkers has shown better performance than single biomarker in TB diagnostic field, we attempted to investigate the diagnostic potential of the combination of multiple indicators using machine learning. Cluster analysis and dimension reduction were applied to evaluate the distribution of ATB patients and LTBI individuals based on various indicators. leaf clustering advocated the possibility of the combination of these indicators for the discrimination between ATB and LTBI (Fig. 2A). We further conducted dimension reduction. Consistent with leaf clustering, dimension reduction performed by PCA, tSNE and UMAP analysis also corroborated that the multiple data had the potential to segregate ATB from LTBI (Fig. 2B–D).

Based on the synergistic effects of various indicators denoted by the above findings, 28 diagnostic models were successfully established using machine learning in accordance with laboratory data. ROC curve analysis was performed and the results demonstrated that most established models could successfully differentiate ATB from LTBI with AUCs more than 0.9. Among them, conditional random forests (cforest) model performed better in comparison to other models. Cforest model is an implementation of the random forest and bagging ensemble algorithms utilizing conditional inference trees as base learners. The cforest algorithm could use multiple decision trees to achieve a robust prediction. The model could also avoid overfitting issue since it takes the average and cancels out the biases. ROC curve analysis provided an AUC of 0.995 (95% CI 0.991–0.998) for cforest model in training set (n = 669, 347 ATB and 322 LTBI). The cutoff of 0.5 rendered 98.85% (95% CI 97.07–99.55%) sensitivity and 95.65% (95% CI 92.84–97.39%) specificity. Meanwhile, cforest model differentiated ATB from LTBI with an AUC of 0.978 (95% CI 0.962–0.993) corresponding to a sensitivity of 93.39% (95% CI 87.50–96.61%) and specificity of 91.18% (95% CI 84.08–95.29%) in test set (n = 223, 121 ATB and 102 LTBI) (Table 2, Fig. 3A). CFP-10 SFC, ESAT-6 SFC, HCT, CD4 cell IFN-γ secretion, FIB, CD8 cell IFN-γ secretion, and CD4 cell number were the indicators with the highest contribution to cforest model (Fig. 3A). Among these parameters, CFP-10 SFC and ESAT-6 SFC indicated the specific response of the host against Mtb. In addition, CD4 cell IFN-γ secretion, CD8 cell IFN-γ secretion, and CD4 cell number indicated the global adaptive immunity of the host. Apart from cforest model, other models also showed effective discriminatory value. For example, the sensitivity and specificity of bart model in test set were 89.26% (95% CI 82.48–93.61%) and 90.20% (95% CI 82.89–94.59%), respectively (Table 2, Fig. 3B). Gamboost model distinguished patients with ATB from those with LTBI with an AUC of 0.969 (95% CI 0.949–0.988) and demonstrated a sensitivity of 85.12% (95% CI 77.71–90.38%) and specificity of 92.16% (95% CI 85.28–95.97%) in test set (Table 2, Fig. 3C). Besides, the sensitivity and specificity for gbm model in test set were 87.60% (95% CI 80.55–92.34%) and 88.24% (95% CI 80.55–93.14%), with an AUC of 0.968 (95% CI 0.949–0.987) (Table 2, Fig. 3D). Meanwhile, log_reg model established based on logistic regression had an AUC of 0.951 (95% CI 0.924–0.978) for discriminating ATB patients from LTBI individuals in test set (Table 2, Fig. 3G). The sensitivity and specificity of log_reg model were 87.60% (95% CI 80.55–92.34%) and 93.14% (95% CI 86.51–96.64%), respectively (Table 2, Fig. 3G). The AUCs of ROC curves of various models for ATB versus LTBI were presented in Fig. 3. The performance parameters for all models in training set and test set were shown in Fig. 5A, B and Additional file 1: Fig. S1A-B.

An independent validation is indispensable for determining the robustness of a developed model based on machine learning. Therefore, another cohort (validation set) was included for the purpose in the study. Consistent with the observation in discovery cohort, cforest model exhibited significant discriminatory ability in validation cohort. Cforest model presented an AUC of 0.963 (95% CI 0.940–0.986) in validation set, with a sensitivity of 92.80% (95% CI 86.88–96.17%) and specificity of 89.86% (95% CI 83.69–93.86%) (Table 3, Fig. 4A). The utility of other models was summarized in Figs. 4, 5C, and Additional file 1: Fig. S1C.