Development and validation of a simple tool composed of items on dyspnea, respiration rates, and C-reactive protein for pneumonia prediction among acute febrile respiratory illness patients in primary care settings

This is a cohort study that was planned as a two-stage program. The first stage was to develop a simple multivariable model for pneumonia prediction in AFRI patients, and all the works were conducted in the fever clinics of Shanghai General Hospital. The second stage was to validate the model in a bigger population and with the participation of more primary care settings from different areas of China. A checklist from Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guideline was completed [7].

Patients who visited fever clinics in the two campuses (north campus in the urban area and south campus in the suburban area) of Shanghai General Hospital between January 22 and February 6, 2020, were enrolled in the first stage of this study. Fever clinics are primary care settings that are used to treat and triage fever patients in China. The inclusion criteria for this study were as follows: (1) patients were ≥ 18 years of age, and presented with respiratory symptoms, including running nose, nasal congestion, sore throat, cough, purulent sputum, and dyspnea; (2) body temperature ≥ 37.3℃ by ear thermometer for 2 times; (3) blood routine test and CRP were taken during the clinic visit. The biomarker procalcitonin wasn’t chosen in the study due to its unavailability for rapid measurement. Pregnant patients were excluded from the study.

Physicians, wearing protective clothing and face masks, recorded a detailed medical history and reviewed the blood test results. Once a patient had fever for more than 3 days or had epidemiological contact with novel coronavirus 2019, they ordered chest high-resolution CT scans.

Clinical characteristics of patients were collected as follows: duration of fever; accompanying symptoms; body temperatures; respiration rates; pulse rates; pulse oxygen saturation at fingertips. Results of blood tests, including CRP, were also collected. All these characteristics were included as predictors to develop a model for pneumonia prediction.

AFRI was identified in febrile patients (body temperature ≥ 37.3℃) who had one of the four respiratory symptoms including runny nose, nasal congestion, sore throat, and cough. Pneumonia was diagnosed when acute alveolar or interstitial infiltrates were found on CT images, which was interpreted independently by two experienced radiologists who were blind to the patients’ clinical characteristics. When radiologists had difficulty in determining whether infiltrates on CT images were acute or not, they excluded the related patients from the analysis.

An external validation population was enrolled in primary care settings from the provinces of Hubei, Jiangxi, Hainan, Guizhou, and Shanghai in China from March 1, 2020, to December 31, 2021. These participants (≥ 18 years of age) presented with fever (body temperature ≥ 37.3℃) and respiratory symptoms when visiting the clinics and agreed to undergo blood tests and CT scans. Physicians collected their data of symptoms, signs, and CRP concentrations, and the prediction model developed in the first stage was used to get a pneumonia risk mark. Patients were sent for CT scans to determine whether they had pneumonia by two radiologists independently. In patients who were diagnosed with pneumonia, pneumonia severity index (PSI) were assessed, and the numbers of infected lung lobes were collected. Sputum of these participants was cultured to clarify the bacterial pathogens, and nucleic acid tests for respiratory specimens (nasopharyngeal or oropharyngeal swab samples) were tested for viral pathogens such as SARS-COV-2. Serological tests on common respiratory pathogens (Influenza A, Influenza B, Para-influenza, Mycoplasma, Chlamydia, Legionella bacteria, Respiratory syncytial virus, Adenovirus, and Coxsackie virus) were also conducted. Then the performance of the prediction model was assessed for pneumonia patients with different severity and pathogens.

SAS software (version 8.0; SAS Institute; Cary, NC, USA) were used for statistical analysis, and graphs were drawn using GraphPad Prism (version 8; GraphPad Software, San Diego, CA, USA). A ‘Trialsize’ package of R software (version 4.0.2) was used to calculate the sample size. Complete-case analysis was used to handle missing data. Quantitative variables of normal distribution were presented as the mean ± SD. Independent t-test was used to determine whether normally distributed variables were different between patients with and without pneumonia. Chi-squared (χ²) test was used to determine whether qualitative variables were different between the two groups. A logistic regression model was used to determine the factors that were independently associated with pneumonia by stepwise selection. Bootstrapping method was used for internal validation. The predicted probability of pneumonia for each subject was calculated using the equation previously reported [8], and then the risk groups were created. For external validation population, a scoring system was made, with one score for each predictor item. Sensitivity and specificity statistics and receiver operating characteristic (ROC) analyses were estimated for pneumonia prediction. Z test was used to compare areas under curves (AUCs) between the two variables or two models by the Hanley and McNeil method. The Spearman correlation coefficient (rs) was used to indicate the association between the two nonparametric variables. A P value < 0.01 was considered to denote statistically significant differences.

A total of 2513 patients visited fever clinics during the model development study, and 1977 of them presented with AFRI and thus were enrolled. Among these patients, 751 underwent Chest HRCT examination, and the determination of pneumonia present or absent was made in 727 patients, including 364 men and 363 women with a mean age (± SD) of 43.5 ± 17.8 years. Of these 727 patients, 227 (31.2%) had pneumonia, and 500 (68.8%) did not (Fig. 1A). AFRI patients who did not undergo CT examination (n = 1226) were treated as participants with missing data, thus were not included in the analysis. The clinical variables between patients with and without pneumonia were shown in Table 1. Compared with non-pneumonia patients, patients with pneumonia were older (t = 7.9, P < 0.0001), and had significantly longer duration of fever (t = 11.9, P < 0.0001), and higher frequencies of purulent sputum (χ² = 93.4, P < 0.0001), dyspnea (χ² = 230.0, P < 0.0001), and thoracic pain (χ² = 89.0, P < 0.0001). Respiration rates were significantly increased in pneumonia patients (χ² = 183.4, P < 0.0001). For blood parameters, higher percentages of neutrophils (t = 3.1, P = 0.0019), lower percentages of lymphocytes (t = 3.7, P = 0.0003) and eosinophils (t = 2.7, P = 0.0064), and lower hematocrits (t = 3.7, P = 0.0002) were found in pneumonia patients. CRP concentrations was significantly higher in pneumonia patients than in non-pneumonia patients (t = 6.9, P < 0.0001).

Figure 2A showed the ORs of five selected variables (duration of fever ≥ 2 days, purulent sputum, dyspnea, thoracic pain, and respiration rates > 20/min) from symptoms and signs that were significantly more frequent in pneumonia patients. ORs were statistically significant in all these five variables, with dyspnea showing the highest value of 97.0 (95% CI, 42.3–199.4). Positive predictive value (PPV) and negative predictive value (NPV) for these variables were calculated (Fig. 2B). Dyspnea and respiration rates > 20/min showed good predictive value in both PPV and NPV. Although duration of fever ≥ 2 days, purulent sputum and thoracic pain had good NPVs, their PPVs were relatively poor, resulting in significantly lower accuracy than that of dyspnea and respiration rates > 20/min (P < 0.01), thus were not selected for further analysis.

Predictive performance for pneumonia was analyzed in five selected blood parameters (percentage counts of neutrophils, lymphocytes, and eosinophils, percentages of hematocrit, and serum concentrations of CRP) that were significantly different between pneumonia and non-pneumonia patients. Corresponding ROC curves for these parameters were constructed in Fig. 2C. In these parameters, CRP yielded significantly higher accuracy than other parameters (Z = 3.3, P < 0.01), with an AUC of 0.7249 (95% CI, 0.7019–0.7433).

The model development had several steps. Firstly, logistic regression analysis was performed with symptom (dyspnea) and sign (respiration rates > 20/min) for pneumonia prediction (227 pneumonia and 500 non-pneumonia). Both variables had statistically significant ORs, and the “symptom and sign” model was constructed as follows.

P = predicted probability of pneumonia. D = dyspnea (yes, 1; no, 0). R = respiration rates (> 20 /min, 1; ≤ 20/min, 0).

Then, the predicted probability of pneumonia for individual patients was calculated according to the model, and its predictive value was analyzed using ROC curves, yielding an AUC of 0.7900 (95% CI, 0.7487–0.8313). When CRP was added, as a continuous variable, to the logistic regression analysis as follows, the predictive value of “symptom and sign” model was significantly improved (Z = 2.7, P < 0.01), with an AUC of 0.8716 (95% CI, 0.8409–0.9023) (Fig. 2D).

P = predicted probability of pneumonia. D = dyspnea (yes, 1; no, 0). R = respiration rates (> 20/min, 1; ≤ 20 /min, 0). C = CRP (mg/l).

ORs and predictive performance for five clinically relevant cut-off values of CRP (> 10 mg/l, > 20 mg/l, > 30 mg/l, > 40 mg/l, and > 50 mg/l) were calculated in the “symptom and sign + CRP” model. All ORs were statistically significant for different CRP cutoffs. When CRP > 20 mg/l was chosen for the model, the AUC reached the highest value of 0.8506 (95% CI, 0.8158–0.8854), thus CRP > 20 mg/l was included in the final model as follows.

P = predicted probability of pneumonia. D = dyspnea (yes, 1; no, 0). R = respiration rates (> 20/min, 1; ≤ 20 /min, 0). C = CRP (> 20 mg/l, 1; ≤ 20 mg/l, 0).

For the convenience of the model application among AFRI patients in primary care settings, a scoring system was made with one score for each item. Based on the DRC model, the high-risk group for pneumonia consisted of patients positive for all three items (3 scores). The combined probability value in this high-risk group of having pneumonia was 99.8% (95% CI = 99.6–99.9%). Moderate-risk group for pneumonia consisted of patients positive for one or two of these items (1–2 scores), and the combined probability value was 59.8% (95% CI = 56.2–65.3%). Low-risk group for pneumonia consisted of patients without positive score (0 score) on the three items, and the combined probability value was 9.7% (95% CI = 9.3%-10.5%).

Setting the criteria of ≥ 1 score as the cut-off point, the sensitivity, and specificity of the DRC model for pneumonia prediction in the model development population (n = 727) was 81.1% and 80.2%, respectively. Notably, in patients whose age was above 60 years (n = 159), its sensitivity and specificity were 84.0% and 84.6%, suggesting the model had good predictive capacity for old patients. If the prediction model was applied to the patients who received CT examination, CT could have been avoided in 401 (55.2%) of these patients, with a risk of missing 43 patients (5.9%) with pneumonia. For the 43 patients mispredicted by the model, all of them had at least one of other risk factors including duration of fever ≥ 2 days, purulent sputum, and thoracic pain.

We assessed the predictive value of the DRC model in an external validation population than included 1472 AFRI patients in primary care settings from 5 different provinces in China. Among these patients, 1299 patients (681 men and 618 women with a mean age (± SD) of 42.4 ± 17.9 years) were included in the analysis (Fig. 1B). HRCT scans determined that 447 patients had pneumonia and the other 852 patients didn’t. For patients who had pneumonia, respiratory pathogens could be identified in 308 patients, including 93 patients with bacterial pathogens, 171 patients with viral pathogens, 41 patients with atypical pathogens, and 3 patients with fungal pathogens. No significant difference was found in the clinical characteristics and blood test results between development population and validation population.

The scores of the DRC model were calculated and analyzed as shown in Fig. 3. Dyspnea was present in 191 (42.7%) pneumonia patients, respiration rates > 20/min was present in 201 (45.0%) pneumonia patients, and CRP > 20 mg/l was present in 259 (57.9%) pneumonia patients. Overlaps among these patients were shown in Fig. 3A. ROC analysis indicated that DRC model had good performance for pneumonia prediction with an AUC of 0.8765 (95% CI, 0.8549–0.8980), and the predictive accuracy was the highest when choosing at least one positive item (1 score) as cut-off point (sensitivity = 87.0%, specificity = 80.5%, Fig. 3B).

PSI scores were assessed for pneumonia patients, and DRC scores were analyzed in different PSI classes. We found DRC scores showed a positive correlation with PSI scores (rs = 0.5020, P < 0.0001). In PSI class V, the percentage of patients who had 3 scores of the DRC model reached 91.7% (Fig. 3C). DRC scores also showed a positive correlation with the numbers of infected lung lobes (rs = 0.6511, P < 0.0001). For patients who got 3 scores, the number of lobes reached 3.3 ± 1.2 (Fig. 3D).

Among patients with detected respiratory pathogens, DRC scores were calculated according to their different infecting pathogen genus (Fig. 3E). As a whole, DRC scores were higher in those with bacterial pathogens (2.0 ± 1.0) than those with viral ones (1.1 ± 0.7, P < 0.01). ROC curve was analyzed for pneumonia prediction with bacterial or viral pathogens. We found the model showed good predictive performance for both bacterial and viral pneumonia, with AUC of 0.9334 and 0.8385 respectively (Fig. 3F). We also analyzed the difference in ROC curves between the DRC model and its simplified form DR model (composed of dyspnea and respiration rates > 20/min only) in different pathogen groups. For bacterial pneumonia, the AUC of the DR model was significantly less than that of the DRC model (Z = 3.2, P = 0.0016). However, for viral pneumonia, no significant difference was found in the AUCs between the two models (Z = 1.0, P = 0.2993), suggesting dyspnea plus respiration rates > 20/min had a good predictive capacity for viral pneumonia even without the result of CRP.