Discriminative performance of pancreatic stone protein in predicting ICU mortality and infection severity in adult patients with infection: a systematic review and individual patient level meta-analysis

A systematic literature search was performed across the Cochrane Central Register of Controlled Trials (CENTRAL and MEDLINE (1966 to February 2022) databases using “pancreatic stone protein”, “PSP”, “regenerative protein”, “infection”, “sepsis”, “lithostatin” as keywords and/or MeSH Terms. The search strategy was prepared according to PRISMA individual patient data guidelines (Supplemental Tables 1 and 2) [28]. The search was restricted to original human clinical trials on PSP/reg published in English before February 2022 that evaluated the performance of PSP for the assessment of the severity of infection as well as for predicting its outcome among unselected adult patients upon their admission to the ED or the ICU. The search was further restricted to studies that determined PSP levels in blood using the enzyme-linked immunosorbent assay technique (ELISA) developed and described by Rolf Graf et al. [20, 29], to impede calculation limitations when plotting equal PSP levels when using different analysing methods. Paediatric trials and autopsy studies were excluded. The definitions of infection used in each of the eligible studies are presented in Supplemental Table 3.

Two reviewers (JP and YAQ) independently assessed trial eligibility based on titles, abstracts, full-text reports, and further information from investigators as needed (Fig. 1). Study protocols and unedited databases containing anonymized individual patient data were obtained from investigators of all eligible trials.

The study was registered on Prospero (#CRD42022308207). The Cantonal Ethical Committee of the State of Bern (#2018-01356_V2.1_25.2.2022) reviewed and approved the meta-analysis research protocol while the respective ethical committees already approved all individual studies.

All raw data were received from their principal investigators with patient specific anonymized ID and contained at least the following information: age, gender, Sequential Organ Failure Assessment (SOFA) Score and blood levels of PSP, CRP and PCT upon admission, days to death and ICU mortality. Data from each eligible study were first checked for duplicates and second against reported results. Queries were resolved with the principal investigator, trial data manager, or statistician whenever indicated.

The primary objective of the study was to evaluate the diagnostic accuracy of PSP in predicting ICU mortality and compare it to CRP and PCT. The secondary objectives were: (i) to evaluate PSP ability to predict disease severity and compare it to CRP and PCT, and (ii) to explore whether different combinations of the three biomarkers further improve the prediction of ICU-mortality and disease severity.

Our primary endpoint was ICU mortality. Secondary outcomes were based on disease severity risk stratification on SOFA score upon admission: (i) non-complicated infection (patients with SOFA score ≤ 1; (ii) sepsis (patients with SOFA score ≥ 2) and (iii) septic shock (patients with SOFA score ≥ 2 and need for vasoactive drugs). We used the combined endpoint (sepsis and septic shock) as secondary outcome.

We adjusted all outcomes for age and sex. For the primary outcome ICU-mortality, we additionally adjusted for sepsis severity (mild moderate infection/infection, sepsis, septic shock).

We described the study population by counts and percentages, median and interquartile range. Missing PSP, CRP or PCT measurements were replaced by median values within each study, because of the low missing value proportion: Percentage of missing values per study ranged from 0.4% for CRP/PCT to 5.6% for PSP (Supplemental Table 4). For adjusted analyses, three missing age values were replaced by the median value of the corresponding study.

We followed the meta-analytic approach used by Prazak et al. [24] and described in Steyerberg et al. [30]. Briefly, we evaluated three different models: (i) a random effect-random slope (RERS) model (random intercept on study and biomarkers as random slopes including a fixed effect on biomarkers for population mean interpretation of the random intercept and slope [31]); (ii) a random effect (RE) model (random intercept on study and fixed biomarker effect); and (iii) a fixed effect (FE) model (fixed biomarker effects without any patient clustering information). We compared models using the Akaike information criterion (AIC) and log-likelihoods. Because of the small number of studies and convergence issues of frequentist random effects models, we used Bayesian hierarchical logistic regression models. We used centered Gaussian priors with a standard deviation of 2.5 for intercept and biomarker effects [32]. For the centered multivariate Gaussian distributed random effects we used a Lewandowski-Kurowicka-Joe prior with a regularization parameter set to 1, a concentration parameter set to 1 and a unit-exponential prior on the scale parameters for the decomposition of the correlation matrix [2]. We used unadjusted models (using only biomarker values as predictors) and adjusted models (biomarker values and all confounding variables) reporting odds ratio with 95% credible intervals (CrI). Biomarker measurements were standardized (centered and divided by population standard deviation) and age centered and expressed as a 10-year increase. We reported study-specific outcome estimates and 95% CrI as well as between-study standard deviation and I-squared. 95% prediction intervals (PI) were calculated from the overall intercept plus a centered Gaussian distributed random variable with a standard deviation equal to the estimated between-study standard deviation. We reported AUC values with 95% confidence intervals (CIs), positive and negative predictive values, and classification plots [33]. A specific risk threshold cutoff was computed based on Youden’s index [34]. All analyses were performed in R version 4.1.2 [27]. Bayesian analyses were implemented in the Stan R interface [2] using 4 Markov chains with 1,000 warmup iterations per chain and 2,000 total iterations per chain.

Among the 48 records published before February 2022 and identified through the literature search, 46 full texts were further assessed for eligibility. 24 records were excluded based on review of title and abstracts. Mainly due to lack of measured biomarkers on admission or addressing a pediatric patient population, only five of the remaining 22 observational studies were included into the final analysis (Fig. 1 ; Table 1). Individual patient data from all patients were used for the evaluation of the primary endpoint ‘ICU mortality’ (Table 2). For the assessment of the secondary endpoint predicting disease severity, the studies of Que et al. [23] and Guadiana-Romualdo et al. (2019) [35] were excluded, since those studies only included patients with severe sepsis or septic shock (Fig. 1; Table 1).

We considered 678 patients in the study; 64% were male with a median age of 65 (Table 2). The biomarkers were measured on 549 patients admitted to ICUs and on 129 admitted to the emergency room. The distributions of the three biomarkers by study disease severity are shown in supplement (Supplemental Figs 1 and 2).

The observed crude overall ICU mortality was 22% (149 out of 678 included patients). Model performance was best for a RERS models based on AIC (Supplemental Table 5). The pooled overall estimate from a RERS model was 17.8%, 95% CrI (9.1–31.5%) with a 95% PI ranging from 4.1–54.6% with a substantial heterogeneity between studies (I-squared 87%), (Fig. 2).

PSP was strongly associated with ICU mortality (OR = 2.7, 95% CrI [1.3–6.0] per one SD increase), even after adjustment for age, gender and sepsis severity (OR = 1.5, 95% CrI [0.98–2.83], Supplemental Fig. 3). The AUC from an unadjusted RERS model was 0.69 [95%CI 0.64–0.74]. We identified a PSP cut-off value of 133.6 ng/ml based on Youden index at a risk threshold at 13% with positive (PPV, 0.32, 95%CI [0.27–0.36]) and negative (NPV, 0.90, 95%CI [0.87–0.93]) predictive values using PSP (Table 3). Calibration plots showed that the sensitivity for PSP was 0.96, 0.52, 0.30 for risk thresholds 10%, 20% and 30%; respective false positive rate values were 0.84, 0.25, 0.10 (Fig. 3). Similar analyses were performed for CRP and PCT. Combining biomarkers in all different models evaluated did not increase the discriminative performance of PSP (Supplemental Fig. 4; Supplemental Table 6).

PSP was higher in patients with sepsis/septic shock compared to those with mild infections and strongly associated with the combined endpoint of sepsis/septic shock in both unadjusted (OR = 11.4, 95% CrI [2.1–54.5]; per one SD increase and age–gender adjusted models (OR = 11.4, 95% CrI [1.9–48.9]), (Supplemental Fig. 5). For the secondary combined outcome of sepsis and septic shock we estimated a pooled overall percentage of 79.9%, with a 95% PI ranging from 5.5% to 99.6% with a considerable heterogeneity between studies (I-squared 93%), (Fig. 4).

Risk thresholds based on Youden index to discriminate mild infection form severe infection/septic shock were 61.7 ng/ml for PSP, 125.9 mg/l for CRP and 1.1 ng/ml for PCT (Fig. 5). Using those, PSP (AUC 0.80, 95%CI [0.75–0.85]) and PCT (AUC 0.79, 95%CI [0.74–0.84]) performed better that CRP (in stratifying patient according to infection severity: AUC was lowest for CRP (AUC 0.56, 95%CI [0.50–0.63]). PPV was the highest for PCT (0.87, 95%CI [0.81–0.92] and NPV for PSP (0.67, 95%CI [0.58–0.75]) (Supplemental Table 6). Discriminative performance (as measured by AUC) did not improve when biomarkers where combined (Supplemental Fig. 6; Supplemental Table 7).