The value of ROX index in predicting the outcome of high flow nasal cannula: a systematic review and meta-analysis

Our meta-analysis was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) [16]. The protocol of this study was registered, with PROSPERO registration number of CRD42021240607.

According to the criteria of literature retrieval strategies, two authors independently searched MEDLINE, EMBASE, and Cochrane Library electronic databases. Since the first relevant article was published in 2016, the retrieval time was from 2016 to April 20, 2021. Without language restrictions, we conducted a search using the following terms: respiratory rate-oxygenation/ROX/ROX index/ROXI and HFNC/high-flow nasal cannula/nasal high-flow oxygen therapy/high-flow oxygen therapy. The study research disagreements were resolved by discussions, and when discussions failed to address the disagreements, a third author was consulted.

Literatures were selected based on the following eligibility criteria: (1) Studies evaluated the ability of ROX index in predicting HFNC outcome among adult patients (age over 18 years). (2) Number of true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN) could be found or calculated by other data from the study. (3) No restrictions were set regarding the races or locations of patients and the types of diseases for which HFNC was employed. We excluded systematic reviews, case reports, pediatric studies, and repeated reports.

Two data collectors independently read the literatures according to inclusion and exclusion criteria and extracted the associated data elements from the enrolled papers using a self-made data extraction form. Inconsistencies were solved by discussion, and when discussion failed to address the disagreements, a third author was consulted. The extracted contents were as follows: (1) The basic characteristics of the included studies: name of the first author, publication year, locations of the study, the diseases that require HFNC use, number of patients, the study design, acquisition time of ROX index and definition of HFNC failure in each enrolled study. (2) Other essential data of each study include TP, TN, FP, FN, cut-off value of ROX index, as well as the sensitivity and specificity to predict HFNC failure. We also contacted the corresponding authors by email when necessary data were not included in the article.

Based on quality assessment of diagnostic accuracy studies 2 (QUADAS-2) guidelines [17], two authors independently assessed the risk of bias in enrolled studies, and a third author was invited to resolve disagreements during methodological quality assessment. The two main domains included risk of bias and applicability concerns. Each study evaluated the domain of risk of bias by evaluating patient selection, index test, reference standard, flow, and timing, whereas the domain of applicability concerns evaluated patient selection, index test, and reference standard. For each item, the risk of bias was classified as high, low, or unclear.

The data analysis was conducted using STATA version 1.4 (Stata Corporation, College Station, TX). This meta-analysis determined pooled sensitivity, specificity, negative likelihood ratio (NLR), positive likelihood ratio (PLR), diagnostic score, and diagnostic odds ratio (DOR). When statistical heterogeneity was not found (P > 0.05, I² < 50), the inconsistency factor (I²) statistics and Cochran-Q test were employed to analyze the potential heterogeneity, and the fixed-effect model was applied to calculate the pooled effect size; otherwise, the random-effects model was utilized. Subgroup analyses were employed to explain source of heterogeneity, and they included the types of diseases associated with HFNC use, cut-off value, and the acquisition time of ROX index. In addition, Deeks’ funnel plot was performed to estimate the potential publication bias.

A total of 371 literatures were searched, including 260 from PubMed, 65 from EMBASE, and 46 from the Cochrane Library databases, 3 from other sources (such as abstracts from conferences). There were 279 articles left after excluding the reduplicated articles. After reviewing the abstracts, some articles were excluded, including systematic reviews, case reports, and coverage not matched, and 17 articles remained, which we read in full. A total of 8 studies were excluded due to the lack of data required for analysis, whereas 9 articles met the full inclusion criteria finally. The detailed PRISMA flowchart is depicted in Fig. 1.

The detailed baseline characteristics are demonstrated in Tables 1 and 2. All the included trials were published between 2016 and 2021. Four studies were conducted in Europe [12‐15], three in Asia [10, 11, 18], and the remaining two in USA [9] and Africa [19]. Four studies were multiple-center [12, 13, 15, 19] and the other five were single-center studies [9‐11, 14, 18]. For the types of diseases requiring HFNC use, three studies were ARF [10, 14, 15], four were COVID-19 [9, 11, 18, 19], and two were pneumonia [12, 13]. The total number of patients was 1933, of which 745 were failure cases. Among these nine studies, five were prospective observational cohort studies [9, 10, 12, 13, 19], three were conducted retrospectively [11, 14, 18], and only one was RCT [15]. The acquisition time of ROX index after applying HFNC varied in different studies; six studies [9‐12, 15, 19] reported 6 h as the acquisition time, one study [13] chose 12 h, another study chose 4 h [18], and the remaining study measured ROX index before each attempt to separate from HFNC [14]. HFNC failure was defined according to the included studies, five studies [9, 10, 12, 13, 15, 18] defined HFNC failure as the subsequent need for invasive MV, while Hu et al. [11] defined HFNC failure as the need for NIV or IMV and/or death, 1 study [19] defined HFNC failure as the need for MV or death, another study [14] defined HFNC failure as requiring HFNC resumption, NIV initiation, intubation, or death. The cut-off values of ROX index ranged from 2.7 to 9.2, and while three studies [12, 13, 15] reported 4.88 as the best cut-off value, other researchers found the values of 3.66 [9], 5.55 [11], 5.8 [10], 9.2 [14], 2.7 [19], 5.31 [18] in each study, and the sensitivity and specificity of ROX index to predict HFNC failure were from 50 to 84%.

After careful evaluation of the methodological quality of all nine enrolled studies, we found that seven studies [9, 10, 12, 13, 15, 18, 19] had a low risk of bias in patient selection, three trials [12, 13, 15] had a low risk of bias in index test, all the included studies exhibited low risk of bias in the item of reference standard, and eight studies [9‐13, 15, 18, 19] had low risk in flow and timing. To sum up, the included seven studies were of good quality (Table 3).

Nine trials with 1933 patients assessed the ability of ROX index in predicting HFNC outcome, and there was statistically significant heterogeneity in the sensitivity (I² = 84.97%), specificity (I² = 86.95%), PLR (I² = 33.93%), NLR (I² = 74.64%), diagnostic score (I² = 41.76%) and DOR (I² = 99.91%). Therefore, we used a random-effect model to conduct this meta-analysis. Pooling all the enrolled studies, the sensitivity, specificity, PLR, NLR, diagnostic score, and DOR of ROX index in predicting the HFNC failure are depicted in Fig. 2. The pooled estimates of ROX index in predicting HFNC failure were as follows: sensitivity, 0.67 (95% CI 0.57–0.76); specificity, 0.72 (95% CI 0.65–0.78); PLR, 2.4 (95% CI 2.0–2.8); NLR, 0.46 (95% CI 0.37–0.58); diagnostic score, 1.65(95% CI 1.37–1.93); and DOR, 5.0 (95% CI 4.0–7.0). We also conducted the summary receiver operating characteristic (SROC) plot to evaluate the predicting accuracy of ROX index (Fig. 3), and the area under the curve (AUC) was 0.75 (95% CI 0.71–0.79), implying that ROX index could predict HFNC failure.

To determine heterogeneity between studies, we conducted subgroup analyses based on the types of diseases correlated with HFNC use, the areas of enrolled studies, and the acquisition time of ROX index (Table 4). For the types of diseases correlated with HFNC use, there were four trials reported data on COVID-19, with pooled sensitivity and specificity of 0.71 (0.56–0.82) and 0.73 (0.63–0.81), respectively, while lower sensitivity (0.63, 0.48–0.75) and specificity (0.71, 0.60–0.80) were found among other diseases. In the subgroup of cut-off value, the sensitivity and specificity in predicting HFNC failure were 0.59 (0.54–0.65), 0.83 (0.79–0.86) when the cut-off value > 5, and 0.67 (0.65–0.76), 0.71 (0.61–0.80) in ≤ 5 cut-off value. The pooled results of six enrolled studies indicated that summary sensitivity and specificity were 0.67 (0.54–0.78) and 0.70 (0.60–0.78) when acquisition time was 6 h after receiving HFNC. However, good performance of ROX index was found in settings with other acquisition times, with sensitivity and specificity of 0.73 (0.67–0.79) and 0.74 (0.70–0.78), respectively.

We performed Deeks’ funnel plot to evaluate the potential publication bias, and the funnel plot demonstrated no statistically significant publication bias in this meta‐analysis (P = 0.74) (Fig. 4).