The role of Shenqi Fuzheng injection as adjuvant therapy for breast cancer: an overview of systematic reviews and meta-analyses

Two reviewers (Jing Xu and Xiaofeng Du) searched the following databases from their inception to October 1, 2022. PubMed, Web of Science, EMbase, Cochrane Library, China National Knowledge Infrastructure Database (CNKI), WanFang database, China Science and Technology Journal Database (VIP), and SinoMed. Additionally, all references included in the SRs/MAs were retrieved a second time. The retrieval strategy used a combination of subject words and free words. The PubMed search strategy is presented in detail in Fig. 1. The search strategies of other databases are shown in Supplementary Material 1.

The reliability criteria followed the PICOS principles, as follows: (1) participants: patients with BC confirmed by histopathology or cytology, regardless of sex, age, race, and course of the disease; (2) interventions: SFI adjuvant CT, without limiting the CT scheme, frequency, and dose; (3) Comparator(s)/control: CT alone, without limiting the CT scheme, frequency, and dose; (4) outcomes: clinical response rate, KPS score, immune function (CD3+, CD4+, CD8+, CD4+/CD8+, NK cells), adverse reactions (incidence of leukopenia and gastrointestinal reactions); and (5) type of study: as randomized controlled trials (RCT) are the gold standard for evaluating clinical evidence, SRs/MAs based on RCT were selected for the overview, unrestricted language.

We excluded network meta-analysis, studies that included TCM decoction and other TCM therapies in the intervention, studies where full text data was not available, and studies with data errors.

The overview used EndNote X9 to perform a secondary search to remove duplicates and read the remaining literature in depth. SRs/MAs were independently screened by two reviewers (Jing Xu and Liyuan Lv) according to the inclusion criteria, data were extracted and cross-reviewed. Any dispute was resolved by the third reviewer (Hou Li). The authors were contacted when necessary to obtain the complete original data. Two reviewers (Jing Xu and Liyuan Lv) independently extracted data including first author, publication year, language, country, number of the included RCTs, sample size, intervention measures, bias risk assessment tools, outcome indicators, funding, and conflict of interest statements.

Overview of the included SRs/MAs can exaggerate the efficacy of the studies due to excessive overlap of the original literature, leading to similar conclusions about SRs/MAs. Therefore, an overview establishes the overlap matrix of SRs/MAs and its included original literature and calculates the Corrected Covered Area (CCA) to evaluate the degree of duplication of SRs/MAs original literature [23]. The calculation formula is CCA=(Nr)/(rc-r), where N is all original SRs/MAs studies (including duplications), r is all original studies after eliminating duplicates, and c is the number of SRs/MAs included in the overview. If the CCA is 100%, it indicates that every review included in the overview contains the same RCTs, while if the CCA is 0%, it indicates that every review included in the overview contains completely different RCTs [23]. A CCA of ‘0% to 5%’ indicates slight overlap, ‘6% to 10%’ indicates moderate overlap, ‘11% to 15%’ indicates high overlap, and ‘≥15%’ indicates extremely high overlap.

AMSTAR-2, an internationally recognized systematic methodological quality assessment tool, was used to evaluate the methodological quality of the included SRs/MAs [24]. AMSTAR-2 includes a total of 16 items, items 2, 4, 7, 9, 11, 13, and 15 are critical items, others are noncritical items. Two reviewers (Jing Xu and Xiao Li) independently performed the evaluations. Any dispute was discussed or decided by the third reviewer (Li Hou). When the corresponding content matched an item, it was marked as ‘yes,’ when it did not match completely, it was marked as ‘partial yes,’ and when it did not match, it was marked as ‘no.’ When there was no or only one non-critical item not satisfied, it was rated as ‘high’ quality. If more than one non-critical item was not satisfied, it was rated as ‘moderate’ quality. If only one critical item was not satisfied, it was rated as low-quality. When more than one critical item did not meet the criteria, it was rated as ‘critically low’ quality.

The statement Preferred Reporting Items for Systematic Review and Meta-analyses (PRISMA) 2020, an updated guideline for reporting systematic reviews, contains 27 items (42 sub items), including seven parts such as title, abstract, preface, method, result, discussion, and others. On completion of SRs/MAs, each item is evaluated as ‘yes,’ ‘no,’ or ‘partially yes’ [25]. The evaluation of the quality of the overview reporting was independently evaluated by two reviewers (Jing Xu and Xiao Li), and any disagreement was discussed or decided by the third reviewer (Li Hou).

The Risk of Bias in Systematic Reviews (ROBIS) is a tool for assessing the risk of bias of SRs/MAs. The tool is divided into three phases and is mainly used to mark different biases in phases 2 and 3. Phase 2 includes four domains: research identification and selection, data collection and study evaluation; extraction, quality evaluation, synthesis and quality evaluation; and synthesis and results presentation). Instead, phase 3 assesses the overall risk of bias. The degree of SRs/MAS bias risk judgment is expressed as ‘low,’ ‘high,’ and ‘uncertain’ [26]. Two reviewers (Jing Xu and Xiao Li) independently assessed the bias risk of SRs/MAs through ROBIS and cross-checked after completion, and any disagreement was discussed or decided by the third reviewer (Li Hou).

The Grades of Recommendation, Evaluation, Development, and Evaluation (GRADE) are applied for evidence quality assessment of included outcomes using five downgrade factors: study design, imprecision, inconsistencies, indirectivity, and risk of bias. After the assessment, the quality of the evidence was classified into four levels: ‘high,’ ‘moderate,’ ‘low,’ and ‘very low’ [27]. Evaluators are trained to reach consensus before performing an assessment. The two reviewers (Jing Xu and Xiao Li) were trained prior to formal evaluation, and any disagreements were discussed or decided by the third researcher (Li Hou).

To better clarify the efficacy of adjuvant CT of SFI for BC, RevMan 5.4 was used for the quantitative analysis of the included SRs/MAs. Results are reported using the standardized mean difference (SMD), odds ratio (OR) or relative risk (RR), 95% confidence interval (CI), random effects model, fixed effects model, and heterogeneity (I²). If P>0.1 and I²≤ 50%, the fixed effect model is adopted. Instead, if P<0.1 and I²≥50%, the random effects model was used [28]. If heterogeneity was significant, subgroup or sensitivity analysis was used to reduce heterogeneity, and the funnel plot was used to determine whether publication bias existed in the included studies.

The process of SRs/MAs retrieval, screening, and selection in the overview was shown in a PRISMA flow chart (Fig. 2). A preliminary search obtained 34 studies, of which 14 were obtained after deleting duplicates, and of these six were excluded due to network meta-analysis and meta-analysis not including SFI for BC. One meta-analysis was repeated in both Chinese and English, and one meta-analysis was excluded due to inconsistent content and title; finally, six SRs/MAs were included in the overall analysis [6‐8, 29‐31].

Six SRs/MAs including 61 RCTs with 5593 patients were eligible for this study after removing duplicates. The overview provided a table of the main characteristics of six SRs/MAs, including five in English and one in Chinese. The time span included in the overview was 2015–2019 (Table 2). Six SRs/MAs were included in the original RCT range from 7 [29] to 49 [7], and the total sample size was between 542 [29] and 4385 [7]. All original RCTs included in SRs/MAs were developed and implemented in China.

The overview included a total of six SRs/MAs, and the number of all original studies included was 145, 61 after the removal of duplicate literature. According to the formula, N was 145, r was 61, and c was 6, CCA=(145–61)/(61×6–61)*100%=27%, which indicated a large overlap. This reflects the unnecessary duplication of SRs/MAs in adjuvant CT treatment with SFI of BC. Future SRs/MAs should only be conducted when the research objectives are significantly different. Prospective registration of SRs/MAs may help avoid unnecessary duplications [32]. Figure 3 showed the ‘cloud’ relationship between the included SRs/MAs and the original RCTs. The more original the research lines, the more frequently they were included.

Table 3 summarizes the results of the methodological quality assessment of six SRs/MAs in the AMSTAR-2 overview. Because all SRs/MAs presented one or more critical weaknesses, the quality of the methodology used to identify them was low or very low. Three SRs/MAs were of very low quality, and three SRs/MAs were of low quality. Systematic methodological quality reviews varied widely and most of the included studies showed some limitations. For example, none of the studies preregistered the study protocol prior to the systematic review, none explained the basis for including only RCTS in the systematic review, and none reported the source of funding for each included study. Two studies [8, 29] did not use duplicate data extraction by double reviewers and did not report all potential sources of conflict of interest. A study [29] did not provide a list of excluded references. Two studies [29, 30] did not assess the impact of the bias risk of each included study on the results of the SRs/MAs, or did not consider the bias risk of the included study, and did not reasonably explain heterogeneity of the results. Two studies [29, 31] did not adequately investigate publication bias.

Table 4 shows the quality of the PRISMA 2020 report. Among the 27 included items, the reporting eligibility of 11 items was 100%, while the reporting eligibility of 11 items was at least 66.67%, indicating that most of the reports were relatively complete. However, there were five substantive deficiencies in the report: Abstract (Item 2), evidence quality evaluation (Item 15), evidence quality evaluation results (Item 22), registration and agreement (Item 24), and data disclosure (Item 27). Only one SR/MA (16.7%) reported retrieval strategies and information availability. Only two SRs/MAs (33.3%) reported supporting data. The remaining entries were only partially completed.

Table 5 shows the bias risk assessment of the overview. In Phase 1 and Domain 1 in Phase 2, the six SRs/MAs were rated as low risk of bias, which evaluated eligibility criteria and identification and selection of studies. In Domain-2, two studies [8, 29] had a high risk of bias because the retrieval database was incomplete or the screening process was not provided. In Domain-3, one study [29] was rated as high bias risk because it did not mention the data extraction process, and one study [8] did not involve double reviewers in the data extraction process. In Domain-4, two studies [29, 30] did not deal with heterogeneity or conduct sensitivity analysis, which had a high risk of bias. In Phase-3, the three studies [29‐31] did not analyze or deal with the bias in SRs/MAs, which had a high risk of bias.

Table 6 shows the 42 results of GRADE assessments for six SRs/MAs; 1 (1/42, 2.38%), 17 (17/42, 40.48%), and 24 (24/42, 57.14%) were rated moderate quality, low quality, and very low quality, respectively. No high-quality evidence results were found. The main reason for the low quality of evidence was the design of RCTs (42/42, 100.00%), which was the most important factor leading to the lower quality of evidence, followed by inconsistency (19/42, 45.24%), publication bias (41/42, 97.62%), and inaccuracy (3/42, 7.14%). Efficacy was the most commonly used outcome indicator, with six SRs/MAs reporting validity, including one moderate level of evidence, four lower level of evidence, and one very lower level of evidence. The KPS score is a commonly used indicator for evaluating the quality of life of patients with BC, and the five included SRs/MAs reported KPS, including results from four low-level evidence and one moderate level of evidence. Safety was an important outcome of the SFI evaluation, and included the results of nine low-level evidence and one very low-level evidence studies.

A meta-analysis of KPS scores was performed on 24 RCTs (1890 participants) included in the six SRs/MAs. There was a large heterogeneity between the studies (P<0.0001, I²=70%). A random effect model was used. The KPS score of the adjuvant CT group of SFI was better than that of the CT group (RR=1.58, 95% CI 1.39, 1.80; P<0.00,001). (Supplementary material 2, Figure S2).

The KPS score had a large heterogeneity (I²=70%). After the comparative analysis of the literature, we removed the research of Song 2004 [33], Su 2016 [34], and Xu 2010 [35], and the heterogeneity of the KPS score decreased significantly (I²=27%). Therefore, we considered that the heterogeneity of the KPS score was mainly related to the low quality of the included studies. (Supplementary material 2, Figure S3).

A meta-analysis of CD3+ assessment was performed on 13 RCTs (1124 participants) in the six SRs/MAs included and showed that there was a large heterogeneity between the studies (P<0.0001, I²⁼96%), and a random effect model was used. The results showed that CD3 + of the SFI adjuvant CT group was better than that of the CT group (SMD=1.51, 95% CI 0.91, 2.10; P<0.00001).

Seventeen RCTs (1382 participants) reported the level of CD4+, which was highly heterogeneous among the studies (P<0.0001, I²⁼97%). A random effect model was used. The results showed that CD4 + from the adjuvant CT group of SFI was better than that of the CT group (SMD=1.87, 95% CI 1.18, 2.56; P<0.00001).

Seventeen RCTs (1382 participants) reported the level of CD8+, which was highly heterogeneous among the studies (P<0.0001, I²⁼96%). The results showed that CD8+ of the adjuvant CT group of SFI was similar to that of the CT group, and the difference was not statistically significant (SMD=-0.21, 95% CI -0.82, 0.41; P=0.51).

Seventeen RCTs (1382 participants) reported the level of CD4+/CD8+, which was highly heterogeneous among the studies (P<0.0001, I²⁼91%). The results showed that the CD4+/CD8+ of the SFI adjuvant CT group was better than that of the CT group (SMD=0.86, 95% CI 0.48, 1.23; P<0.00001).

Ten RCTs (943 participants) reported the level of NK cells, which was highly heterogeneous among the studies (P<0.0001, I²⁼80%). The results showed that the NK cells of the SFI adjuvant CT group were better than those of the CT group (SMD=0.94, 95% CI 0.63, 1.24; P<0.00001) (Fig. 5).

A meta-analysis of leukopenia was conducted in seventeen RCTs (1480 participants) in the six included SRs/MAs, which that showed that there was no heterogeneity among the studies (P=0.80, I²⁼0%), and a fixed effect model was used. The results indicated that the incidence of leukopenia in the adjuvant CT group of SFI was lower than in the CT group (RR=0.53, 95% CI 0.46, 0.62; P<0.00001).

Eleven RCTs (820 participants) reported the gastrointestinal reactions, which did not show heterogeneity between studies (P=0.59, I²⁼0%). The incidence of gastrointestinal reaction of the adjuvant CT group of SFI was lower than that of the CT group (RR=0.48, 95% CI 0.39, 0.58; P<0.00001) (Fig. 6).

In this overview, the evidence for the efficacy and safety of SFI in the treatment of BC was derived from six SRs/MAs. The results of the quantitative meta-analysis found that adjuvant CT of SFI in the treatment of patients with BC was beneficial, specifically in improving the clinical response rate, quality of life, regulating immune function of the body, and with fewer leukopenia and gastrointestinal reactions. However, the overall methodological quality and evidence quality of the original RCTs included in these SRs/MAs were generally poor, lacking large-scale multicenter, placebo-controlled studies, resulting in almost all included SRs/MAs being unable to draw firm and reliable conclusions on the efficacy and safety of adjuvant CT treatment with SFI for BC. As shown by the results of the evaluation of AMSTAR-2, PRISMA 2020, ROBIS, and GRADE, the methods and quality of the evidence for most SRs/MAs are not satisfactory. Therefore, there is an urgent need to further improve the design of RCTs and SRs/MAs research protocols to determine the true efficacy and safety of SFI in the treatment of BC.

Evidence rated as high quality or moderate quality is considered relatively reliable [36]. The methodological quality of six SRs/MAs studies was evaluated by AMSTAR-2, among which three studies were of very low quality and three studies were of low quality, no high quality, or moderate quality. Although six SRs/MAs followed the PICO principle and used double reviewers to detect and assess the risk of bias in each included study and applied appropriate statistical methods, methodological deficiencies include the failure to publish predesigned study protocols, the failure to provide a detailed list of reasons for excluding studies, and the failure to report funding sources for inclusion in RCTs. Similar to the results of AMSTAR-2, the PRISMA 2020 evaluation showed that the included SRs/MAs also had these deficiencies. Furthermore, six SRs/MAs had different degrees of information deficiencies in the literature retrieval, data extraction, and description, discussion of bias risk, interest statements, and information disclosure. In the ROBIS, further analysis found insufficient explanations for the risk of bias, the risk of identifying and selecting bias in research, and insufficient evaluation of publication bias were the main factors leading to a high risk of bias. In summary, the unsatisfactory results of AMSTAR-2, PRISMA 2020, and ROBIS were mainly limited to the registration of the study protocols, the explanation of the basis for the inclusion of only RCTs, the description of the funding sources for each included study, the incomplete search strategy and screening process, addressing of heterogeneity and sensitivity, and reporting of potential conflicts of interest.

The overview evaluated the quality of the evidence of the outcome indicators using GRADE. Our results indicated that adjuvant SFI CT is beneficial for outcome indicators such as response rate, KPS score, CD3+, CD4+, CD4+/CD8+, NK cells, leukopenia, and gastrointestinal reactions in patients with BC. However, it is worth noting that the quality level of this evidence varies from ‘moderate’ to ‘very low.’ The main reasons for the design of RCTs (42/42, 100.00%), inconsistency (19/42, 45.24%), publication bias (41/42, 97.62%), and inaccuracy (3/42, 7.14%). The main reasons for the poor quality of the study were the design of the RCTs, specifically the defects of randomization, assignment concealment, the implementation of blinded methodology, and the risk of selective reporting. Second, there was publication bias, or no bias test was conducted, and the source of bias analysis was not analyzed. Additionally, some studies were downgraded in terms of inconsistency due to the large heterogeneity of the included literature and the lack of analysis on the source of heterogeneity. Furthermore, due to the small sample size included in a few studies, the effect size of the combined outcome indicators showed a wide confidence interval, which contributed to a decrease in precision.

In summary of the above deficiencies, the following implications should be applied to future SRs/MAs. First, whether it is SRs/MAs or RCTs, it is important to determine whether similar topics already exist or are ‘in progress’ before starting the study. Preregistration can maximize resource waste, reduce bias in the research process, and increase the authenticity, precision, and completeness of the study. The high overlap of CCA further reflected the unnecessary duplication of studies on the treatment of BC by adjuvant CT of SFI. Future SRs/MAs should only be conducted when the research objectives are significantly different. Preregistration of SRs/MAs may help avoid unnecessary duplications. Second, a comprehensive and detailed search strategy and indexing criteria should be formulated. The gray literature should not be ignored. Comprehensive, objective, and repeatable search and screening strategies are the basis for high-quality SRs/MAs. Third, the research should report on sources of funding and conflicts of interest. Fourth, all studies should explain the sources of heterogeneity and the risk of bias, and conduct subgroup and sensitivity analyses. Fifth, although RCT is the gold standard for clinical evidence, the specific reasons for choosing this type of study should also be explained. Furthermore, the principles of multi-person independence, repetition, strict screening, and data extraction must also be strictly implemented. Based on the above, the quality of SRs/MAs can be improved.

The GRADE analysis of the included studies indicated that for most outcome measures, the quality of evidence was low or very low. It is mainly the design of RCTs that leads to degradation of study quality. Future studies should focus on the design of high-quality RCTs with large sample sizes and placebo-controlled controls. Another factor that leads to a low quality of evidence is inconsistency. Most studies had a high heterogeneity in outcome indicators, but no analysis was performed. In future SRs/MAs, subgroup analysis and sensitivity analysis should be conducted to identify the sources of heterogeneity. If heterogeneity cannot be reduced, descriptive analysis can be considered. Additionally, an insufficient sample size also leads to a decrease in accuracy. Finally, it should be mentioned that survival is an important outcome indicator for evaluating tumor patients. In RCTs research, there is a lack of research on SFI for the survival of patients with BC. Prolonging survival is the ultimate goal of adjuvant SFI treatment for BC, and future research should focus on evaluating survival-related indicators.

In general, existing evidence strongly shows that adjuvant SFI CT is beneficial for patients with BC. SFI, as a TCM injection, is widely used in the treatment of BC due to its good enhancement effect and reduction in toxicity. Its mechanism may be as follows: (1) improve immune cell function: SFI can regulate the level of T-cell subsets, enhance Th1 cells, and reduce Th2 cells to regulate their balance [37, 38]; (2) SFI can improve cisplatin drug sensitivity by regulating M2 tumor-associated macrophages (TAMs), reduce cisplatin-induced IL-10 and PGE2 release, and improve CT drug sensitivity [39]; (3) SFI can inhibit the proliferation of human BC MDA-MB-231 cells, block the cell cycle, and induce cell apoptosis by up-regulating the PUMA gene [40] and (4) SFI can protect bone marrow hemotopoietic function, enhance body sensitivity to granulocyte stimulating factor, protect white blood cells, and play a role in reducing toxicity and enhancing the antitumor effect [41]. Furthermore, network pharmacology studies have found that the SFI treatment can exert an antibreast cancer activity through nitrogen metabolism, adherent junctions, gap junctions, the HIF-1 signaling pathway, and other signaling pathways [42, 43].

This is the first overview study to examine the quality of evidence on the safety and efficacy of SFI for patients with BC using AMSTAR-2, PRISMA 2020, ROBIS, and GRADE. However, there were some limitations to our study. First, the pathological subtypes and clinical stages of patients with BC were not unified, and CT regimens were completely inconsistent. Second, all studies lacked long-term follow-up data to evaluate the long-term efficacy of SFI for patients with BC. Third, all evaluation tools used in the overview were adopted by skilled researchers, although the quality evaluation of the results was subjective. Fourth, all the RCTs in this study were conducted in China, which reduced the credibility of the results reporting.