Methodological quality for systematic reviews of adverse events with surgical interventions: a cross-sectional survey

Systematic reviews and meta-analyses of surgical interventions with safety as the exclusive outcome were of interest. A surgical intervention is defined as a therapeutic or adjunctive invasive intervention performed by a trained clinician using hands, instruments, and/or devices. It includes operative, radiological, and endoscopic procedures [27, 28]. We considered any population that received surgical interventions with any types of safety outcomes. The comparison could be any other types of interventions (e.g., surgery, drug) or a placebo. However, we did not consider systematic reviews combined with original research because such studies generally put more focus on the original research. We took the definition of a systematic review by the Cochrane Collaboration, which refers to a type of review that attempts to identify, appraise, and synthesize all empirical evidence that meets pre-specified eligibility criteria to answer a specific research question [29]. However, in practice, some of the four key components in the definition may be missing, especially for non-Cochrane reviews. Therefore, we treated those with a pre-specified eligibility criterion, clear identification (literature search and screen) process, and synthesis scheme (qualitative or quantitative) as systematic reviews.

To ensure the representativeness and sufficient statistical power, we estimated the minimum sample size (N) needed for the current study. This was done by setting 3 plausible proportions (p = 0.1, 0.3, 0.5) of adherence of the methodological items and a margin error (E) of 0.1 by the formula: N = 1.96²p(1 − p)/E² [30]. As a result, the minimal sample size required by the current study ranged from 35 to 96 under different settings.

We searched PubMed for eligible systematic reviews and meta-analyses from 1st-Jan, 2015 to 1st-Jan, 2020 without language restriction. This time frame was arbitrary, while we expected that the samples could be representative. In addition, systematic reviews in recent years were more time-sensitive for clinical practice. The literature search was done by a well-trained author. The search strategy was developed by a librarian based on the discussion with the team members, which was presented in the supplementary materials.

Two authors screened the records through the Rayyan application (https://​rayyan.​qcri.​org/​), which allowed the project manager to blind the two authors during the screening process. Titles and abstracts were first viewed to exclude studies that obviously did not meet the eligibility criteria. In this step, only those excluded by both of the screen authors were excluded. Then the full texts of the remaining studies were further reviewed for a final decision. Any disagreements were solved by discussing with the lead author. The agreement rate between the two authors during the full-text process was documented.

The following baseline characteristics were extracted by two authors independently: name of first author, publication date, region of corresponding author, language of the review, any reporting and methodological guidelines referred in the review, number of included studies, type of included studies, type of the review (quantitative or qualitative), methods for rating evidence (e.g., Grading of Recommendations, Assessment, Development and Evaluations, GRADE [31]), and funding information. Then these data were double-checked, and any disagreements were solved by discussion.

We used the AMSTAR 2.0 instrument to assess the methodological quality of eligible systematic reviews and meta-analyses (see supplementary materials) [32]. It was primarily designed for evaluating systematic reviews of randomized or non-randomized studies of healthcare interventions [32]. Therefore, it is a valid tool to assess the methodological quality. The AMSTAR 2.0 instrument contains 16 domains with 7 assigned (item 2, 4, 7, 9, 11, 13, 15) as critical domains [32]. Any methodological weakness of the 7 domains would have a great impact on the quality of the systematic review.

The global rating of the methodological quality (high, moderate, low, critically low) of a systematic review was judged by the counts of identified critical and non-critical weaknesses. In brief, high quality requires no or only one non-critical weakness; moderate quality refers to two or more non-critical weaknesses but without critical weakness; low quality means one critical weakness with or without non-critical weaknesses; and those with two or more critical weaknesses were rated as critically low [32]. It should be noted that among the 16 domains, 3 were (11, 12, 15) not applicable for qualitative systematic reviews. The rating of the global methodological quality of qualitative systematic reviews was then based on the remaining 8 non-critical items and 5 critical items (item 2, 4, 7, 9, 13).

For each domain, there were two basic responses, i.e., “Yes” and “No”. Some of the domains may have an additional response as “Partially Yes”. A list of the required components was available to help raters qualitatively make a judgment for a certain domain. For example, item 4 (Did the review authors use a comprehensive literature search strategy?) had three responses: “Yes”, “Partial Yes”, and “No.” The “Partial Yes” option required three components: searching at least 2 databases, providing keyword and/or search strategy, justifying publication restrictions (e.g., language). For the “Yes” option, more components were needed, such as searching the reference lists, searching trial/study registries, etc. Similar to item 4, the boundary between “Partial Yes” and “Yes” was clear for most of the items. However, for domain 8 (Did the review authors describe the included studies in adequate detail?), there was no clear indicator to distinguish “Partially Yes” and “Yes”. The “Yes” option required that all the listed five components should be specified “in detail” [32]; the “Partially Yes” option also required the five components but did not require them “in detail”. There was no clear boundary for determining whether a component was “in detail” or not. We have contacted the lead author of AMSTAR 2.0 for clarification but failed to receive a response, so we rated all eligible systematic reviews that described the required components of this domain as “Partially Yes” to make a conservative evaluation [33]. This did not impact the rating of the methodological quality according to the rule of AMSTAR 2.0 [32].

The methodological quality was assessed by two well-trained authors through the Access software (Microsoft, USA), one methodologist and one anesthetist, and further checked by the lead author. Any disagreements were solved by discussion until consensus was achieved.

Our primary interests were the number of weaknesses (i.e., those assigned as “No”) and the global methodological quality of each eligible systematic review. The median number as well as the interquartile range (IQR) were used to measure the central tendency and variability [30]. Both the counts of “No” from all domains and those from the critical domains were summarized.

Our secondary interest was the potential difference in the proportion of domain-specified weaknesses stratified by some pre-specified variables. These variables included: any reporting or methodology guidance referred in the review (yes vs. no), publication date (2018–2019 vs. 2015–2017), and development of protocol (yes vs. no). Considering that the development of protocol is already a part of AMSTAR 2.0, we removed the domain and only compared the remaining 15 domains under this variable. We also compared the proportion of weaknesses of systematic reviews with meta-analyses to those without meta-analyses for applicable domains. As aforementioned, for qualitative systematic reviews, three items (items 11, 12, and 15) were not applicable, which were not compared. For the publication date, considering the time lag, we used 2018 as the cut-off as the AMSTAR 2.0 was released in 2017 [32]. The proportion difference (PD) was used to measure the potential difference since this effect estimator has been proven to be valid to deal with potential zero-events in a single group or both groups [29]. The two-tailed z-test was used for the test of statistical significance. Since all the above variables can be obtained from the systematic reviews, we expect no missing data involved.

We used the MetaXL software (version 5.3, EpiGear, Australia) to run the analyses and the Excel 2013 (Microsoft, USA) to visualize the results. The level of statistical significance was pre-specified as alpha = 0.05(two-sides).

Through the primary literature search, we obtained 18,636 records, where 1967 were identified as duplicates. For the 16,669 records that were viewed by titles and abstracts, 15,399 were excluded. From the remaining 1330 records, we identified 542 systematic reviews of interventions on safety outcomes, where 127 (19.48%) referred to surgical interventions (Fig. 1). The agreement rate was 0.74 in the full-text screening process. The details of excluded studies during the full-text screen process have been documented in our previous work [25]. According to the minimal requirement, the sample size of the current study could be sufficient.

Table 1 presents the baseline characteristics of these systematic reviews. Among the 127 eligible reviews, 96 (75.59%) were quantitative reviews (with meta-analyses), while 31 (24.41%) were qualitative reviews (without meta-analyses). Also, 81 (63.78%) were published between 2015 and 2017, and 46 (36.22%) were published between 2018 and 2019 (4 were published in issue in 2020 but online first in 2019). For the language, we identified 125 (98.43%) used English while 2 (1.57%) did not (1 Spanish and 1 French). There were 74 (58.27%) systematic reviews that referred to at least one guideline to design and conduct the study or report the results, while 41.73% failed to refer to any guideline. Only 5 (3.94%) reviews referred to a methodological guideline to design and conduct the study, and thus was not further analyzed.

In addition, 47 (37.01%) systematic reviews included randomized controlled trials (RCTs) only, 25 (19.69%) included non-randomized studies of intervention (NRSI) only, and 46 (36.22%) included both types of studies. The median number of included studies was 14 (IQR: 8 to 28); 79 (62.2%) reviews included at least 10 studies. Funding information was available in 88 (69.29%) systematic reviews, where 28 (22.05%) received non-profit funding, 2 (1.57%) received profit funding, and the remaining 58 (45.67%) did not receive any funding support.

Figure 2 presents the methodological quality of each domain of the 127 systematic reviews. For the 16 domains, only 4 (1, 4, 8, and 16) in most (> 80%) of the systematic reviews showed no methodological weakness. Of the four domains, 1 (domain 4) was a critical domain. In addition, 6 domains with about a half to two-thirds of the systematic reviews showed no methodological weakness; they were domains 5, 6, 9, 11, 14, and 15, where 3 (9, 11, and 15) were critical ones. For the remaining 6 domains (2, 3, 7, 10, 12, 13), most (70%) of the systematic reviews had methodological weaknesses. Again, 3 (2, 7, 13) were critical domains.

In terms of global methodological quality, none of the 127 systematic reviews could be rated as “high”. In contrast, they were either rated as “low” (n = 18, 14.17%) or as “critically low” (n = 109, 85.83%). The median number of methodological weaknesses of these systematic reviews was 8 (IQR: 6 to 9), of which 4 (IQR: 2 to 4) were critical weaknesses. This meant that half of the systematic reviews had 8 or more methodological weaknesses in total, and 4 or more for critical ones. Similarly, 75% of the systematic reviews had 6 or more methodological weaknesses (Fig. 3).

Figure 4 shows the comparisons of the pre-defined factors on the proportion of methodological weaknesses. In general, the systematic reviews that referred to any reporting guidelines (e.g., domain 13, PD = -0.22, 95% CI: − 0.39, − 0.06; p = 0.01), developed a protocol in advance (e.g., domain 6, PD = -0.20, 95% CI: − 0.39, − 0.01; p = 0.04), and applied a quantitative analysis (e.g., domain 14, PD = -0.25, 95% CI: − 0.44, − 0.06, p = 0.01) tended to have less methodological weaknesses in some domains. However, for some domains, there were no obvious improvement (e.g., domain 8, PD = 0.04, 95%CI: − 0.14, 0.21; p = 0.68; with protocol vs. without). In addition, more recent systematic reviews showed slightly fewer methodological weaknesses, but in most of the domains, the difference was not significant from both clinical and statistical perspectives.