Introduction
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach is a system for rating the quality of a body of evidence in systematic reviews (SRs) and other evidence syntheses, such as health technology assessments, and guidelines and grading recommendations in health care [
1]. GRADE approach offers a transparent and structured process for developing and presenting evidence summaries and for carrying out the steps involved in developing recommendations. It can be used to develop clinical practice guidelines and other health care recommendations (e.g. in public health, health policy and systems and coverage decisions), and is becoming increasingly popular among guideline developers and systematic reviewers [
1]. Up to date, more than 110 organizations from 19 countries around the world have endorsed GRADE approach, including the World Health Organization, the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines, the World Allergy Organization, and the World Society of the Abdominal Compartment Syndrome and the Cochrane Collaboration [
1]. In addition, it is also becoming an international standard for judging the evidence in SRs and clinical guidelines [
2]. GRADE approach differs from other appraisal tools for three reasons: (i) because it separates quality of evidence and strength of recommendation, (ii) the quality of evidence is assessed for each outcome, and (iii) observational studies can be ‘upgraded’ if they meet certain criteria [
3]. Thus, appropriate application of the GRADE approach can provide quality of evidence and strength of recommendations that is explicit, comprehensive, transparent, and pragmatic.
Systematic reviews (SRs) attempt to identify, select, synthesize, and appraise all high-quality research evidence relevant to a well-honed question. There have been many studies based on SRs, such as the GBD database [
4], the World Cancer Research Fund/American Institute for Cancer Research report [
5], as well as dietary guidelines [
6]. As far as we know, untreated urological conditions are the major burden of patients all over the world. So far, many studies have summarized the factors affecting the urinary system and the prognosis of the urinary system disease using SRs.
However, no studies to date have assessed how SRs in urology and nephrology journals have used the GRADE approach to evaluate the certainty (or quality) of evidence. Therefore, the purpose of this study was to identify and describe all relevant SRs that use the GRADE approach to evaluate the outcome-specific certainty of evidence published from 2016–2020 in the top 10 urology and nephrology journals with the highest impact factor according to the 2020 Journal Citation Reports (JCR), and to summarize and present the GRADE-specific information, such as the outcomes rated, the number of primary studies, the exposure category, the use of summary of findings tables, the total number of down- and up-grading domains, while also considering the study design (SRs of randomized-controlled trials RCTs vs. non-randomized studies NRSs).
Methods
Search strategy
SRs published in the top 10 urology and nephrology journals with the highest impact factor according to the 2020 JCR between 1 January 2016 and 31December 2020 were identified by searching in the PubMed database (Supplementary Appendix
1).
Selection of documents
The SRs were included when the following criteria were met: (1) SRs utilizing the GRADE approach to assess the certainty of evidence. We excluded the following SRs: (1) a modified version of the GRADE approach was applied to assess the certainty of evidence; and (2) failure to provide details of the GRADE approach evaluation process and results.
First, two reviewers (SZ, S-X L.) independently screened identified articles by title and abstract. Only specific irrelevant articles were excluded at this stage. Second, we independently obtained and checked all potentially relevant articles for final inclusion by two reviewers (SZ, S-X L.). Selection disagreements were resolved by discussion or consultation with a third author (Q-J W.).
We extracted the following information for all identified SRs: year of publication; journal name; number of primary studies included; type of studies included RCTs vs. NRSs (including, non-randomized intervention, case–control, cohort, and cross-sectional studies) vs. combination RCTs/NRSs; number of participants; description of intervention(s)/exposure(s); number and types of outcome(s) and comparison(s) rated; category of certainty of evidence ratings (high, moderate, low, or very low); meta-analysis conducted (yes vs. no); summary of findings table reported (yes vs. no); number of down- and/or up-grading (count of the respective downgrading/upgrading domain used at the outcome level); and reasons for down- and/or up-grading. The one reviewer (SZ) extracted the study characteristics of the SRs, then one reviewer cross-checked all data (S-X L). We extracted all downgrading factors listed in the SRs of RCTs and NRSs according to the study design. There were five SRs with pooled RCTs and NRSs in which the combined evidence was rated [
7‐
11].
We referred to minimum criteria proposed by GRADE working group when rating the certainty of evidence in this GRADE methodologic survey, which refers to the evidence that was assessed and the methods that were used to identify and appraise that evidence were not be clearly described for SRs that were assessed in our study.
Discussion
Summary of findings
This is the first study to evaluate the application of the GRADE approach in SRs published in the top 10 urology and nephrology journals. In general, there are relatively few urology and nephrology SRs using GRADE approach to rate the certainty of evidence, but an increasing trend in the level of implementation was noted in the past 2 years [
9,
11,
17,
19‐
29,
31,
33,
40‐
43,
45‐
51,
53,
55]. We identified 49 SRs that rated the outcome-specific certainty of evidence. Overall, a low and very low certainty of evidence accounted for 32.1% and 33% of the individual outcomes, respectively. In addition, the certainty of evidence was downgraded most of for RoB and imprecision among SRs of RCTs and NRSs.
Strengths and limitations
There were several strengths in our study. First, we have searched all published SRs that utilized the GRADE approach among the top 10 urology and nephrology journals from 2016–2020. A wide range of relevant information was extracted, such as the classification of outcomes assessment, and the number and reasons for down- and up-grading domains. Second, this is the first study to summarize and present the GRADE-specific information on SRs published in urology and nephrology journals, which can provide essential information for follow-up research. There were also some limitations in our study. First, our results are limited due to the risk of selection bias. We only included the top 10 urology and nephrology journals with the highest impact factors within 5 years (n = 60). Therefore, we failed to obtain information from medical journals with lower impact factors. Nevertheless, the journals with the highest impact factor are persuasive, which can provide a basis for future clinical research. Second, because there was no clear and definite research protocol to follow, we adopted the PRISMA framework to report our findings. In fact, the PRISMA framework can provide comprehensive content and is often used in SRs and meta-analyses. Third, our study was based on a descriptive examination of the application of the GRADE approach in urology and nephrology SRs rather than to determine if the SR authors correctly followed guidance issued by the GRADE working group to rate the certainty of evidence. In the process of data extraction, we found that some authors did not follow minimum criteria of GRADE approach in the evaluation. Specifically, the certainty of evidence in some SRs was upgraded due to a low RoB, narrow confidence interval, a very low P value, and/or mild statistical heterogeneity rather than undergoing evaluation according to the GRADE approach domains for upgrading. Therefore, future research should have a priority for focusing on the optimal use of the GRADE approach, so that those applying the approach can have awareness brought to the main issues and difficulties faced when applying GRADE approach, to be able to correct them. Finally, this report does not address or assess any potential time trends. It is possible that the adoption of GRADE approach has increased and improved over time.
Findings from other studies
The GRADE approach has been used in the fields of urology and nephrology during the recent 5 years. KDIGO Working Group formulated the scope of guidelines and graded evidence according to the GRADE approach, which now serves as the practice standard for KDIGO [
67]. Similar to the findings in our study, the level of evidence was very low in most cases in the study conducted by Zare and colleagues [
68]. The most frequent limitation involved indirectness due to the limited number of studies for each pair. In addition, the evidence quality was mainly downgraded due to the RoB and inconsistency. Moreover, presenting comprehensive evidence from RCTs and NRSs, as in our study, was similar to the Cuello-Garcia et al. study [
69], which indicated that most respondents presented integrated data separately from RCTs and NRSs, either in a single summary of findings table or a standalone table. Compared to our study, these studies or guidelines merely indicated the use of the GRADE approach when rating the certainty of evidence or the preferences of experts when integrating RCTs and NRSs.
Implications for broader research
Although the GRADE approach has been used for RCTs and NRSs in the fields of urology and nephrology, there are still some challenges in evaluating the certainty of evidence in NRSs with the GRADE approach. On the basis of the GRADE approach, the quality grade for an aggregate of RCTs would begin at the high entry level. A collection of observational studies would begin at the low entry level and evidence from other study designs, such as case–control studies, would begin at the very low level [
70]. The GRADE approach, especially with respect to RoB assessment, is challenging and could lead to excessive downgrading. GRADE approach users may inappropriately double calculate the confounding and selection bias risk by downgrading the initial body of evidence to low, followed by further downgrading owing to unknown confounders in observational studies. In our study, in addition to the initial downgrading of two levels according to the study design, 53% of the SRs based on NRSs were further downgraded due to RoB.
The GRADE approach requires separate evaluations of RCTs or NRSs by any validated tool, but the specific method is not recommended because the selection of tools depends on the context [
71]. With the emergence of tools that use the concept of target trials as a reference point, such as risk of bias in non-randomized intervention studies (ROBINS-I), the initial certainty of evidence is also considered high for bodies of evidence from NRSs [
72,
73]. Several SRs have used the ROBINS-I tool in the field of nephrology thus far [
74‐
76]. ROBINS-I can more systematically and accurately assess the RoB in NRSs to avoid extensive downgrading [
73]. In general, the purpose of integrating rigorous RoB assessment into the GRADE approach, such as the application of the ROBINS-I tool in the SRs based on NRSs, is to improve the trustworthiness and credibility of the specific evidence [
73]. To accurately rate the certainty of evidence, it is recommended to follow the current standards of RoB assessments, such as the ROBINS-I and the RoB tool by Cochrane [
73,
77].
Conclusion
The GRADE approach provides a system for rating quality of evidence and strength of recommendations that is explicit, comprehensive, transparent, and pragmatic. The GRADE approach is increasingly adopted by professional organizations worldwide. Our study demonstrated that the GRADE approach is not widely used (only 13.5% (60/445) of SRs reported using GRADE approach), within urology and nephrology SRs in top 10 journals. Thus, future research should focus on the optimal use of the GRADE approach by following the criteria proposed by the GRADE working group.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit
http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (
http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.