Quality assessment of Clinical Practice Guidelines (CPG) for the diagnosis and treatment of inflammatory bowel disease using the AGREE II instrument: a systematic review

A systematic search was performed up to January 2022 to look for CPG on the diagnosis and treatment of IBD. CPGs were searched on databases (MEDLINE - PubMed, EMBASE, CINAHL, LILACS), professional societies (CAG, British Society of Gastroenterology, AGA, Brazilian Society of Gastroenterology), registries and guideline developers’ websites (NICE, SIGN). The full search strategy is detailed in Additional file 1.

We included: 1.- CPGs with specific recommendations for the diagnosis and treatment of IBD, both for Crohn’s disease (CD) and ulcerative colitis (UC); 2.- CPGs on IBD that included pediatric, young, adult, and elderly populations; 3. - CPGs that provide the full search strategy that was conducted; 4.- CPGs that mentioned the process how they reached recommendations; 6.- CPGs published without date restriction until January 2022; and 6.- Last published available version of CPGs. The following documents were excluded: 1.- CPGs exclusively dealing with other clinical scenarios such as diagnostics (e.g. endoscopic, imaging), nutrition, immunological or surgical interventions for IBD; 2.- secondary publications (e.g., systematic reviews and meta-analyses) and 3.- abstracts from CPGs.

Five reviewers working in pairs (DH, CMG, PA, RZ, RV) independently peer-screened the guidelines by title and abstract following the above inclusion and exclusion criteria. If the inclusion criteria were met, the full-text article were retrieved and screened by pairs for eligibility. All the screening process was performed using Rayyan (Rayyan Systems Inc) [21]. Two reviewers independently extracted the following data for each CPG: title, year of publication, submitting organization, type of funding, method used to collect evidence, number of sources documented, methods used to assess the quality and validity of the evidence, methods used to formulate the recommendations, country, and language. In case of disagreement, a third reviewer (VA, DSR) was involved.

The AGREE II instrument [18‐20, 22] was used to evaluate the quality of the included CPGs. This instrument provides criteria for assessing the quality of the clinical practice guidelines through 23 items or questions, divided into 6 domains or categories; including: 1.- scope and purpose, 2.- stakeholder involvement, 3.- rigor of development, 4.- clarity of presentation, 5.- applicability, and 6.- editorial independence. The first domain evaluates the general objective of the CPG, specific health aspects and the target population; the second domain refers to the degree to which the guideline has been developed by the appropriate stakeholders and represents the views of intended users; the third domain refers to the process used to gather and synthesize evidence, the methods used to formulate and update recommendations; the fourth domain focuses on the language, structure and format of the guideline; the fifth domain refers to barriers and facilitators to CPG implementation, strategies for its adoption and resource considerations; and finally, the sixth domain is about the formulation of recommendations, to understand whether they are biased by conflicts of interest [19].

Each of the 23 items or questions is classified on a 7-point Likert-type scale, 7 being the maximum score corresponding to “strongly agree” and 1 the minimum score corresponding to “strongly disagree”.

For the global guideline evaluation, we used a 3-point scale: 1 “not recommended”, 2 “recommended with modifications” and 3 “recommended”. Six reviewers (DH, CMG, PA, RZ, JAF, RV), with clinical and methodological expertise, independently peer-scored each of the 23 items of the 6 domains of the AGREE II instrument for each CPG that was included. In case of disagreements with the assessment, a consensus was reached with the support of a third reviewer (AV, DSR).

A descriptive analysis of the CPGs was performed using the general characteristics of each CPG from the extracted data. To calculate the score for each domain of the AGREE II tool, all item scores were summed up and the total value was standardized as a percentage of the maximum possible score for that domain, using the following formula:

With this method, the standardized score for each domain ranged from 0 to 100%. The result of the standardized score for each domain for all the guidelines is presented through the mean, median, first quartile (Q1), third quartile (Q3), interquartile range (IQR) and a boxplot. The degree of agreement between reviewers was assessed through the intraclass correlation coefficient (ICC) with a 95% confidence interval (CI). To visualize and compare the mean AGREE II scores obtained by the 26 CPGs assessed in this study, we generated a hexagonal radar graph where each domain is represented on a radial axis centered at 0 and the maximum score of each domain corresponds to each vertex of the hexagon. Finally, for the analysis of quality change over time, Student’s t-test was used to compare the means and categorize the CPGs into two periods: 2012 to 2017 and 2018 to 2022. Data analysis was performed in the statistical software RStudio v.1.4 [23] using the libraries ggplot2 [24], irr [25], tidyverse [26] and Table 1 [27].

Eight thousand seven hundred twenty-three records were retrieved from the search strategy and 8165 remained after deduplication. 203 records were subsequently screened by full-text, of which 26 CPGs were included for data extraction after meeting the inclusion criteria (Fig. 1). Details on the characteristics of the included CPGs are shown in Table 1 [28‐53].

Of the 26 included CPGs, four were from the United States (15.38%) and four were developed by an international collaboration (15.38%); three were from the United Kingdom, three from Canada and three from Japan (11.53% each), two were from Brazil and two from Mexico (7.69% each); one was from Germany, Israel, South Korea, the Netherlands and Poland (3.84% each). Included guidelines were published between 2012 and 2021 (see Table 1).

Three of the 26 guidelines focused exclusively on the pediatric population while the others were mainly focused on adults [29, 33, 51]. In terms of the scope of the CPGs, 22 dealt with diagnosis and clinical management [28‐30, 32‐41, 43‐45, 47‐49, 51‐53], two with the use of biologic drugs only [42, 46], one with surgical management in the emergency setting [50] and one with the surgical management of ulcerative colitis [31]. All guidelines were considered evidence-based according to our a priori criteria.

Eighteen guidelines (69.23%) used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology to assess the quality of evidence and grade the strength of recommendations. Seven guidelines (26.92%) used the Oxford Centre for Evidence-Based Medicine criteria, and one guideline (3.84%) used a self-grading system to assess the quality of evidence (Table 1).

The agreement between the 6 reviewers was moderate with an ICC of 0.74 (95% CI: 0.36-0.89, p-value = 6.83e⁻⁴). A summary of the ICCs achieved by each pair of reviewers is shown in Table 2.

Figure 2 shows a boxplot summarizing the statistical analysis of the standardized scores for each domain assessed with the AGREE II tool. In addition, Table 3 shows the standardized scores for all domain assessed in each clinical practice guideline.

With respect to quality change over time, no statistically significant differences were found for the means of the standardized scores for each AGREE II domain between the guidelines published during the 2012-2017 period and those published between 2018 and 2022 (Table 4).

This review showed that the evaluated IBD CPGs had an acceptable quality based on the AGREE II instrument since 7 out of the 26 evaluated guidelines were “recommended”, 15 were “recommended with modifications” and only 4 were “not recommended”. The domains with the highest scores were “clarity of presentation” and “scope and purpose”, which reached values over 60%, indicating that most of the assessed guidelines had well-defined general and specific objectives, the population to which the guideline was intended to apply was well defined, and the recommendations were clearly described and identifiable. Rigor of development was the domain that received the third best score with 69.95%; this domain could be argued to have the greatest effect on the quality of a clinical practice guideline, since it has to do with the entire process used to formulate and construct the recommendations and it is the one that comprises the most items within AGREE II for its evaluation [54]. We consider that a score over 60% is more than acceptable for “rigor of development”, which achieved this score due to most guidelines were partly penalized for being unclear with the description of external experts’ assessment and for not having an explicit updating statement.

The domains “stakeholder involvement” and “editorial independence” obtained scores slightly over 60% (60.90 and 62.02%, respectively; Fig. 3), which indicates that the views and preferences of patients still need to be considered when the CPG is drafted and that an expert methodologist/epidemiologist should be included in the guideline drafting group. In addition, both domains achieved low scores due to the limited information most guidelines provided in terms of funding and its influence on the guidelines’ content, as well as the lack of detail they included regarding conflicts of interest and how these conflicts were dealt. Considering these limitations on the development of CPGs could contribute to their improvement.

The “applicability” domain was the worst scored domain in this review with an average score of 26.60% (Fig. 3), well below the 60% cut-off point for this domain. The main reason for this is that most guideline developers do not fully consider guideline’s implementation in terms of facilitators and barriers for guidelines’ applicability or they do not fully consider the resources and tools that are available in a specific context. We also noted that most of the guidelines did not consider the economic impact of their recommendations on resources and health budgets, for example, most guidelines did not include health economists in the guideline development group or did not perform cost-benefit analysis. The limitations and omissions that have been observed in the included guidelines restraint the translation of these documents into clinical practice, thus hindering its operability.

Regarding quality change over time, this study failed to demonstrate statistically significant differences between guidelines published during the 2012-2017 period versus guidelines published between 2018 and 2022 (see Table 4) for any domain covered by AGREE II. This finding may be due to the small sample size in this study, which is associated to the specific inclusion criteria applied in the selection of CPGs as well as the large variety of CPGs for IBD (clinical, surgical, preventive, etc.) we encountered when screening. In addition, the time ranges we compared were too short since guidelines’ development in terms of IBD and our study’s criteria has been an early activity. However, one point to highlight is the implementation and dissemination of the GRADE methodology in the development of guidelines, especially in those produced in the last 4 years; our study found that 17 out of the 26 included CPGs had used this methodology as a framework for grading the evidence and formulating their recommendations.

While this review is not the first to evaluate clinical practice guidelines on inflammatory bowel disease, it is the first to evaluate a large sample of CPGs as there was no date restriction in its search, which gave us a much broader picture of what has been produced in the past and current time. Thus, in line with other reviews of CPG for IBD conducted by other investigators, and addressing different contexts of inflammatory bowel disease, the domains with the highest scores were “clarity of presentation” and “scope and purpose” and the domains with the lowest scores were “stakeholder involvement” and “applicability” [55‐57]. These results are also similar to previous CPG evaluations for other clinical-surgical areas such as interventional radiology, pediatrics or dermatology [58‐60].

In addition, other studies that investigated quality changes over time for clinical practice guidelines in other specialties did not find evidence of significant changes in quality in the different evaluated periods of time [61‐63]. These results are consistent with the findings of this study. However, studies by Bhatt et al. [64] for pediatric type II diabetes CPG and Acuña-Izcaray et al. [65] for asthma CPG, found statistically significant differences in quality over time for the selected periods for each individual domain, while a statistical significance has not been found for all domains at the same time.

Although a strength of our systematic review was the broad and exhaustive approach of our search – carried out in databases, compiling entities and guideline developers, with a sensitive strategy designed for this purpose – it is possible that our review may have missed some CPGs that were not adequately indexed or that dealt with other contexts related to inflammatory bowel disease. Likewise, our study only included CPGs published in English or Spanish, factors that could have contributed to a potential selection bias.

Likewise, having chosen CPGs with well-defined inclusion criteria, it is likely that our results have overestimated the score obtained by selecting guidelines that would score higher than the entire possible universe of CPGs for IBD. Therefore, our conclusions acquire more relevance when evaluating this type of guidelines.

On the other hand, although the degree of agreement reached by the reviewers was moderate (ICC = 0.74), this may be due to the fact that the AGREE II instrument weights each item with a 7-point Likert-type scale, where only the extreme values of this scale are well defined, but it is prone to subjectivity for intermediate values 3, 4 and 5 on the scale. As our research had a large number of reviewers (six), reaching a higher value for the intraclass correlation coefficient (ICC) to improve reliability was difficult. However, we consider that the value achieved does provide adequate reliability [66].

In addition, since the implementation of the AGREE II tool in 2010, it has become the most widely used and popular resource for assessing the quality of CPGs, choosing a cut-off point above which a guideline can be defined as having good quality is subjective and this selection will depend on the context in which the review is being performed. As Brouwers et al. [67] noted, “there is no evidence that if a guideline exceeds a certain score, the recommendations are easier to adopt, or improve processes of care, or lead to better patient outcomes than guidelines that do not achieve that score”.⁽ [67]^{, p.195)} That is, the validity of the overall assessment may be limited, as there are no clear rules yet on how to weigh the different domain scores to make a decision on whether or not to recommend guidelines.

Overall, this study determined that the quality of clinical practice guidelines for the diagnosis and treatment of inflammatory bowel disease is acceptable and that there is still room for improvement, especially in terms of stakeholder participation (inclusion of patients, expert methodologists/epidemiologists) and applicability (enablers, barriers, optimization of resources, external review). It is desirable that guideline developers consider these shortcomings in the future for the overall improvement of guidelines’ quality to reduce clinical practice heterogeneity in IBD.