Advancing guideline quality through country-wide and regional quality assessment of CPGs using AGREE: a scoping review

A predefined search strategy was used to conduct a comprehensive search in the following databases: Embase (Ovid), Medline (Pubmed), Epistemonikos, and grey literature (Web of Science grey literature, greylit.org, and contacting key experts). The search was conducted on 5 October 2021. Studies published in any language were included until full-text stage. The full Medline and Embase search strategies are listed in Additional file 1: Appendix B.

We exported the retrieved records into a Mendeley Library and subsequently uploaded it to the Rayyan web platform [32] and removed duplicates. Two reviewers (MMA, SS) screened titles and abstracts independently for assessment against the inclusion criteria for the review. Potentially relevant records were retrieved in full, and citation details imported into a Microsoft Excel sheet. A single reviewer (MMA) assessed the full text of selected records in detail against the inclusion criteria. At full-text screening stage, only English (or studies translated into English), and Spanish studies were included. Reasons for exclusion of sources at full text screening stage are reported in the table of excluded studies (see Additional file 1: Appendix C). Any disagreements between the reviewers at each stage were resolved through discussion.

A single reviewer (MMA) extracted data from most included records using a data extraction tool (created in Microsoft Excel), assisted by a reviewer extracting the four Spanish records (IF) and checked by another reviewer (SS). This tool was piloted on a small sample of possible included studies, identified in a previous review. Only English and Spanish records were included, as the study was limited in its access to translation services. Data extracted included study types and methodology; characteristics of included CPGs; and AGREE tool use including domain and overall score results. Included synthesis studies were the units of analysis, and simple descriptive statistics was used to conduct the analysis using STATA 14.

Categorical variables were described as percentages; whereas continuous variables were described by means and standard deviations (sd) where data was normally distributed, otherwise reported as median and interquartile range. Data normality was determined graphically and using the skewness-kurtosis test. Studies that calculated median overall scores for AGREE domains were converted to a mean score, as recommended by Hozo et al.      [33]. This allowed one standard summary statistic across domains. Regions were measured according to United Nation Sustainable Development Goals (SDG) regional groupings [34], however other regional groupings were also considered. SDG regions were chosen, due to the meaningful geographical presentation of different regions with comparable income status.

AGREE scores were analysed across subgroups into one of the seven SDG regions. A list of included studies is found in Additional file 1: Appendix D.

Fifty-seven studies were included in the analysis. The median number of included CPGs per study was 20 (IQR 44) and the median year-range of included CPGs was 8.5 years (IQR 6). The general aim of most studies was to describe and determine quality (or lack thereof) of CPGs, in either specific topical areas or country-wide; to enhance future CPG development and/or implementation of current CPGs. This included identifying areas of variability and vulnerability to speak to compliance and conformity. Most studies then used a cross-sectional design and topical purposeful sampling and were published by academic societies or researchers. However, a substantial number (n = 14) of studies evaluated all country-wide or regional CPGs. Most studies (n = 34, 60%) only searched local/regional databases and grey literature for including CPGs, whilst 20 studies (35%) searched both locally and internationally (including for example, Medline and guideline clearing houses). The number of included studies increased as the year of publication range increased, mostly published after 2019 (Table 1). One study only utilized the overall assessment and did not report domain scores and one study assessed one domain only.

Overall domain scores were low for Rigor of development (ROD), Applicability, and Editorial independence (EI); moderate for Stakeholder involvement; and higher for Scope and purpose and Clarity and presentation (Table 3). Figure 2 depicts the low global scores for ROD and EI per SDG region; simultaneously noting the low overall range of a maximum of 62.1% for ROD and a maximum of 46.3% for EI.

Figure 3 looks broadly at domain ROD, in relation to Gross Domestic Product (GDP) per capita [35], country-income profile [36], and population size. Here, there is a seemingly linear relationship between the ROD quality score, GDP per capita and country-income status. Lastly, we did not identify any studies representing low-income countries.

This scoping review provided a global view investigating the heterogeneity of CPG synthesis studies across different countries and regions, as noted in other studies [27, 37]. Despite the variation in quality, included CPG synthesis studies (and their included CPGs), were consistently described and adequately assessed and reported with the AGREE tool. This highlights the importance and value of quality assessment tools that are straightforward to use. Characteristics of these synthesis studies, mostly performed by researchers from the academia, were uniformly consisting of cross-sectional reviews where topical sampling was performed. Though high- and middle-income countries and regions were well represented, there were no synthesis studies from low-income countries. Generally, three AGREE domains of Rigor of development, Applicability and Editorial independence had the lowest domain scores, and this was most prevalent in middle-income countries and regions. Editorial independence repeatedly scored as the lowest domain, even in countries that had acceptable scores for other domains (including Rigor of development). Despite this, many AGREE domains across geography had either moderate or high-quality scores, and the tendency to appraise country-wide or regional CPGs appear to be increasing.

Regions of Eastern and South-Eastern Asia, and Europe and Northern America were the best represented in terms of quantity of studies included. This was recently seen in a living systematic review of Covid-19 rapid guidelines, where 45% of guidelines originated from high income countries (HICs) and no guidelines from low-income countries [38]. The included CPGs were either topical or all country-wide, regional, or governmental CPGs; mainly initiated by the medical society, corresponding to findings of previous reviews [16, 39]. Similar reviews found that high-quality guidelines were more often produced by government-supported organizations and/or within regulated, coordinated programmes [21, 40‐42], in comparison to guidelines produced by professional societies.

Regarding the use of GRADE methods, either in evidence synthesis or as part of the evidence to decision process, only 11 synthesis studies (19%) reported using any GRADE methods, two of these primarily assessing GRADE adherence. Lack of reporting of GRADE and other methods of judging the certainty of evidence, along with how evidence is used to develop recommendations, is concerning. Reviewers of CPGs are encouraged to highlight the importance of how these evidence-to-decision processes evolve, to strengthen guideline development and assessment in the future. Regional quality assessments have shown that there is a slow but increasing incorporation of these requirements [43, 44]. Online tools, such as GRADEpro [45], can make the process transparent, faster, and more user-friendly in a fast-paced age.

Reciprocally, the low reporting rate of guideline development methods and absence of Covid-19 or rapid guidelines included in this scoping review, could signify areas of much needed growth. Through multiple online high-quality/tech savvy platforms created for the Covid pandemic, the guideline community can be inspired to make use of these types of resources to further advance guideline quality. This can assist in processes like guideline adaptation, that aim to reduce research waste and duplication of effort; by using living reviews and guideline repositories [46], recommendation mapping [47], and guideline development resources such as produced by e-COVID [48]. A future research focus on adaptation methodology such as GRADE Adolopment and others [49], can be further supported in LMICs to build capacity for guideline development in these regions.

The Inclusive Internet Index is an example of a starting point to plug LMICs into these resources. As the 2021 report [50] indicates, there is still a ‘digital divide’ preventing most LMICs from accessing adequate internet supplies, where sub-Saharan Africa was shown to have the most constrained connectivity globally. Future research can involve investigating the challenges of LMICs on internet connectivity, especially regarding uptake and implementation of CPGs. This can include investigating the uptake of evidence-to-decision online tools, and how this translates into CPG quality.

AGREE tool use was well reported where most synthesis studies used a recommended two to four assessors per CPG, reported on interobserver agreement and assessed all 6 domains; similar to previous reviews [16, 39]. The idea of rating the overall quality of CPGs has been the subject of much debate. The AGREE tool recommends focusing on all six domains and not solely using the overall assessment to judge quality. Several authors have suggested different thresholds for this overall score, mostly suggesting using the strength of the Rigor of development domain. Two recent studies attempted rating the importance of the different AGREE domains on the overall assessment, indicating that Rigor of development, Clarity of presentation, Applicability, and Editorial independence had a significant influence on the overall quality assessment [37, 51]. Almost three quarters (74%) of the synthesis studies included in this scoping review, used this overall assessment to define its CPGs’ quality, and this is similar to the finding of two other publications [51, 52]. Those opting to use this overall assessment used multiple ways to modify it, similar to a review where only 23% of included studies provided clear criteria for generating the cut-offs applied [53]. This could indicate that several users would welcome an explicit distinction between high- and low-quality guidelines [5]. The important domain of Rigor of development, particularly used in the overall CPG quality consideration, is often scored as low [8, 10, 21], similarly found in this study. Notably, domain Editorial independence is becoming increasingly scrutinized as important for guideline quality [24]. In fact, Molino et al. showed that higher quality CPGs reported its funding sources 10 times more often than those of lower quality [21]. Multiple recent studies reaffirmed the low scores across guidelines for this domain [54‐56]; and this gap is crucial in LMICs especially [57], however, it can be argued that it is equally important for all countries world-wide.

The scoping review design used in this study allowed us to attain a global overall view. This review focused on studies that used the AGREE tool for quality assessment and cannot comment on synthesis studies of guidelines using other appraisal tools. Additionally, there might be included studies that applied AGREE to other types of documents (such as end-user guidance documents or broad summary documents). There has been an indication that other types of ‘guidance documents’ have lower AGREE scores than true CPGs [58], and this could have overestimated the results of this study. This scoping review cannot conclude that CPGs from HICs are of higher quality than those of LMICs, as we are limited by the availability of studies summarizing the available guidelines per regions. In addition to this, the CPGs assessed for each region was often a sub-sample of all available CPGs in that region. Importantly, regional or national CPG synthesis studies, may be preferentially submitted to local journals, without indexing in international databases. Subsequently, these researchers may be inclined to not publish at all, simply posting results on association or organizational websites, resulting in potential missed studies. Cumulatively, this could have influenced the results of this scoping review, especially the associations with world regions and income groups, as LMICs could be more inclined to solely make use of local distributions of quality assessments of CPGs.