Quality assessment of clinical practice guidelines for chronic kidney disease: a systematic review

A systematic search was conducted in the main databases of organisations that develop or compile CPGs including: the Turning Research Into Practice (TRIP) database, National Guideline Clearinghouse (via https://​www.​ahrq.​gov/​topics/​national-guideline-clearinghouse-ngc.​html), National Institute for Health and Care Excellence (NICE), Scottish Intercollegiate Guidelines Network (SIGN), Guidelines International Network (G-I-N), and the National Health System library of Spain. An additional search was conducted in the international nephrology societies or associations webpages of each country or region. Additionally, a systematic search was performed in MEDLINE and EMBASE, investigating the medical terms headers related to CKD, applying CPG filters.

Three authors independently (JC, RV, KS) assessed the quality of each CPG, under the guidance of an expert in Investigation Methodology (DO). The AGREE II instrument [5], which consists of 23 items organized in six domains, followed by 2 items of global score (overall assessment), was used. Each domain embraces a unique dimension in the CPG quality: scope and purpose, stakeholder(s) involvement, clarity of presentation, rigour of development, applicability, and editorial independence. The overall assessment includes a score for the general quality of the CPG and an assessment whether it is recommended for use in clinical practice. Each item was assessed using a 7-point scale (from 1 “strongly disagree” to 7 “strongly agree”), even if it was not applicable. The domain scores was expressed as a percentage over the highest possible score using the following formula:

The final score is the sum of the total scores assigned to each domain by each reviewer. The maximum possible score is 7 (strongly agree), multiplied by the number of items in the domain and the number of reviewers. The minimum possible score is 1 (strongly disagree) multiplied by the number of items in the domain and the number of reviewers. For the overall assessment, a score from 1 to 7 was consigned, as well as a recommendation regarding the use of the CPG in clinical practice classified as: recommended, recommended with modifications, and not recommended.

The degree of agreement between reviewers was determined by the measurement of intraclass correlation coefficient (ICC) and its 95% confidence interval. An ICC of > 0.9 was considered “very good”, between 0.71 and 0.9 “good”, between 0.51 and 0.7 “moderate”, between 0.31 and 0.5 “fair”, and < 0.31 “poor” or “non-existent”.

The degree of agreement between the three reviewers was good, with an ICC of 0.88 (95% CI: 0.67–0.96) for the overall score. The CPG quality score varied from 3 to 7 and the reviewers recommended five CPGs (38.5%; CPG scores between 5 and 7: Scotland, Malaysia, Australasia, USA and United Kingdom). Six CPGs were recommended with modification (46.1%; CPG scores between 4 to 5: Mexico, Argentina, Chile, England, Italy and SLANH), and two CPGs were not recommended (15.4%; CPG scores between 3 to 4: Spain and Netherlands). The average score of each domain of all included CPGs and their respective recommendation can be observed in Table 2, and Fig. 2.

This domain refers to the CPG general objective, the specific health aspects, and the population to whom this CPG is intended (items 1–3). The overall average score was 80.1% (median = 88.9%; interquartile range (IQR), 50 to 98.2%). Furthermore, 61.5% of the included CPGs had an average score of 80% or higher, which is considered high category. The average score of the recommended CPGs was 93, 77.5% for the CPGs recommended with modifications, and 62% for the CPGs that were not recommended.

This domain refers to the degree to which stakeholders have developed the CPG and the representation of the points of view of the intended users (items 4–6). The overall average was 58.2% (median = 59.3%; IQR, 27.8 to 94.4%). Furthermore, 15.3% of the CPGs were evaluated as high category, 15.3% as medium category, and 69.4% as low or very low category. The CPGs that were recommended had an average score of 74.4%, the CPGs recommended with modifications scored 53.4%, and the CPGs that were not recommended scored 39.8%.

The rigour of development domain considers the process used to gather and synthesize the evidence, the methods to formulate the recommendations, and the updating process (items 7–14). The overall average score was 64.9% (median = 68.4%; IQR, 22.9 to 93.1%). Furthermore, 38.4% of the CPGs were considered within the high category, 23% within the medium category, and 38.6% within the low or very low category. The average score of the CPGs that were recommended for use in clinical practice was 85.3%, for CPGs recommended with modifications was 61.7%, and for those not recommended 31.9%.

The clarity of presentation domain refers to the language, structure, and the CPGs format (items 15–17). The overall average was 88.9% (median = 92.59%; IQR, 62.96 to 96.3%). In total, 92.3% of the CPGs were considered within the high category. The average score of the CPGs recommended for use in clinical practice was 93.3%, for those recommended with modifications 91%, and for those not recommended 72.2%.

This domain is related to the barriers and facilitators to its implementation, strategies to improve uptake, and resource implications when applying the CPGs (items 18–21). The overall average score was 47.9% (median = 43.75; IQR, 2.78 to 87.5%). In total, 23% of the CPGs were assessed as the high category, 15.3% as the medium category, and 61.7% as the low or very low category. The average score of the recommended CPGs was 69.4%, for those recommended with modifications 45.8%, and for those not recommended 6.9%.

Finally, the editorial independence domain refers to the conflicts of interest of the panel members and the role of the funding body (items 22–23). The overall average was 62.7% (median = 59.72%; IQR, 19.44 to 94.44%). In total, 23% of the CPGs were in the high category, 30.7% in the medium category, and 46.3% in the low or very low category. The average score of the recommended CPGs was 85%, those recommended with modifications scored 55.6%, and those not recommended scored 36.1%.

When comparing the AGREE II domains between the recommended CPGs and the CPGs recommended with modifications, we found a significant difference in the stakeholder involvement (74.4 ± 14.3 vs. 53.4 ± 8.2 respectively; p = 0.013), rigor of development (85.3 ± 5 vs. 61.7 ± 11.5 respectively; p = 0.002), and the editorial independence domains (85.0 ± 10 vs. 55.6 ± 10.4 respectively; p = 0.001). Between the recommended CPGs and the CPGs that were not recommended there was a statistically significant difference in all domains. A statistically significant difference was found between the CPGs recommended with modifications compared to those not recommended in the rigor of development (61.7 ± 11.5 vs. 31.9 ± 12.3 respectively; p = 0.02) and clarity of presentation domains (91.1 ± 1.8 vs. 72.2 ± 13.1 respectively; p = 0.006).

The overall averages and the domain score of the included CGPs according to each subgroup are described as follows and resumed in Table 3.

Out of the six CPGs developed in Europe, two were recommended (Scotland and United Kingdom), two recommended with modifications (England and Italy), and two were not recommended (Spain and Netherlands). The four Latin American CPGs (Mexico, Argentina, Chile and SLANH) were all recommended with modifications. The CPGs from Asia (Malaysia), Oceania (Australasia) and USA were all recommended. When comparing the domain scores of the CPGs from Asia and Oceania with the Latin American CPGs, we found a statistically significant difference in the stakeholder involvement domain (68.5 ± 7.9 vs. 58.3 ± 1 respectively; p = 0.04), however, when comparing with the European CPGs, no difference was found (68.5 ± 7.9 vs. 57.1 ± 25.6 respectively; p = 0.57).

In Latin American CPGs, the domains with the lowest scores were stakeholder involvement, applicability, and editorial independence. Similarly, in European CPGs, the domains with the lowest scores were stakeholder involvement and applicability. When comparing Latin American and European CPGs, we found no significant difference in the AGREE II domains.

In total, three CPGs were developed by medical societies (Spain, Netherlands and USA), two were developed jointly by medical societies and organizations that develop CPGs (Argentina and SLANH) and the remaining eight CPGs by organizations responsible for developing CPGs.

For the organizations that develop CPGs and medical societies, the domains with the lowest scores were stakeholder involvement and applicability. When comparing these domains between the organizations that develop CPGs and medical societies, no statistically significant difference was found in the stakeholder involvement (64.6 ± 18.1 vs. 46.3 ± 16.5 respectively; p = 0.16) domain, however, we found a difference in the applicability domain (64.58 ± 22.9 vs. 15.74 ± 15.8 respectively; p = 0.009).

For the period between 2008 and 2011, eight CPGs (61.5%) were published, out of which two were recommended, four were recommended with modifications, and two were not recommended. In the time frame between 2012 and 2016, five CPGs (38.5%) were published, out of which three were recommended and two were recommended with modifications.

We identified general recommendations for CKD in the included CPGs: early diagnosis, investigations or diagnostic test, interventions for slowing the progression and criteria for referral to the nephrologist (Table 4). For the early diagnosis of CKD, all CPGs recommended to study CKD in patients with diabetes mellitus and systemic arterial hypertension. Furthermore, the majority of the CPGs stated that patients with cardiovascular diseases, systemic diseases that affect the kidney and family history of end-stage kidney disease must be investigated.

To evaluate the presence of CKD, all CPGs recommended estimating the glomerular filtration with the formulas based in the serum creatinine and to measure the albuminuria. The formula most used to estimate the glomerular filtration is MDRD (Modification of Diet in Renal Disease).

To avoid the progression of CKD, all CPGs recommended stabilizing the arterial pressure and using angiotensin converting enzyme inhibitors or angiotensin II receptors blockers. The majority of the CPGs recommended controlling proteinuria, stabilize weight, avoid smoking, and controlling lipids.

Eleven (85%) included CPGs reported criteria to obtain a referral to the nephrology department, this concerns especially patients with a low estimated glomerular filtration and proteinuria or albuminuria. The majority of the CPGs (73%) recommend a referral to a nephrologist when the estimated glomerular filtration is below 30 mL/min/1.73m².

In general, all CPGs have their lowest averages in the stakeholder involvement, applicability and editorial independence domains, which could partly explain why the incidence of the CKD hasn’t decreased [3]. In the review by López-Vargas et al., regarding the CPGs published between 2002 and 2011, similar results have been found [6]. The domains with the lowest average were stakeholder involvement, applicability, and editorial independence. The review of Sekercioglu et al., regarding CPGs published between 2003 and 2015 that focused in the alterations in the bone mineral metabolism in CKD, reported as well that the domains with the lowest averages (all being in the category very low) were stakeholder involvement, applicability, and editorial independence [7]. Our results are similar to those found by Gagliardi and Brouwers; they analyzed systematic reviews regarding different pathologies, which included CPGs published since 2008, finding that the three domains with the lowest scores were stakeholder involvement, applicability, and editorial independence. Among the factors associated with the applicability, there was a significantly higher average in the CPGs elaborated by groups that develop CPGs. Unlike our study, they found a significantly higher average in the CPGs that were published between 2010 and 2012. In their conclusions, they state that the applicability of the CPGs has not increased in relation to the those published before 2008 and that the cost to elaborate CPGs is not being rewarded by their applicability [24].

The findings of our review and those of others point to the same weakness in the current CPGs. Due to the weaknesses in the implementation, the recommendations with the best evidence are not being used fully in the daily practice. This, as we mentioned before, can be one of the determining factors in the increasing incidence of CKD, despite the existing recommendations in the CPGs regarding prevention. To enhance the applicability of the CPGs, it has been recommended, among other strategies, to endow clinics with instruments that facilitate the implementation of these CPGs. The idea is to simplify the recommendations to be executed and to achieve the facilitation of the application and understanding of the patients and the healthcare providers [25, 26]. In a recent study, all ideal characteristics for a CPGs’ implementation instrument were explored, and identified 12 items. Among these items, the following were highlighted: identification of target users; involvement of target users were in the tool development; and conduction of pilot-test of the tool in target users [27]. Kastner et al. found that the factors associated with the implementation of the CPGs are the ones related to the creation of the content of the CPG and the proper communication of its content [28]. In relation to the content, they consider the stakeholder involvement and the feasibility important among others, similarly to our study. Another study was executed to develop a model based in the evidence for the implementation of the CPGs of clinical practice. Based on their results, the GUIDE-M (Guideline Implementability Decision Excellence Model) was created with the purpose of helping organizations that developed CPGs to create CPGs with recommendations easier to implement, facilitate the users’ adoption and to motivate researchers for deeper investigations in the topic [29].

On the other hand, the AGREE II instrument has been used more as an instrument for assessing CPGs’ quality despite the fact that it was also created to help achieve the requirements of the development and presentation of the CPGs. In our study, we reported that only 23.1% (3/13) of the CPGs, performed a process of self-assessment or compliance verification of the requisites needed for the development of the CPG, hence this could have influence the low scores of the stakeholder involvement and applicability domains. In 2016 the AGREE Reporting Checklist was published, which seeks, among its objectives, to help the CPGs’ developers take into account each of the quality requirements proposed in AGREE II [30]; with the use of this instrument, it would be possible in future CPGs to overcome the weaknesses found in our study and in the other ones aforementioned. Within the AGREE platform, a research project is registered to create a new instrument that complements AGREE II, which has been named AGREE-REX (Appraisal of CPGs REsearch and Evaluation – Recommendations EXcellence) [31]. The purpose of this project is to create an instrument that is useful for the development, report and evaluation activities related to the credibility, optimization, reliability and implementability of the CPGs recommendations. It is expected to be ready for publication in the following years.

The general recommendation especially in low- and middle-incomes countries, is the adaptation of high quality CPGs to their context by using recommendations based in the best evidence available and focused in the early diagnosis of the CKD [32]. Ours results allow the endorsement of the CPGs developed in Scotland, Malaysia, Australasia and the United Kingdom (NICE guideline) to support other developer groups to create their own CPGs or adapt them to their context. For this last purpose, the CPG from the USA is not endorsed due to a low and very low score on the stakeholder involvement and applicability domains, respectively.

We considered that the adapted CPGs from Mexico, Argentina, Chile, Italy and SLANH have recommendations of utility in their context but they require an improvement in the domains stakeholder involvement and applicability in future updates so they can truly contribute to make an impact over the incidence of the CKD.

As far as we know, our study represents the first analysis published on CPGs quality over the last 8 years worldwide, with no restriction regarding language, focused on the early diagnosis of the CKD including recommendations based on the latest evidence. Among the strengths, we included a systematic search of the published CPGs, a high degree of agreement among reviewers, a great expertise of methodological experts in the evaluation of CPGs using the AGREE II instrument. Additionally, two of the reviewers are specialists in the area of nephrology. We highlight as a strength the fact that the nephrologist reviewers work in different continents and are native from different countries.

Among the weaknesses, we mentioned that our inclusion criteria only admitted potentially high quality CPGs, which can be a selection bias. Although no quality threshold has been established in the AGREE II instrument, we accepted as satisfactory an average score of 60% or more in the domains, which may be argued by other authors.