Quality assessment of evidence-based clinical practice guidelines for the management of pregnant women with sickle cell disease using the AGREE II instrument: a systematic review

Three reviewers independently reviewed the retrieved CPGs, based on the following inclusion criteria: evidence-based with clear and detailed documentation of the development methods; available in the English language; obtained from original sources (de novo developed); had a national or international scope; published, amended or updated between January 1, 2014, and December 31, 2018 (the search was repeated before the submission of the final manuscript to identify new relevant CPGs; those published over the last five years only (2014–2018) were included in this study (because the period between two updates was reported to range from two to five years, in several CPG handbooks). CPGs published by an organization or having group authorship in a CPG database or peer-reviewed journal were also included in this study. Only the most current version of each Source CPG was included [11, 12].

The exclusion criteria included CPGs that were published earlier than 2014, written in a non-English language, adapted from other Source CPG (s), presented as consensus or expert-based statements, and written by a single author [11].

Literature searches of bibliographic databases (Medline/PubMed and Google Scholar), EBSCO DynaMed Plus (USA) and relevant CPG databases, such as the ECRI Institute Guidelines Trust, National Institute of Health and Care Excellence (NICE; UK), Guidelines International Network (G-I-N) International guideline library, Scottish Intercollegiate Guidelines Network (SIGN; UK), and the Australian National Health and Medical Research Council (NHMRC; Australia) were performed. Moreover, we searched databases of national and international societies specializing in fields related to the topic of SCD in pregnancy, such as the American College of Obstetricians and Gynecologists (ACOG), Royal College of Obstetricians and Gynaecologists (RCOG), Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG), Society for Maternal Fetal Medicine (SMFM), Saudi Society for Obstetrics and Gynecology (SSOG), and Arab Association of Obstetrics and Gynaecology Societies (FAGOS). The keywords used were “sickle cell disease” AND “pregnancy” OR “pregnant women” AND “guideline,” “practice guideline,” “clinical practice guideline,” “practice parameter,” “guidance,” OR “recommendations” [11].

The PubMed electronic search strategy included the following: “anemia, sickle cell“[MeSH Terms] OR (“anemia“[Title] AND “sickle“[Title] AND “cell“[Title]) OR “sickle cell anemia“[Title] OR (“sickle“[Title] AND “cell“[Title] AND “disease“[Title]) OR “sickle cell disease“[Title]) AND “pregnan“[Title] AND (“pregnancy“[MeSH Terms] OR “pregnancy“[Title]) OR (“pregnant women“[MeSH Terms] OR (“pregnant“[Title] AND “women“[Title]) OR “pregnant women“[Title]) AND (“guideline“[Publication Type] OR “guidelines as topic“[MeSH Terms] OR “guidelines“[Title]) AND (Practice Guideline [ptyp] AND (“2014/01/01“[PDAT] : “2019/12/31“[PDAT]) AND “humans“[MeSH Terms]) AND (practiceguideline [Filter]). Furthermore, the PIPOH (Patient Population, Interventions, Professionals, Outcomes, and Healthcare Setting) model was used to support the CPG eligibility process [10]. Three reviewers (YA, MA, YS) independently screened titles and abstracts of the retrieved CPGs and articles that met the inclusion criteria. The screening was re-checked by three different reviewers (GE, AA, OK). Disagreements were resolved by focus group discussions after retrieval and review of the full-text articles or full CPG documents, including links to any accessible online supplementary documents or web resources. The search was repeated before the submission of the final manuscript to retrieve any new eligible CPG.

The AGREE II instrument (www.​agreetrust.​org) consists of 23 items organized in 6 domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence [13]. Each item is scored on a Likert scale (1–7). The AGREE II evaluation was guided by utilizing its online version, “My AGREE PLUS,” which supports the presence of a CPG “appraisal group” for each CPG that compiles and calculates the ratings of the items into domain ratings, and comments [13]. The four AGREE II appraisers in the study had the relevant clinical expertise in obstetrics and gynecology (YS, AA), internal medicine (GE, MA), and hematology (GE, MA); furthermore, an expert CPG methodologist (YA) was included. At the outset, the CPG methodologist conducted capacity building sessions for the reviewers via hands-on sessions on the concepts, evidence-based CPG standards, and use of the AGREE II instrument. Each reviewer scored his/her assigned CPGs, and the included CPGs was critically appraised by all the five reviewers. The CPG documents, including any updates, plus any relevant supplementary information or links to online webpages related to the CPG methods or CPG implementation tools were reviewed in full by all the appraisers. For each item, the AGREE appraisers were asked to record the justifications for their scores in the “Comments” section. Wide discrepancies between the assessors’ scores for items or questions (a difference of more than 3 between the scores) were resolved by asking those who had provided the outlying scores to re-assess the questions after discussions with the entire group. The standardized AGREE domain scores or ratings (%) were automatically calculated using My AGREE PLUS. A cut-off point of 70% was set for each AGREE standardized domain score or rating. After the appraisal, we focused on the scores of domains 3 and 5 to facilitate the filtration and final evaluation of the reporting quality of the included CPGs. Similar cut-off values have been reported in previous studies [14, 15]. In addition to the classification of the six AGREE II domains, the evidence-bases of the included CPGs and their references sections were screened for systematic reviews or meta-analyses, specifically for Cochrane reviews. We utilized the PRISMA statement flow diagram and checklist for reporting our review [16‐18]. There was no patient nor public involvement in this study.

Inter-rater reliability assessment tests (IRR) were conducted to determine the agreement levels between the raters. Percent agreement IRR was used for every question (or item) in each standardized domain in the four CPGs included in this study to assess the level of agreement among the four raters and the percent agreement of the first overall assessment (OA1) of the AGREE II instrument. Furthermore, the Intra-class correlation coefficient (ICC) was used to measure the consistency in the ratings or capacities of the datasets that were gathered as clusters or arranged into clusters, including the second overall assessment (OA2 or “recommend this CPG for use”). ICC is one of the most prevalent IRR approaches used when the number of raters is more than two. A high ICC (or Kappa; K) value (near 1) indicated a high resemblance between standards from the same set. A low Kappa value (near zero) indicated that the standards from the same set are not alike. One-Way Random analysis of variance was used SPSS Statistics, version 21 was used to analyze the data in this study. The ICC was used due to the diversity in the numerical data obtained from the groups or clusters. It helped us in detecting the reproducibility of the results and in determining how closely the peers resembled each other with regard to certain traits or characteristics. The agreement between two ordinal scale classifications was evaluated, and weighted Kappa (Quadratic Weights) was used because the data was obtained from an ordered scale.

Linear weights were used because the difference between the first and second categories had the same importance as that between the second and third categories, and so on. The agreement was quantified using the K statistic, [19, 20] where K = 1 when there is perfect agreement between the classification systems, K = 0 when there is no agreement better than chance, and K is negative when the agreement is worse than chance. The strength of the agreement based on the K value was classified as follows: < 0.20 (poor), 0.21–0.30 (fair), 0.31–0.40 (moderate), 0.41–0.60 (good), 0.61–0.80 (very good), and 0.81–1.00 (excellent) [21].