The quantity and quality of complementary and alternative medicine clinical practice guidelines on herbal medicines, acupuncture and spinal manipulation: systematic review and assessment using AGREE II

A systematic review was conducted to identify CAM guidelines using standard methods [36] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria [37]. A protocol was not registered. Eligible guidelines were assessed with the widely used and validated Appraisal of Guidelines, Research and Evaluation II (AGREE II) instrument [38]. AGREE II is a tool that assesses the methodological rigour and transparency in which a guideline is developed, and is the international “gold standard” for the assessment of guidelines. Detailed information is available on the AGREE web site [www.​agreetrust.​org]. It consists of 23 items grouped in six domains: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence.

Eligibility criteria for CAM guidelines were based on the Population, Intervention, Comparison and Outcomes framework. Eligible populations were adults aged 19 years and older with any diseases or conditions. With respect to interventions, guidelines were more likely to have been published on CAM interventions for which evidence has accumulated. We referred to a bibliometric and content analysis of CAM trials in the Cochrane Library by Wieland et al. [39] which found that the CAM therapies most commonly evaluated in trials included herbal supplements (non-vitamin, non-mineral dietary supplements or Chinese herbal medicine), acupuncture, and chiropractic or osteopathic manipulation [39]. For this study, guidelines were eligible if they specifically focused on any of these CAM therapies (category 1 – CAM-specific), or were general CAM guidelines that included at least one recommendation (for or against) at least of these CAM therapies (category 2 – CAM-general). We excluded general guidelines, which includes many hundreds and perhaps thousands, as it would have been challenging to search for and screen them for potential mention of CAM. Comparisons pertained to the assessed quality of CAM guidelines. Outcomes were AGREE II scores which reflect guideline content and format. The following conditions were also applied to define eligible guidelines: developed by non-profit organizations including academic institutions, government agencies, disease-specific foundations, or professional associations or societies; published in 2003 or later, which corresponds to the publication of AGREE II which provides developers with criteria for developing high-quality guidelines; English language; and either publicly available or could be ordered through our library system. Publications in the form of consensus statements, protocols, abstracts, conference proceedings, letters or editorials; based on primary studies that evaluated CAM therapies; or focused on CAM curriculum, education, training, research, professional certification or performance were not eligible.

MEDLINE, EMBASE and CINAHL were searched on January 28, 2016 from 2003 to 2015 inclusive. The search strategy (Additional file 1) included Medical Subject Headings and keywords that reflect terms commonly used in the literature to refer to CAM [7]. We also searched the National Guideline Clearinghouse, a publicly available repository of guidelines [http://​www.​guideline.​gov/​] using keyword searches restricted based on the eligibility criteria including “acupressure”, “acupuncture”, “Chinese medicine”, “chiropractic”, “chiropractor”, “herbal medicine”, “herbal supplement”, “herbal therapy”, “osteopath”, “phytotherapy”, “plant extract” and “spinal manipulation”. Next, we searched the NCCIH web site which contained a single list of CAM guidelines [https://​nccih.​nih.​gov/​health/​providers/​clinicalpractice​.​htm]; and the tables of contents of two CAM journals with the highest impact factors: BMC Complementary and Alternative Medicine [https://​bmccomplementalt​ernmed.​biomedcentral.​com/​] and the Journal of Complementary and Alternative Medicine [http://​www.​liebertpub.​com/​overview/​journal-of-alternative-and-complementary-medicine-the/​26/​] from January 2011 and December 2015. All three authors independently screened the titles and abstracts recovered from MEDLINE to standardize screening by discussing and resolving selection differences. Following this, JYN and LL screened titles and abstracts from all other sources. JYN and ARG screened full-text items to confirm eligibility.

The following data were extracted from each guideline and summarized: date of publication, country of first author; type of organization that published the guideline (academic institutions, government agencies, disease-specific foundations, or professional associations or societies); topic category 1 (CAM-specific) or category 2 (CAM-general); and guideline topic including type of CAM therapy and disease or condition. Most data were available in the guideline; to assess applicability, the web site of each developer was browsed and searched for any associated knowledge-based resources in support of implementation.

The extraction and analysis of data from eligible guidelines followed standardized methods for applying the AGREE II instrument [38]. To do this we used the instructional manual provided by AGREE for this purpose. This is a 60-page document that first describes the AGREE instrument, provides instructions on how to apply the instrument then, for each domain, provides detailed guidance on where to look in the guideline for relevant content to judge that domain and how to rate each item in that domain. First a pilot test of the AGREE II instrument was conducted with two guidelines during which all three authors independently assessed both guidelines with the AGREE II instrument. Discrepancies were discussed and resolved. JYN and LL then independently assessed all eligible guidelines for 23 items across six domains using a seven-point Likert scale from strongly disagree (1) to strongly agree (7) that the item is met; rated the overall quality of each guideline (1 to 7); and used that information to recommend for or against use of each guideline. ARG resolved differences. Average appraisal scores were calculated by taking the average rating for all 23 items of a single appraiser of a single guideline, followed by taking the average of this value for both appraisers. Average overall assessments were calculated as the average of both appraisers’ “overall guideline assessment” scores for each guideline. Scaled domain percentages were generated for inter-domain comparison, and were calculated by adding both appraisers’ ratings of items within each domain, and scaling by maximum and minimum possible domain scores, before converting this into a percentage. Average appraisal scores, average overall assessments and scaled domain percentages for each guideline was tabulated for comparison.

Searches retrieved 3,350 items, 3,126 were unique, and 3,095 titles and abstracts were eliminated, leaving 31 full-text guidelines that were considered. Of those, 14 were not eligible, primarily because they were not focused on CAM (7), they could not be retrieved (3), or did not meet other eligibility criteria (4), leaving 17 guidelines eligible for review.

Eligible guidelines were published in 2003 or later in Canada, the United States, United Kingdom, China, and Australia [40‐56]. The guidelines were funded and/or developed by professional associations or societies (13), academic (3), and an international agency (1). Nine guidelines were CAM-specific (2 herbal medicine, three acupuncture, four spinal manipulation) and 8 were CAM-general. Clinical topics included anorexia nervosa, breast cancer, cancer (general) diabetes, headache, herpes zoster, low back pain, lung cancer, major depressive disorder, migraine, multiple sclerosis, neck pain, and Parkinson’s disease.

Average appraisal scores, average overall assessments, and recommendation regarding use for each guideline are shown in Additional file 2. The average appraisal scores for each of the 17 guidelines ranged from 3.3 to 5.5 on the seven-point Likert scale (where seven equals strongly agree that the item is met); 14 guidelines achieved or exceeded an average appraisal score of 4.0, and seven guidelines achieved or exceeded an average appraisal score of 5.0. Average overall assessments for the 17 guidelines ranged between 3.0 (lowest) and 5.5 (highest), including 14 guidelines equalling or exceeding a score of 4.0, and 7 guidelines equalling or exceeding a score of 5.0.

None of the 17 guidelines were recommended by both appraisers. Appraisers agreed in their overall recommendation for 13 of 17 guidelines including 2 No [52, 55], and 11 Yes with modifications [40‐43, 47‐49, 51, 53, 54, 56]. Of the remaining four guidelines, three were rated by the two appraisers as No and Yes with modifications [44, 45, 50], while 1 guideline was rated at Yes and Yes with modifications [46].

With regards to scaled domain percentages, scope and purpose scores were 52.8 to 100.0 %, stakeholder involvement scores were 11.1 to 86.1 %, rigor-of-development scores were 14.6 to 92.7 %, clarity-of-presentation scores ranged from 69.4 to 97.2 %, applicability scores were 0.00 to 60.42 %, and editorial independence scores ranged from 0.0 to 95.8 %.

The overall objectives and health questions were generally well-defined in all but one guideline [45]. Authors provided the goal of the guideline, the types of CAM they sought to assess, and the disease or condition that was the target of CAM therapy or therapies. The population to whom the guideline was meant to apply was sometimes less detailed. For example, two guidelines referred to the intended population as “patients” [46, 48].

Most guidelines thoroughly in detailed the characteristics of the members of the guideline development group, typically including degrees held by, and institutional affiliation of each member, in addition to some of the following: subject discipline, geographical location, and description of member’s role in the group [41‐44, 46‐49, 53‐56]. Some guidelines detailed the views and preferences of the target population [44, 46, 53, 56] while most did not [40‐43, 45, 47‐52, 54, 55]. Target users of the guideline were typically inconsistently defined. Some guidelines offered clear descriptions, for example, type of practitioner, specialty [40, 46, 49, 50, 52, 53, 56], while other guidelines offered few details about target users [43, 45, 47, 50, 54, 55].

Systematic methods were almost always used to search for evidence and the criteria for selecting the evidence were almost always clearly described [40‐44, 46‐51, 53, 54, 56], with the exception of a few guidelines [45, 52, 55]. The strengths and limitations of the body of evidence were clearly described in all guidelines except for one [45]. The methods for formulating the recommendations varied; while some guidelines provided a fair amount of detail on how consensus was reached [40, 42, 46, 48, 49, 53, 54, 56], other guidelines provided minimal information if not none at all [40, 41, 45, 47, 51, 55]. All authors considered some health benefits, side effects, and/or risks in formulating their recommendations, with the exception of one [52]. Nearly all guidelines provided an explicit link between their recommendations and the supporting evidence with the exception of two guidelines in which this was inconsistent [49, 52]. While some guidelines explicitly stated that they were externally reviewed by experts prior to publication [41, 46, 54, 56], many did not [42‐44, 47, 48, 52, 55]. Some guidelines failed to mention the purpose and intent for, or the methods employed for the external review [40, 45, 49‐51, 53]. Most guidelines did not include a procedure for updating the guideline [42, 44, 45, 47, 48, 50‐56] and, among those that did, one guideline provided a detailed methodology [46].

Generally, all guidelines offered specific and unambiguous recommendations. However, many typically lacked one or more of the following details: identification of the intent/purpose, relevant population, or caveats. All 17 guidelines scored highly in presenting different options for the management of the condition or health issue, thus contributing to this high scaled domain percentage [40‐56]. Key recommendations were also generally very easily identifiable.

One guideline discussed facilitators and barriers to implementation of the recommendations [49]. Three guidelines included advice and/or tools to support implementation of the recommendations [49, 54, 56]. No guidelines addressed the resource implications of implementing the recommendations. Two guidelines provided monitoring and auditing criteria, while 14 guidelines contained little to no such information.

Guidelines varied in reporting of the funding source or competing interests of the members of the guideline development panel. Several guidelines that declared a funding source did not state whether funding source influenced the content of the guideline [41, 42, 48, 53, 54, 56].

No guidelines explicitly stated that no funding supported their development. Guidelines also varied in reporting of competing interests. Several guidelines did not address competing interests [45, 51, 52, 55, 56]. While remaining guidelines did so, two did not specify how potential competing interests were identified or considered, or how they may have influenced the guideline development process or issuing of recommendations [50, 53].