Introduction

Guidelines such as the American College of Cardiology/American Heart Association/European Society of Cardiology (ACC/AHA/ESC) guidelines on cardiovascular disease provide criteria and recommendations for decision making with regard to diagnosis, management, and treatment in specific areas of healthcare. Despite their self-declared limitations, they are often perceived as evidence based and valid and have become the final arbiters of care for many clinicians. However, findings published in 1999 showed that guidelines did not adhere well to established methodological standards in terms of their identification and evaluation of the scientific evidence.¹ In recent years, there has been an improvement in methodological standards including a more sophisticated grading system and a more transparent link between the evidence and the grade of the recommendations. However, the question still remains about the methodological rigor in the adherence to the grading system when linking the evidence to recommendations.²

Editorial see p 119

Clinical Perspective on p 210

Currently a grading scheme based on class of recommendation and level of evidence for a given recommendation is used. Definitions of the classes of recommendation are as follows:

	Class I: conditions for which there is evidence and/or a general agreement that a given procedure/therapy is beneficial, useful and effective.
	Class II: conditions for which there is conflicting evidence and/or a divergence of opinion about the usefulness/efficacy of performing the procedure/therapy.
	Class IIa: weight of evidence/opinion is in favor of usefulness/efficacy.
	Class IIb: usefulness/efficacy is less well established by evidence/opinion.
	Class III: conditions for which there is evidence and/or a general agreement that a procedure/therapy is not useful or effective and in some cases may be harmful.

A recommendation may therefore be based on scientific evidence OR on opinion/consensus of the writing committee. In contrast, a level of evidence (LOE) to support a given recommendation is strictly based on available scientific literature. The weight of evidence is ranked from highest (A) to lowest (C) as follows:

	Level of evidence A: data derived from multiple randomized clinical trials or meta-analyses.
	Level of evidence B: data derived from a single randomized trial or nonrandomized studies.
	Level of evidence C: only consensus opinion of experts, case studies, or standard of care.

Thus, only a recommendation Class I LOE-A and a Class III LOE-A, respectively, are based on highest scientific evidence and suggest clearly positive or negative conclusions. However, shortcomings of recommendation Class I LOE-A and a Class III LOE-A are that they are based on patients who are referred and recruited to participate in randomized studies and may not represent the population to whom the results are applied.

All recommendations of Class IIa or Class IIb, independent of their level of evidence, reflect divergent evidence or opinions. Recommendations Class I or III, based on an intermediate level of evidence (LOE-B) or expert opinion (LOE-C), are less likely to be reliable and reproducible. They are more prone to be biased by financial or intellectual interests than recommendations on the highest level of evidence.^2–5

A recent study evaluated the distribution of recommendations and level of evidence of 16 ACC/AHA clinical practice guidelines.⁶ Results showed that recommendations are largely developed from lower levels of evidence or expert opinion (LOE-C, 48%) and only 314 of 2711 recommendations (11%) are classified as LOE-A. Furthermore, 41% of all recommendations were of no conclusive evidence (Class II). The percentage of Class I LOE-A and Class III LOE-A recommendations with a definite scientific evidence-based statement was only 9.5% and 0.6%, respectively. However, the authors reported only the distribution across classes of recommendations and LOE without any judgment for the accuracy of the referenced literature underlying the LOE.

Our study extends the existing literature by performing a systematic review of the currently available ESC clinical practice guidelines related to arrhythmia with intent to evaluate (1) the accuracy of the referenced literature underlying the recommendations of LOE-A and (2) to summarize the clear definite positive (Class I LOE-A) and negative (Class III LOE-A) statements as an estimation of the scientific evidence on which these 5 guidelines are based. We postulate that by strictly following the grading scheme, the proportion of the correctly referenced LOE-A recommendations is lower than that categorized in the guidelines.

Materials and Methods

The following 5 full-text practice guidelines related to arrhythmia posted on the ESC web site on February 07, 2010 (http://www.escardio.org/guidelines-surveys/esc-guidelines/Pages/GuidelinesList.aspx) were analyzed:

	ACC/AHA/ESC guidelines for the management of patients with supraventricular arrhythmias.7
	ESC/Heart Rhythm Society guidelines for the diagnosis and management of syncope (version 2009).8
	ACC/AHA/ESC 2006 guidelines for management of patients with ventricular arrhythmias and the prevention of sudden cardiac death.9
	ACC/AHA/ESC 2006 guidelines for the management of patients with atrial fibrillation: full text.10
	ESC guidelines for cardiac pacing and cardiac resynchronization therapy.11

First, to obtain the number of recommendations and the distribution of LOE within each class of recommendation, all guideline texts were manually analyzed to assess their general characteristics. Second, 457 relevant references for all recommendations with a LOE-A were investigated to reassess the accuracy of the referenced literature underlying each LOE-A recommendation (see Figure 1 for an overview). The type of study, the methods, and the results for each relevant reference were analyzed. According to the requirements for a level of evidence A, only (a) results of randomized trials (open-labeled or blinded, as well as post hoc analyses of primarily randomized conditions of those trials) directly addressing a given recommendation and (b) meta-analyses of randomized trials were accepted as adequate. Consequently, references of nonrandomized studies, animal studies, reviews, editorials, guidelines, as well as results from registries/surveys/databases were excluded. Third, the actual level of evidence behind a given guideline was derived by counting the Class I and III recommendations with correctly referenced LOE-A. Because 4 of 5 of the guidelines reviewed are joint guidelines between the ESC and ACC/AHA or Heart Rhythm Society, the results and conclusions of our study may have an impact on all 4 cardiology societies.

• Download figure
• Download PowerPoint

Statistics

Descriptive data were reported as frequencies, means, and standard deviations. Because individual guidelines vary widely in their numbers of recommendations, the median of the percentage reported for each guideline was shown. Two-sided t tests for independent samples were used for continuous variables. All statistics were computed with SPSS software (SPSS Inc, Chicago, IL). All probability values are 2-sided, with values of <0.05 considered significant.

Results

General Characteristics of the Analyzed Guidelines Related to Arrhythmia

Three of 5 clinical practice guidelines related to arrhythmia posted on the ESC web site were developed in conjunction with the ACC/AHA Task force^7,9,10 or Heart Rhythm Society.⁸ One guideline was released only by the ESC.¹¹ The general characteristics of all 5 guidelines are summarized in Table 1 . Each guideline contained a mean of 626 references, of which 38% were older than 10 years at the time of publication of the specific guideline, whereas 33% were less than 5 years old. On average, 17 committee members and 32 reviewers were involved in establishing a guideline. Conflicts of interest were listed in only 3 of the 5 guidelines^8–10; the mean number of conflicts per committee member was 4.

Distribution Across Classes of Recommendation and LOE

The distribution of all 698 recommendations with regard to classes of recommendations and LOE is shown as an overview in Figure 2 and subdivided by guidelines in Table 2. A median of 87.6% were positive Class I and II recommendations (interquartile range [IQR] 87.1% to 93.2%) compared with 12.4% negative Class III recommendations (IQR, 6.8% to 12.9%, P=0.01). Almost all recommendations were classified as having low or intermediate evidence (LOE-C and LOE-B, median 94.6%, IQR 90.3% to 95.1%). Only 5.4% of the recommendations were categorized as having highest evidence (LOE-A, IQR, 4.9% to 9.7%, P=0.002). The distribution per guideline for the classifications I to III of recommendations categorized as LOE-A is depicted in Table 3.

• Download figure
• Download PowerPoint

Accuracy of the Referenced Literature Underlying the LOE-A Recommendations

The number of relevant references assessed for all recommendations of LOE-A is shown as an overview in Figure 1. The reassessment of the accuracy of the referenced literature underlying every individual recommendation of LOE-A is shown in detail in Table 4. The table reveals that 67.8% (40 of 59) of LOE-A recommendations were correctly referenced according to the proposed grading scheme for LOE-A and were indeed “derived from multiple randomized clinical trials or meta-analyses” (Tables 3 and 4).

Recommendations With a Definite Conclusion

A median of only 1.2% (IQR 0.95% to 3.7%, n=27) clearly defined positive (Class I) and negative (Class III) recommendations with correctly classified highest level of evidence remained after excluding Class IIa and IIb recommendations with conflicting evidence or divergence of opinion (Tables 3 and 4). Only these 27 of 698 recommendations showed definite evidence-based conclusions. They covered a maximum of 3 similar areas per guideline, and totaled 8 positive and 2 negative directions/instructions:

The 10 Evidence-Based Recommendations for Arrhythmia Treatment

1. Reduce morbidity and mortality with cardiac resynchronization therapy through the use of a biventricular pacemaker in symptomatic heart failure patients with left ventricular ejection fraction ≤35% and QRS ≥120 ms.

2. Reduce total mortality with implantable cardioverter defibrillator therapy for primary and secondary prevention in patients with left ventricular dysfunction because of prior myocardial infarction with left ventricular ejection fraction 30% to 40% or symptomatic heart failure.

   2a. Reduce total mortality with implantable cardioverter defibrillator therapy for secondary prevention in patients with nonischemic dilated cardiomyopathy and significant left ventricular dysfunction.

3. Avoid Class Ic antiarrhythmic drugs in patients with a history of myocardial infarction.

4. Treat elderly patients experiencing ventricular arrhythmias in the same manner as younger individuals.

5. Use antithrombotic therapy (aspirin/vitamin K antagonists) to prevent thromboembolism in all patients with atrial fibrillation according to their risk stratification for stroke and bleeding.

6. Administer dofetilide, flecainide, ibutilide, or propafenone, but avoid digoxin and sotalol for pharmacological cardioversion of atrial fibrillation.

7. Avoid antiarrhythmic agents to maintain sinus rhythm in patients with atrial fibrillation who are at risk for developing an arrhythmia with this drug.

8. Treat patients with oral β-blockers to prevent postoperative atrial fibrillation.

9. Avoid β-blocker therapy as a treatment for the prevention of syncope.

10. Control the rate of stable atrial flutter with a nondihydropyridine calcium channel blocker.

Incorrectly Categorized LOE-A Recommendations

Nineteen of 59 recommendations (18 Class I) were incorrectly categorized as LOE-A. We divided these into 3 major categories:

1. Ten recommendations were not based on any randomized study because of possible difficulties that might have been encountered if they had been randomized in clinical trials: four of them were obvious statements; for example, “Acute management of cardiac arrest: Cardiopulmonary resuscitation (CPR) should be implemented immediately after contacting a response team.”⁹ Four recommendations would have resulted in ethical issues if they had been randomized into trials; for example, “Withdrawal of any offending drugs and correction of electrolyte abnormalities are recommended in patients presenting with torsades de pointes.”⁹ Two of the recommendations were diagnostic procedures, which are more difficult to randomize in clinical trials compared with studies that investigate a therapeutic outcome, eg, “Resting 12-lead ECG is indicated in all patients who are evaluated for ventricular arrhythmias.”⁹

2. Seven of 19 recommendations could have been randomized, but none or only one study was referenced, eg, “INR should be determined at least weekly during initiation of therapy and monthly when anticoagulation is stable.”¹⁰ Three of these 7 recommendations are no longer of interest for research, eg, “Atrial or transesophageal pacing is recommended for conversion of stable atrial flutter.”⁷

3. Finally, 2 recommendations, although being accurately supported by randomized studies, were based on other guidelines as tertiary source, instead of the original research articles as required for LOE-A, eg, “In patients with ischemic cardiomyopathy with severely depressed left ventricular ejection fraction or HF, implantable cardioverter defibrillator therapy is indicated according to current guidelines for implantable cardioverter defibrillator-cardiac resynchronization therapy implantation.”⁸

Despite the importance of these 19 recommendations in clinical practice, there is still no justification for their being ranked as LOE-A.

Discussion

The present study analyzed not only the classes of recommendations and level of evidence in 5 currently available international guidelines related to cardiac arrhythmias, but also reassessed the referenced literature underlying these recommendations and extracted the accurately referenced recommendations with a definite conclusion.

As previously found most recommendations were positive (Class I and II) and of intermediate (LOE-B) or low (LOE-C) evidence.⁶ Only some of the recommendations were categorized as having highest level of evidence (LOE-A). Among those, one third were not at all or not adequately supported by randomized clinical trials or meta-analyses. This highlights a lack of accuracy in the rating of the scientific evidence underlying even those recommendations that are supposed to have the strongest empirical evidence. Our study raises the question about the accuracy of categorized LOE-A in guidelines for other health topics. It supports suggestions for the reform of the guideline-writing process because recommendations with incorrectly attributed LOE-A may reflect a lack of critical reviews of the guideline document within the guideline committee, as well as the fact that guidelines do not undergo the usual procedures of independent reviews in the publication process.⁵ Some of the recommendations incorrectly ranked as LOE-A cannot be analyzed in randomized studies, eg, because of ethical reasons. In these cases, surveys that are actually ranked as LOE-C have been used to retrospectively underline the importance of such recommendations. As an alternative, recommendations with consistent evidence derived from surveys could be summarized in a new category of LOE-S in order to not compromise correctly supported recommendations with LOE-A and attach more scientific importance than simply LOE-C.

As an indicator for improvement in the guideline process, the number of inappropriate uses of LOE-A is decreasing in the most recent guidelines.^10,11 In addition, the guidelines for the management of patients with atrial fibrillation¹⁰ quantified the number needed for the LOE-A label to 3 to 5 randomized studies or meta-analyses, which is more difficult to reach and could be added in updates of guidelines.

More than a third of the references were older than 10 years at the time of release of the analyzed guidelines. This is surprising because 3 of 5 guidelines were updates. Although some older milestone studies remain pivotal, some other trials from the early 1990s used a less sophisticated design, or treatments, procedures, or instrumentation that are no longer in use. As a consequence, a list of research topics attached at the end of the guideline text could serve as an indicator for areas in which studies with a new methodological approach are needed.

Ninety-nine percent of the recommendations lack sufficient scientific support and are situated in a gray zone of evidence, which include Class I and Class III recommendations of LOE-B and LOE-C and all Class II recommendations independent of the level of evidence. Class I and Class III recommendations of LOE-B (13% and 2.3%, respectively) imply the existence of only one randomized (or even a nonrandomized) trial. Further studies might yield similar results, leading to a LOE-A label, or might yield opposite results, switching the recommendation from Class I to Class II. Updates in guidelines led to the highest increase in the category of recommendations with conflicting evidence (Class II),⁶ suggesting that additional studies lead to more conflicting results. The crux of all Class II recommendations (46.1% in our study) independent of the LOE is that conflicting evidence has to be assessed by expert opinion to look into the methodology and the outcomes of the studies. Finally, all LOE-C recommendations (58.5% in our study) are by definition “only consensus opinion of experts or standard-of-care.”

All recommendations in the gray zone are prone to a potential bias such as financial and intellectual interests of the committee members.^2–5,12 Additionally, methodological shortcomings may lead to biased recommendations. Decision making in the guideline-writing process often does not follow clearly defined and transparent methodological standards. Less than 10% of the guidelines used and described formal methods of combining expert opinion.¹ For example the definition of usefulness/efficacy is not precisely quantified in terms of quality of life, number needed to treat, cost-benefit, cost of year of life gained, or other cost benefit analysis, as well as clinical relevance. There is a difference between the level of evidence that can be derived from large randomized trials with hard end points and results from smaller randomized trials with surrogate markers or meta-analyses. Furthermore, interpretation of systematic reviews with meta-analyses includes a subjective component. This can lead to discordant conclusions¹³ or even to different recommendations between guidelines.^2,6 In addition, general agreement and consensus opinion of experts in the decision-making process is not clearly defined, ie, whether the full or the simple majority of committee members has to agree.

As a consequence of these shortcomings, a reform of the guideline process has been suggested by different authors. Sniderman and Furberg proposed to post a prefinal version of a guideline on the internet to give an opportunity to exchange scientific opinion before final decisions are taken.⁵ Hirsch and Guyatt suggested that unconflicted methodologists instead of potentially biased experts should have the final responsibility for recommendations.²

In order to further improve future guideline-writing processes, the most recent ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2008¹⁴ summarize the major “gaps in the evidence” in an attempt to focus future clinical research on important issues that have not been adequately addressed. However, the definition of “gap in the evidence” is defined as absence of any recommendations in a certain area. Therefore, it differs from our definition which includes nonconclusive evidence.

A main limitation of this study is the focus on Class I and III recommendations with LOE-A. We did not carry out a detailed exploration of Class IIa and IIb recommendations with LOE-A which imply much more expert opinion than classifications I or III. Furthermore, to analyze whether recommendations with LOE-B are really “derived from single randomized or nonrandomized trials” was beyond the scope of this article. Because there are more LOE-B than LOE-A recommendations, a critical interpretation of guidelines in general might have even more of an impact than just focusing on LOE-A recommendations alone.

Conclusions

Our findings raise the question of the accuracy of LOE-A in medical guidelines in general and highlight the importance of a critical use of all recommendations. Moreover, they underline the need for improving the guideline-writing process. Although guidelines are improving by adding recommendation classifications and grading schemes for LOE, the points mentioned above highlight the fact that guidelines are mostly driven by consensus or expert opinion. Even recommendations with LOE-A need to be reassessed for sufficient empirical support. In the absence of highest correctly referenced evidence (Class I and III recommendations of LOE-A) clinicians should be even more cautious in implementing recommendations into clinical practice, also taking into consideration primary data, their own knowledge, and a multidisciplinary approach. Further randomized, double-blinded, and/or crossover-designed studies should focus on areas with a gap in the evidence, such as existing but not yet convincing (LOE-B) or conflicting (Class II) evidence.

Sources of Funding

Jeannette Brodbeck was supported by a research fellowship from the Swiss National Science Foundation.

Disclosures

Pedro Brugada discloses that his institution has received research grants and himself speaker fees from: Medtronic, Boston Scientific, St Jude Medical, Biosense Webster and Biotronik.

Footnotes