Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the ninth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

For over 25 years a growing number of organisations have employed various systems to grade the quality of evidence (sometimes called levels of evidence) and the strength of recommendations [1]. Unfortunately, different organisations use various grading systems, which may lead to confusion among consumers.

Groups making recommendations always make judgements about the quality of evidence and the balance of benefits and downsides (harms, burden and costs). Frequently these judgements are made implicitly rather than explicitly and judgements about the quality of evidence are confused with judgements about the balance of benefits and downsides. Many systems that are used to grade the quality of evidence and the strength of recommendations also confuse these judgements by equating the strength of recommendation with the quality of evidence, for example by grading recommendations for which there is high quality evidence as strong, without explicitly considering the balance of benefits and downsides.

Knowing the quality of evidence is essential, but not sufficient for making judgements about the strength of a recommendation. For instance, high quality evidence from well executed randomized controlled trials showed that oral anticoagulation administered for more than one year reduces the risk for recurrent thromboembolic events in patients after a first episode of spontaneous deep venous thrombosis. However, because oral anticoagulation is associated with harms (bleeding risk), burden (taking medication and monitoring anticoagulation levels) and cost (anticoagulation clinics or monitoring devices) the recommendation to anticoagulate all patients is weak because the benefits and downsides are finely balanced and individual patients will make different choices [2]. Both judgements about the quality of evidence and about the strength of a recommendation are complex and require consideration of a number of factors.

In this paper we addressed the following questions:

Questions related to what evidence should be included, how it should be synthesized and reported are addressed in other papers in this series [3‐5].

WHO groups (e.g. WHO Europe) have acknowledged the need for evaluating or developing a grading system [6, 7], and the Guidelines for WHO Guidelines recommend using a specific, uniform grading system [8]. However, this system, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, has scarcely been used within WHO [9, 10]. Some WHO groups have developed their own grading systems [11, 12], despite of the guidelines for WHO guidelines suggestion to use GRADE. Most have not explicitly graded either the quality of evidence or the strength of recommendations [13, 14].

Most, but not all organizations that develop guidelines use a grading system to express the strength of a recommendation or the quality of evidence. For example, the US Preventive Services Task Force (USPSTF) uses a grading system that assigns one of three grades of evidence: good, fair, or poor [15]. The Task Force uses its assessment of the evidence and magnitude of net benefit to make a recommendation, coded as a letter: from A (strongly recommended) to D (recommend against). The UK National Institute for Health and Clinical Excellence (NICE) has not yet made a decision as to which grading system to use [16]. The Scottish Intercollegiate Guideline Network (SIGN) has developed its own grading system for application to SIGN guidelines [17]. The Australian Medical Research Council is currently developing a grading system that will probably include grading recommendations according to strength of recommendations and quality of evidence [18]. The US Task Force on Community Preventive Services uses a system in which the quality of the evidence of effectiveness links directly the strength of the recommendation [19, 20]. Professional organizations use a variety of systems, many of them, however, based on two prominent grading approaches: the system derived from the Canadian Task Force on the Periodic Health Examination [1, 21] and a successor of that system, the Oxford Centre for Evidence Based Medicine approach [22].

More recently, medical societies have begun to form collaborations within specialties to develop grading systems on their own. For example a group of specialty societies in rehabilitation sciences formed a panel to develop an approach to grading the quality of evidence and strength of recommendations [23]. This panel developed a set of criteria for grading the strength of both the evidence and the recommendation. Similarly, the world leading urology associations have come together to adopt a uniform grading system and approach that would be useful for urologists around the world rather than each association using a different grading system [24]. This latter collaboration named, Evidence Based Urology, is exploring using the GRADE approach. The GRADE approach is being used increasingly by organisations around the world [25‐28], although in some cases with slight modifications [29]. It has been used for public health questions such as the pharmacological management of human influenza A(H5N1) infection (avian flu) [30], although it more commonly has been used for clinical questions up to now. A group of family practice and primary care journals has also developed a system to grade the strength of a recommendation [31].

The methods used to prepare this review are described in the introduction to this series [32]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We reviewed existing guidelines for guidelines to identify grading system currently in use. We also searched PubMed using (grading system) and (methods) (MESH headings/keywords) for systematic reviews and studies of methods for grading the quality of evidence. In addition, we searched databases maintained by the Agency for Healthcare Research and Quality (AHRQ, [33]) and the Guidelines International Network (GIN, [34]). These searches were supplemented with information obtained directly from guideline development organizations and our own files. Because of our involvement with organizations that produce guidelines and prior work with grading systems, in particular the GRADE system, we had in depth knowledge about several systems [25, 28, 29, 35, 36].

We identified one systematic review dealing with the evaluation of grading systems. In 2002, the US Agency for Healthcare Research and Quality (AHRQ) published a systematic review of existing systems to grade the quality of evidence and strength of recommendations [37]. The AHRQ review considered 40 systems until the year 2000 that addressed grading the strength of a body of evidence. The important domains and elements for the systems to grade the strength of evidence that the authors agreed on were quality (the aggregate of quality ratings for individual studies, predicated on the extent to which bias was minimized), quantity (magnitude of effect, numbers of studies, and sample size or power) and consistency (for any given topic, the extent to which similar findings are reported using similar and different study designs).

More recently, independent work by the Canadian Optimal Medication Prescribing and Utilization Service (COMPUS) used a detailed process to evaluate and select an evidence grading system and expanded the work by AHRQ (while accepting it) until the year 2005 [38]. COMPUS, which identifies, evaluates, promotes and facilitates best practices in drug prescribing and use among health care providers and consumers in Canada [39], is a nationally coordinated program, funded by Health Canada and delivered by the Canadian Coordinating Office for Health Technology Assessment (CCOHTA, [39]). They assembled a working group of internal researchers, information specialists, methodology experts, and external researchers to update the work of AHRQ. COMPUS searched for and selected review articles for the period 2000 to 2005. This resulted in more than 3,000 citations for selection. Eleven review articles were selected for further analysis based on a priori selection criteria specified by the working group. Nearly 50 evidence grading systems were identified from the 11 review articles. Canadian and international experts in evidence evaluation methodology helped identify an additional 10 instruments or systems not included in the list of identified grading systems. The identified instruments and systems were evaluated using the AHRQ evaluation grids. The highest scoring instruments were the GRADE and the SIGN approach [38]. A second round of expert consultation and stakeholder input from all interested parties confirmed the selection of these instruments.

The GRADE system was developed through an intensive international collaboration of methodologists, guideline developers and clinicians and incorporates the factors identified in the AHRQ review and described above [35, 36].

WHO has made a decision to use a grading system to grade the quality of evidence and strength of recommendations that is sensible and is being used widely, the GRADE system [36]. WHO has been involved in the development of this system from the beginning, and consideration has been given to the potential for application of the system to WHO guidelines in developing the GRADE approach. This might have been expected to facilitate the dissemination and adoption of this approach by WHO guideline developers. However, interest in GRADE workshops at WHO has been limited, there is not a tradition of grading the quality of evidence or strength of recommendations at WHO, and few resources have been invested in supporting the use of GRADE specifically, or in supporting more rigorous guidelines development methods generally.

More recently, however, the WHO rapid advice guideline panel for the pharmacological management of human infection with avian influenza A (H5N1) virus applied GRADE successfully [30] and several WHO guidelines are under development using GRADE (Sue Hill, personal communication). In general, the evidence that graded recommendations have advantages over non-graded recommendations is limited, but there are strong arguments, including the clear and transparent communication of how much confidence users can place in recommendations and the evidence underlying them. Another limitation is that both the quality of evidence and the strength of recommendations exist on a continuum. Categorization of quality into four categories and recommendations for or against treatments into two grades, strong and weak, may oversimplify complex health care recommendations, but guidelines consumers are generally likely to benefit from this simplification as they are most interested in which recommendations to follow.

We have found a large body of work on the development and evaluation of various grading systems. Problems have arisen because many different grading systems exist. Future efforts should focus on forging a consensus on using a sensible and uniform approach to grade the quality of evidence and strength of recommendations, building on the work of the GRADE working group. Use of the GRADE system by WHO, as is currently recommended by the Guidelines for WHO Guidelines, could help by obtaining more experience, particularly with non-clinical interventions, contribute to improvements in the existing system, contribute to agreeing on a common international approach to grading of recommendations and help to ensure the quality and transparency of the judgements that are made across various groups that make recommendations on behalf of WHO. Development of software and a detailed manual to simplify the use of the GRADE system is underway and should facilitate the use of this system and its further development.