Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies

First, policy evaluations in the form of systematic reviews and other relevant documents published in peer-reviewed journals were identified through searches in the databases MEDLINE, Web of Science, and the Cochrane Library. The search strategy was developed with the support of an information specialist and adapted for each database accordingly. Databases were searched up until 4 of July 2019, and the search was not restricted to any language. The MEDLINE (Ovid) search strategy is presented in Additional file 2. Moreover, forward citation tracking and reference checking from included documents were performed.

Secondly, an internet-based search through the Google™ search engine and through Google Scholar™ was performed to identify gray literature and different relevant institutions and organizations, such as the Food and Agriculture Organization for the United Nations (FAO), World Health Organization (WHO), World Cancer Research Fund International (WCRF), the National Institute for Health and Care Excellence (NICE), and the Campbell Collaboration. Following this, the types of documents relevant for policymaking and which address dietary behavior, physical activity, and/or sedentary behavior, such as guidelines, reports, or systematic reviews, were searched on the websites of relevant organizations. Documents were found both through the Google search directly and also retrieved from the websites of those organizations and institutions identified in the search.

Thirdly, an extensive overview of existing policies including policy evaluations from the NOURISHING-database by the WCRF was considered [21]. Lastly, science platforms and databases such as ProQuest, Dimensions, BASE, Metager, Oaister, and the Political Science Database (Berkeley) were searched.

Because of the lack of a standard definition of the term “policy” across various areas of health policy and policy decision-making, the current search also aimed to identify documents that might not have used the term “policies” or “policy evaluations.” We focused on documents that described a body of evidence which was assessed for the development or evaluation of a policy, including documents labeled as “guidelines,” or systematic reviews used to inform policymaking [22]. Furthermore, we established criteria for interventions and/or recommendations (i.e., policies) to be eligible. Documents had to meet the following criteria to be included in this scoping review:

There were three exclusion criteria:

First, title and abstract screening was performed by one reviewer (JZ). Only clearly irrelevant references were excluded at this stage. Second, for all potentially relevant references, full-text publications were obtained and checked for final inclusion by two reviewers (JZ, LS) independently. Uncertainties were resolved through discussion with a third author (JJM).

For included documents, one reviewer (JZ) extracted the following characteristics: name of first author, year of publication, document type, study aim, outcome of interventions (physical activity, dietary behavior, sedentary behavior, multi-component), design of primary studies included in the document, policy/intervention(s), target group/study setting, use of GRADE, GRADE ratings for relevant outcomes (including down- and upgrading factors), facilitators, and barriers to apply GRADE. The data extraction form was piloted and proved to be applicable to all types of retrieved documents. Data extraction was verified by a second reviewer (LS).

The documents were categorized according to policy evaluation, evidence of effectiveness for policymaking, or a recommendation to guide policymaking (see Table 1). Furthermore, we categorized the documents according to the applied interventions (type of policy) in the studies (e.g., school program and mass media), according to the setting of the studies (e.g., workplace and community) and according to PEN-relevant outcomes (dietary behavior, physical activity, and sedentary behavior). All these characteristics were collected for each document and were displayed in Table 2. In further tables, we displayed the GRADE ratings, facilitators, and barriers for the use of GRADE (Table 3), as well as the strengths of recommendations (Additional file 4).

To complement the scoping review, a survey was conducted by contacting Ministries of Health or other responsible institutions that play a role in health policymaking in all 53 European countries, as listed by the WHO (Additional file 3) [59]. Contact information was searched through the FAO websites, the WHO websites, and respective websites of national governments. An e-mail was sent in English with a brief description of the project and the question whether the GRADE approach had been used for nutrition and physical activity policymaking in these countries.

The characteristics of the included documents are displayed in Table 2. All documents were published in English, and they were either systematic reviews with meta-analyses (n = 10; 28%), systematic review protocols (n = 8; 22%), systematic reviews without meta-analyses (n = 9; 25 %), guidelines (n = 6; 17%), an umbrella review (n = 1; 3%), or papers on guideline making (n = 2; 5%). From these findings, 17 documents (47%) were Cochrane reviews (including five Cochrane review protocols). Four documents were international guidelines by the WHO and one guideline by NICE (physical activity and the environment [45]). Furthermore, two sets of national-level guideline recommendations to guide policymaking were found: the Canadian 24-Hour Movement Guidelines for Children and Youth [52, 53]. Documents were published between 2011 and 2019, and half of those (n = 18) were published within the last 3 years. Settings of the studies included communities, schools, or workplaces. The majority of recommendations/interventions addressed the general population through various forms of mass media. Furthermore, the included documents examined various combinations of interventions addressing dietary behavior, physical activity, and sedentary behavior, and also combinations of these with other health-related topics. Most of the studies investigated interventions related to dietary behavior alone (39%), followed by physical activity (17%). There was only one study that assessed interventions aiming to diminish sedentary behavior alone (3%), and all other documents (41%) were based on interventions that targeted various outcomes, focusing on multicomponent interventions that addressed general health and aspects such as alcohol or tobacco consumption.

Many of the documents included did not review implemented policies and focused rather on interventions that have the potential to drive policy in the future. Seven of the 36 documents include studies in which the effects of implemented nutrition and physical activity policies were examined, including nutrition labeling [27, 49], SSB-taxes [35], the WHO Health Promoting Schools framework [39], taxation of fat content in food [40], sodium restriction [43], and taxation of unprocessed sugar or sugar-added foods [48]. Five of these are based on policies issued by governments (see Table 2). Twenty-two of the included documents applied the GRADE approach for grading the certainty of evidence for PEN-related outcomes (for instance, increase of physical activity, change of body-mass index (BMI), and sugar or salt intake) (see Table 3). The remaining 14 documents were either systematic review protocols (n = 8, [26, 33, 35, 36, 40, 44, 48, 50]) where GRADE will be used for grading the certainty of evidence, one recommendation to guide policymaking which used GRADE-ADOLOPMENT (GRADE Evidence to Decision frameworks for adoption, adaptation, and de novo development of trustworthy recommendations) (n = 1, [47]), documents in which GRADE was used on another development stage of the document (i.e., recommendations where GRADE ratings of included systematic reviews were evaluated, n = 2, [52, 53]), or documents in which the barriers against the use of GRADE were stated (n = 3, [30, 32, 42]) (see Table 4).

Table 3 shows the ratings of the certainty of evidence for all PEN-related outcomes for each study (max. seven ratings per identified document are presented in the table), and the reasons for down- and upgrading. Overall, there were 313 individual outcome ratings (107 GRADE ratings were done for the NICE guideline alone [45]): 34% of very low, 38% of low, 21% of moderate, and 7% of high certainty of evidence. The main reasons for downgrading were high risk of bias, imprecision (e.g., for wide confidence interval, small sample size, and incomplete outcome data), or there was an initial low starting level of a body of evidence from non-randomized study design type (Table 3).

The WHO guidelines were the only documents which, in addition to rating the certainty of evidence for each outcome, also provided the “strength of recommendations” for each recommendation included in the guideline (Additional file 4).

In two documents, an adaption of GRADE was used for primary outcomes [24, 30]. In three documents, the authors mentioned the GRADE approach or presented specific barriers that prevented them from applying GRADE [30, 32, 42], and in one document, the GRADE-ADOLOPMENT approach was used [47] (see Table 4).

Barriers for the use of GRADE for the assessment of the certainty of evidence were mentioned only in six documents. Reasons cited by authors for their decision not to use GRADE included, among others, incomplete, insufficient, or inaccessible data [24, 32, 42]. In another document, it was mentioned that difficulties in the application of the GRADE approach were seen due to the research topic, where RCTs are difficult to conduct, or because of subjective outcome measures such as quality of life [42]. Dyson et al. [30] mentioned some general challenges with the GRADE approach, such as the initial low starting level of a body of evidence from cohort studies and the difficulties of applying GRADE for individualized interventions. Instead, Dyson et al. adopted a grading system very similar to the GRADE system, in which recommendations on the certainty of evidence are provided on a scale from 1 to 4 (strong to very low strength) [30]. The motivation for using this adapted approach is not discussed. In the NICE guideline, a modified version of GRADE was used where NRSs started as “high quality evidence” when the study design was the most feasible one for the examined intervention [45]. Only one document mentioned facilitators for the use of GRADE; however, this was for another GRADE methodology, namely the GRADE-ADOLOPMENT, which was used by Okely et al. for the development of new guidelines [47]. There was no mention of facilitators or barriers for assessing the strengths of recommendations in any of the included documents. Facilitators and barriers are presented in Table 4.

No documents were retrieved through the survey sent to ministries and authorities in 53 European countries. Staff members from eleven countries replied to the request, stating that their national policies were not developed according to the GRADE approach (Additional file 3). Representatives from those countries, namely Austria, Lithuania, Luxemburg, Monaco, the Netherlands, Northern Ireland, and Poland, replied that they do not use GRADE in any way for policymaking. The German Nutrition Society used a newly developed grading system called NutriGrade for their protein guideline [60]. As a response to the survey, Bosnia and Herzegovina showed explicit interest in using GRADE for future policymaking. Iceland and Norway stated that they do not use GRADE for policymaking but in some cases for national clinical guidelines. Monaco replied that they refer to the WHO or French national recommendations. Furthermore, the Nordic Nutrition Recommendations use WHO guidelines, and Norway and Iceland reported that their policies are either based on the Nordic Nutrition Recommendations or they make policies which are based on WHO guidelines directly. No reasons were given whatsoever why GRADE has not been used directly.

To the best of our knowledge, this scoping review is the first to examine if and to what extent the GRADE approach has been used in nutrition and physical activity policymaking and policy evaluation in the areas of dietary behavior, physical activity, and sedentary behavior. There were no findings of actual policies or formulation processes from governmental bodies; however, five systematic reviews included evidence from interventions that were based on policies issued by a government or governmental organization (see Table 2). This scoping review found that the GRADE approach has been used for policy evaluations, in the evaluation of the effectiveness of policy-relevant interventions, as well as for guidelines and recommendations intended to guide policymaking, where the evidence for a recommendation or an intervention has to be assessed. These results may contribute to improving the process of evidence-informed policymaking in the areas of dietary behavior, physical activity, and sedentary behavior.

Almost half of the included documents that used the GRADE approach to assess the certainty of evidence are Cochrane Reviews (n = 17). The reason for this is that Cochrane adopted the GRADE approach for the evaluation of the certainty of evidence in systematic reviews [61]. The strength of recommendations has only been determined in WHO guidelines, and no barriers to its use were mentioned in these cases. Already in 2003, the WHO selected the GRADE approach as the method of choice to be used in their guideline development [62]. This explains why the guidelines identified in the context of this scoping review are mostly WHO guidelines.

Although guideline and recommendation development are a major part of policymaking, a number of perceived concerns may prevent institutions from adopting available methods. With respect to applying GRADE to policies affecting dietary behavior, physical activity, and sedentary behavior, concerns that are frequently raised include the assessment of risk of bias and the initial low starting level of a body of evidence obtained from NRSs. According to the GRADE approach, the certainty of evidence is initially determined by study design: a body of evidence from RCTs starts with a high certainty, whereas a body of evidence from NRSs, such as prospective observational studies, starts with low certainty due to confounding and selection bias issues associated with NRSs [63, 64]. However, with the advent of tools that use the concept of a target trial as reference point, such as the ROBINS-I (risk of bias in non-randomized studies of interventions), initial certainty of the evidence can also be high for bodies of evidence from NRSs [63, 64]. Nevertheless, as stated above, it is likely that NRSs would be downrated to low certainty because of confounding and selection bias [63].

There has been a long debate regarding what constitutes best evidence in nutrition research and whether it emerges from RCTs in which the effects of a dietary change on disease, intermediate or surrogate markers, or recognized risk markers are evaluated [65]. However, most dietary intervention RCTs are of short duration and do not target patient-relevant outcomes such as morbidity or mortality. RCTs, if well-designed and conducted, provide robust answers to the research questions they address and are considered the ideal methodology for causal inference. NRSs, on the other hand, provide less-robust information regarding causality but are usually considered more applicable for nutrition research. Additionally, all common dietary assessment methods used in nutrition research, such as food consumption records, 24 h recalls, dietary records, dietary history, and food frequency questionnaires suffer from inherent limitations, for example, because the measures are subjective or the results are difficult to reproduce [66].

Apart from the WHO guidelines and the NICE guideline, there were no publications closer to the legislative policymaking process (e.g., policy briefs, policy documents, proposals to inform laws or regulations) that used GRADE. The legislative process underlying policymaking is complex and time-consuming. In addition, there are specific challenges associated with translating empirical findings from the lab into actionable policies in government [66‐69]. The absence of documents could be due to various factors: for example, key documents not being publicly available (e.g., internal position papers by political parties), documents being informed by GRADE not explicitly using the term, or the search strategy being specifically tailored to identify those types of policy documents. It could also be due to GRADE not being employed to inform policymakers’ decisions, for example, due to issues in uptake of evidence in the policymaking process [67‐69], inadequate knowledge-translation strategies [70‐72], or an unawareness about GRADE. Future research needs to address this issue.

This scoping review has several strengths. Firstly, the inclusion criteria and the search strategy planned for this scoping review were broad, including an extensive search of gray literature such as databases of the WHO or the WCRF in addition to more traditional sources. This access to a wide variety of documents in the research field yields a better overview than previously available. Secondly, we performed a rigorous approach to screening and data extraction, which was performed by two reviewers independently. Thirdly, the survey among national policymakers of the European region brought relevant input and insights into this study by giving an idea about the current use of GRADE within policymaking institutions.

This is the first scoping review on the use of GRADE in nutrition and physical activity policy evaluation, but other publications have also addressed the complexity of interventions in the public health field, which might prevent researchers and policymakers from applying this approach. Public health and health policy interventions are often thought of as events in systems [73], whereby complex intervention (e.g., characterized by several interacting active components intended to influence multiple behaviors [74]) are implemented in an interactive system (e.g., characterized by changes of the system due to emergent properties, adaptivity, or feedback mechanisms [73, 75]). However, there is specific guidance on how the GRADE approach can be applied in these cases [76, 77].

A study conducted by Rehfuess and Akl in 2013 examined the experience with applying the GRADE approach to public health interventions [77]. In their survey performed in 2013, NICE, which is the national institute that provides guidance and advice to improve health and social care in the UK, stated that the most important reason for not using GRADE when investigating (complex) public health interventions was the fact that most of the evidence originates from NRSs [77]. However, since 2014, NICE included the GRADE approach in their manual for guideline development, also for public health guidelines [78]. Since bodies of NRSs start with low certainty of evidence, it seemed that this would hinder the application of GRADE in such cases [30]. This issue was also brought up in the findings of this scoping review as mentioned above.

Recently, Blake and colleagues showed that from 32 food-based dietary guidelines published since 2010, none of the guidelines used the GRADE approach to evaluate the certainty of evidence or graded the strength of recommendations [79]. However, the authors also acknowledged that methods may not be directly applicable to the types of studies included in food-based dietary guidelines. Of note, GRADE has established working groups focusing on an approach for public health, modeling studies, or environmental exposure studies.

It has been pointed out that the use of the GRADE approach in particular in relation to bias assessment is challenging (e.g., users of GRADE may inappropriately double count the risk of confounding and selection bias so the certainty of evidence is downgraded too much) [60, 80]. Further, the usefulness of GRADE to assess the evidence of effectiveness in domains of health policy other than dietary behavior, physical activity, and sedentary behavior needs to be explored. In general, knowledge translation and exchange strategies (i.e., methods of bringing scientific findings to policymakers) have provided mixed results regarding the use or uptake of scientific evidence in the policymaking process [70‐72]. One key issue has been the perceived limited relevance, generalizability, and applicability of research findings to a specific setting [81]. Future research should focus on assessing the views of policymakers on the usefulness of GRADE to inform their decisions.

Moreover, beyond the evidence of effectiveness in a given context, public health and health policy decision-makers have to consider numerous additional aspects [82‐85], for example, stakeholder interests, ethical concerns, equity/equality considerations, financial implications, or societal and cultural values and norms. These have to be taken into account along with scientific evidence of effectiveness which is often not the most important consideration [68]. In addition, there are individual, group, or systemic factors such as own experiences of policymakers, the political system, institutional mechanisms (structures, processes, or regulations), or contextual, societal factors [69, 86]. The GRADE EtD frameworks might be a suitable approach to address these issues. They might help policymakers to identify, prioritize, and address issues beyond benefit-harm considerations in the healthcare decision-making process in a structured and transparent way such as resource use and cost-effectiveness, equity, acceptability, and feasibility [15, 17]. However, we did not identify any documents which used the EtD frameworks.