Formatting modifications in GRADE evidence profiles improved guideline panelists comprehension and accessibility to information. A randomized trial

doi:10.1016/j.jclinepi.2011.11.013

Journal of Clinical Epidemiology

Volume 65, Issue 7, July 2012, Pages 748-755

https://doi.org/10.1016/j.jclinepi.2011.11.013 Get rights and content

Abstract

Objective

To determine the effects of formatting alternatives in Grading of Recommendations Assessment, Development, and Evaluation (GRADE) evidence profiles on guideline panelists’ preferences, comprehension, and accessibility.

Study Design and Setting

We randomized 116 antithrombotic therapy guideline panelists to review one of two table formats with four formatting alternatives. After answering relevant questions, panelists reviewed the other format and reported their preferences for specific formatting alternatives.

Results

Panelists (88 of 116 invited [76%]) preferred presentation of study event rates over no study event rates (median 1 [interquartile range (IQR) 1] on 1–7 scale), absolute risk differences over absolute risks (median 2 [IQR 3]), and additional information in table cells over footnotes (median 1 [IQR 2]). Panelists presented with time frame information in the tables, and not only in footnotes, were more likely to correctly answer questions regarding time frame (58% vs. 11%, P < 0.0001), and those presented with risk differences and not absolute risks were more likely to correctly interpret confidence intervals for absolute effects (95% vs. 54%, P < 0.0001). Information was considered easy to find, easy to comprehend, and helpful in making recommendations regardless of table format (median 6, IQR 0–1).

Conclusion

Panelists found information in GRADE evidence profiles accessible. Correct comprehension of some key information was improved by providing additional information in table and presenting risk differences.

Section snippets

Background

Most clinical practice guidelines suffer from limitations in their methodological approach to assess and apply research evidence supporting their recommendations [1], [2]. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a systematic, explicit, and transparent process for evaluating and reporting quality of research evidence and moving from evidence to recommendations [3], [4], [5], [6], [7]. Key features of the GRADE system are structured

Overview of design

We conducted a randomized controlled trial (RCT) with guideline panelists comparing two formats of the evidence profile (table A and table B) differing by four features (see Fig. 1). The content of the tested evidence profiles was based on two different clinical questions. We ran the trial during ongoing work on the AT9 guidelines. The Research Ethics Board of McMaster University approved the study.

The AT9 executive committee issued e-mail invitations to panelists to complete an online

Results

Of 116 panelists invited, 88 (76%) participated in the trial (31% females, 45% <45 years, 52% between 46 and 65 years, and 3% above 65 years of age). Of the 88 participants 24% were editors, 31% had no formal training in health research methodology, 43% had not previously participated in guideline panels, 48% had looked at five or fewer GRADE evidence profiles, and 81% were most comfortable with speaking and reading English. See Fig. 1 for flowchart. Stratification resulted in equal

Main findings

Our results show that guideline panelists preferred presentation of additional information in table cells over footnotes (Fig. 2) and absolute risk differences over absolute risk estimates (Fig. 4). These presentation features were associated with improved comprehension of key information and reduced time spent finding this information. Guideline panelists agreed that information about quality assessment and effect estimates in GRADE evidence profiles was easy to find, easy to use, and helpful

Acknowledgments

The authors thank Aravin Duraikannan for valuable help with the randomization procedure and Rachel Gutterman for help with organizing the conduct of the study.

References (16)

G. Guyatt et al.
GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
(2011)
G.H. Guyatt et al.
GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
J Clin Epidemiol
(2011)
H.J. Schunemann et al.
Methodology for antithrombotic and thrombolytic therapy guideline development: American College of Chest Physicians Evidence-based Clinical Practice Guidelines (8th edition)
Chest
(2008)
S.E. Rosenbaum et al.
Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information
J Clin Epidemiol
(2010)
S.E. Rosenbaum et al.
User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews
J Clin Epidemiol
(2010)
D. Atkins et al.
Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group
BMC Health Serv Res.
(2004)
R. Jaeschke et al.
Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive
BMJ
(2008)
G.H. Guyatt et al.
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
BMJ
(2008)

There are more references available in the full text version of this article.

Cited by (29)

A multimethods randomized trial found that plain language versions improved adults understanding of health recommendations
2024, Journal of Clinical Epidemiology
To make informed decisions, the general population should have access to accessible and understandable health recommendations. To compare understanding, accessibility, usability, satisfaction, intention to implement, and preference of adults provided with a digital “Plain Language Recommendation” (PLR) format vs. the original “Standard Language Version” (SLV).
An allocation-concealed, blinded, controlled superiority trial and a qualitative study to understand participant preferences. An international on-line survey. 488 adults with some English proficiency. 67.8% of participants identified as female, 62.3% were from the Americas, 70.1% identified as white, 32.2% had a bachelor's degree as their highest completed education, and 42% said they were very comfortable reading health information. In collaboration with patient partners, advisors, and the Cochrane Consumer Network, we developed a plain language format of guideline recommendations (PLRs) to compare their effectiveness vs. the original standard language versions (SLVs) as published in the source guideline. We selected two recommendations about COVID-19 vaccine, similar in their content, to compare our versions, one from the World Health Organization (WHO) and one from Centers for Disease Control and Prevention (CDC). The primary outcome was understanding, measured as the proportion of correct responses to seven comprehension questions. Secondary outcomes were accessibility, usability, satisfaction, preference, and intended behavior, measured on a 1-7 scale.
Participants randomized to the PLR group had a higher proportion of correct responses to the understanding questions for the WHO recommendation (mean difference [MD] of 19.8%, 95% confidence interval [CI] 14.7–24.9%; P < 0.001) but this difference was smaller and not statistically significant for the CDC recommendation (MD of 3.9%, 95% CI −0.7% to 8.3%; P = 0.096). However, regardless of the recommendation, participants found the PLRs more accessible, (MD of 1.2 on the seven-point scale, 95% CI 0.9–1.4%; P < 0.001) and more satisfying (MD of 1.2, 95% CI 0.9–1.4%; P < 0.001). They were also more likely to follow the recommendation if they had not already followed it (MD of 1.2, 95% CI 0.7–1.8%; P < 0.001) and share it with other people they know (MD of 1.9, 95% CI 0.5–1.2%; P < 0.001). There was no significant difference in the preference between the two formats (MD of −0.3, 95% CI −0.5% to 0.03%; P = 0.078). The qualitative interviews supported and contextualized these findings.
Health information provided in a PLR format improved understanding, accessibility, usability, and satisfaction and thereby has the potential to shape public decision-making behavior.
GRADE guidance 23: considering cost-effectiveness evidence in moving from evidence to health-related recommendations
2023, Journal of Clinical Epidemiology
This is the 23rd in a series of articles describing the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to grading the certainty of evidence and strength of recommendations for systematic reviews, health technology assessments, and clinical guideline development.
We outline how resource utilization and cost-effectiveness analyses are integrated into health-related recommendations, using the GRADE Evidence to Decision (EtD) frameworks.
Through iterative discussions and refinement, in-person, and online meetings, and through e-mail communication, we developed draft guidance to incorporate economic evidence in the formulation of health-related recommendations. We developed scenarios to operationalize the guidance. We presented a summary of the results to members of the GRADE Economic Evaluation Project Group.
We describe how to estimate the cost of preventing (or achieving) an event to inform assessments of cost-effectiveness of alternative treatments, when there are no published economic evaluations. Evidence profiles and Summary of Findings tables based on systematic reviews of cost-effectiveness analyses can be created to provide top-level summaries of results and quality of multiple published economic evaluations. We also describe how this information could be integrated in GRADE’s EtD frameworks to inform health-related recommendations. Three scenarios representing various levels of available cost-effectiveness evidence were used to illustrate the integration process.
This GRADE guidance provides practical information for presenting cost-effectiveness data and its integration in the development of health-related recommendations, using the EtD frameworks.
A multimethods randomized trial found that plain language versions improved parents’ understanding of health recommendations
2023, Journal of Clinical Epidemiology
To assess the effectiveness of plain language compared with standard language versions of COVID-19 recommendations specific to child health.
Pragmatic, allocation-concealed, blinded, superiority randomized controlled trial with nested qualitative component. Trial was conducted online, internationally. Parents or legal guardians (≥18 years) of a child (<18 years) were eligible. Participants were randomized to receive a plain language recommendation (PLR) or standard (SLV) verison of a COVID-19 recommendation specific to child health. Primary outcome was understanding. Secondary outcomes included: preference, accessibility, usability, satisfaction, and intended behavior. Interviews explored perceptions and preferences for each format.
Between July and August 2022, 295 parents were randomized; 241 (81.7%) completed the study (intervention n = 121, control n = 120). Mean understanding scores were significantly different between groups (PLR 3.96 [standard deviation (SD) 2.02], SLV 3.33 [SD 1.88], P = 0.014). Overall participants preferred the PLR version: mean rating 5.05/7.00 (95% CI 4.81, 5.29). Interviews (n = 12 parents) highlighted their preference for the PLR and provided insight on elements to enhance future knowledge mobilization of health recommendations.
Compared to SLVs, parents preferred PLRs and better understood the recommendation. Guideline developers should strive to use plain language to increase understanding, uptake, and implementation of evidence by the public.
Improving grading of recommendations assessment, development, and evaluation evidence tables part 4: a three-arm noninferiority randomized trial demonstrates improved understanding of content in summary of findings tables with a new format
2023, Journal of Clinical Epidemiology
To evaluate alternative formats of summary of findings (SoF) tables for single comparison with multiple outcomes.
We conducted a three-arm randomized controlled noninferiority trial (RCT) in the following systematic review (SR) users: researchers, clinical practice guideline developers, health care providers, policymakers, and knowledge transfer organizations to measure understanding, accessibility, satisfaction, and preference across the current grading of recommendations assessment, development, and evaluation (GRADE) SoF, an alternative GRADE SoF, or an adapted evidence-based practice center (EPC) program SoF table.
One Hundred Seventy-Nine participants were randomized, and 129 participants completed the RCT (n = 47 current GRADE, n = 41 alternative GRADE, n = 41 adapted EPC). Understanding the certainty of evidence and treatment effect was comparable across groups. The adapted EPC SoF table was inferior for quantifying risk and RD compared to the alternatives (<35% correct vs. >85% correct). Participants reported increased satisfaction when SoF tables presented number needed to treat (NNT), anticipated absolute effect differences, and narrative syntheses for evidence that could not be meta-analyzed. Participants reported accessibility to information as significantly better in both GRADE SoF tables, when compared with the adapted EPC SoF table. Participants preferred the alternative GRADE SoF table format.
The alternative GRADE SoF table is a promising format for SR users preferring a comprehensive presentation of SR results for single comparisons.
No difference in knowledge obtained from infographic or plain language summary of a Cochrane systematic review: three randomized controlled trials
2018, Journal of Clinical Epidemiology
The aim of this study was to test the usefulness of an infographic in the translation of knowledge about health information from a Cochrane systematic review to lay and professional populations in comparison to a plain language summary (PLS) and scientific abstract (SA).
We conducted three parallel randomized trials with university students (n = 171), consumers (n = 99), and doctors (n = 64), to examine the effect of different summary formats of a Cochrane systematic review on the knowledge about health information presented in the review, reading experience, and perceived user-friendliness. In the trials involving students and doctors, an infographic was compared to a PLS and a SA, while in those with consumers, an infographic was compared to a PLS.
We found no difference in knowledge between the infographic and the text-based PLS in any of the trials or in the whole participant sample. All three participant groups preferred the infographic and gave it higher ratings for reading experience (d = 0.48 in the overall sample) and user-friendliness (d = 0.46 in the overall sample).
Although the infographic format was perceived as more enjoyable for reading, it was not better than a traditional, text-based PLS in the translation of knowledge about findings from a Cochrane systematic review.
RE: Which estimates comprise optimal reporting in systematic reviews?
2017, Journal of Clinical Epidemiology

View all citing articles on Scopus

View full text

Original ArticleFormatting modifications in GRADE evidence profiles improved guideline panelists comprehension and accessibility to information. A randomized trial

Abstract

Objective

Study Design and Setting

Results

Conclusion

Section snippets

Background

Overview of design

Results

Main findings

Acknowledgments

J Clin Epidemiol

J Clin Epidemiol

Chest

J Clin Epidemiol

J Clin Epidemiol

Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group

BMC Health Serv Res.

Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive

BMJ

GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

BMJ

Original Article
Formatting modifications in GRADE evidence profiles improved guideline panelists comprehension and accessibility to information. A randomized trial