Reporting quality of European and Croatian health practice guidelines according to the RIGHT reporting checklist

Health practice guidelines (HPGs) are “statements that include recommendations intended to optimize patient care. They are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [1]. Guidelines that can be trusted are not only based on a systematic review of best available evidence, but they should be developed by a multidisciplinary panel in a transparent, stakeholder inclusive, documented, and accessible process [1].

HPGs have become a standard in clinical practice and their number has been on a continual increase. PubMed search for MeSH terms “Practice Guideline [Publication Type]” or “Practice Guidelines as Topic” returned 126,573 publications on May 2, 2018, with 23,976 publications related to Europe. Despite this high production rate of and interest in HPGs, there is a great variation in their development, quality, implementation, and use, as demonstrated by the recent overview of the framework for HPGs development at the national level in Europe [2]. In many countries, the development of HPGs is tasked to different specialty/professional societies [2], which thus have a central role in translation of evidence to local practice, most often in the local language. The local guidelines are usually published in national journals or on the websites of professional societies, which serve as dissemination outlets for the HPGs.

Quality of reporting has been recognized as a key measure for successful translation of evidence to practice, reducing overall waste in research and helping in eliminating non-replicable studies [3]. Whereas a number of guidelines for reporting research results have been developed for different types of research [4], the quality of reporting of HPGs is not well known, primarily because there have been no tools specially developed for HPGs. Experience from other fields shows that reporting guidelines, such as Consolidated Standards of Reporting Trials (CONSORT) for randomized controlled trials and STARD for diagnostic studies, are effective in improving completeness of reporting in their respective fields [5‐7]. Recently, two reporting checklists have become available for HPGs: Appraisal of Guidelines, Research and Evaluation (AGREE) Reporting Checklist [8] and Reporting Items for Practice Guidelines in HealThcare (RIGHT) [9]. Whereas the AGREE checklist was developed on the concept of the AGREE tools for the methodological rigor and transparency of HPGs [10], the RIGHT checklist follows standard reporting guideline approach established by other reporting statements, such as CONSORT and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [11, 12]. The RIGHT checklist was developed to assist guideline developers in reporting, journal editors and peer reviewers in decision making, and health care practitioners in understanding and implementing guidelines [9]. So far, RIGHT checklist has only been used to evaluate Chinese guidelines for sepsis and for acupuncture [13, 14].

The aim of our study was to assess and compare the quality of reporting in nationally and transnationally produced (European) HPGs by using the RIGHT reporting checklist. We assessed HPGs developed by professional societies in Croatia, a non-English speaking country, and compared them to similar European HPGs produced by transnational professional societies. We included only European HPGs as comparators on the basis of similarity of populations they address, health problems that are explored, and also because Croatian guidelines often rely on, adapt, and cite their European counterparts.

We reviewed all published Croatian HPGs (n = 26) and 24 matching European HPGs. Eleven Croatian HPGs did not have European counterparts, and 3 of them had multiple European counterparts. Because of these discrepancies, we did not do a matched analysis. Kappa coefficient for individual 35 items of RIGHT checklist ranged from 0.76 to 1.0. All of the European guidelines were produced by transnational specialty societies, and most of Croatian guidelines were produced by national specialty societies (23 out of 26; 3 were produced by individual authors, independently from specialty societies; 1 of those declared to be supported by professional societies). All of the guidelines were produced or updated in 2014 or later. The full list of guidelines and individual results of the adherence to the RIGHT checklist are available in the Additional file 1.

Overall, Croatian and European HPGs adhered to less than 50% of RIGHT checklist items. Croatian HPGs reported a median of 14.0 (95% CI 13.0–15.0) RIGHT reporting items. Their European counterparts reported a median of 16.0 (95% CI 14.0–17.2) out of the total of 35 checklist items. The difference was not significant (P = 0.048, Mann-Whitney U test, BF₁₀ = 1.543). The lowest total number of addressed RIGHT items was 11 (31%) (Croatian HPGs on melanoma and cystine urolithiasis), and the highest was 22 (63%, European resuscitation guideline).

The results of adherence to individual RIGHT checklist items are presented in Tables 1, 2, and 3. For nine checklist items, both groups of HPGs had a satisfactory level of reporting (≥ 80% of HPGs in each group adhered to the checklist item): item 1a—clear guideline titles, described as mentioning the focus of the guideline; item 1c—defining it as guidelines or recommendations in the title; item 4—providing an address of a corresponding author; item 9a—lists of authors and their affiliations; item 5—description of basic epidemiology of health problems; item 7a—description of primary populations addressed by the HPG; item 13a—clear and actionable recommendations; item 13b—separate recommendations for important subgroups; and item 13c—indicated strengths of recommendations and certainty of evidence.

For 12 out of 35 RIGHT items, both sets of HPGs had poor reporting (≤ 20% of HPGs in each group adhered to the checklist item): item 9a—description of roles and responsibilities of authors and contributors; item 10a—a clear and structured healthcare question in PICO or other appropriate format; item 10b—clearly defined outcomes; items 11a and 11b—process of selection of systematic reviews; item 17—description of quality assurance process; item 18b—reporting on the roles of funding bodies in the development; item 19b—management of conflicts of interest; item 22—description of limitations of the guidelines, such as risk of bias, conflicts of interest or lack of patient involvement; and item 20—availability of guidelines and additional documents.

European guidelines were significantly better in providing a HPG summary (item 2; BF₁₀ = 4.44), reporting stakeholder involvement and values and preferences (item 14a; BF₁₀ = 80.63), as well as describing the implications of costs and resources (item 14b; BF₁₀ = 55.15, Table 2). They also reported better than Croatian HPGs on existing conflicts of interest (item 16; BF₁₀ = 55.35), external reviews (item 19a; BF₁₀ = 1.02 × 10⁸), and on sufficiency of evidence for recommendations or provided suggestions for future research (item 21; BF₁₀ = 733.70).

Croatian HPGs were significantly better in reporting specified aims of the guideline (item 6, BF₁₀ = 16.90), their primary intended users (item 8a, BF₁₀ = 8.70), and sources of funding (item 18a, BF₁₀ = 122.90).

For item 12 (description of the approach used to assess the certainty of the body of evidence), the Croatian and European HPGs sets had similar adherence (73.1%, 95% CI 51.9–87.7 for Croatian, and 75.0%, 95% CI 55.1–88.0 for European HPGs; BF₁₀ = 0.30). We also extracted the information on what sort of approach was used in the HPGs. Grading of Recommendations, Assessment, Development and Evaluation (GRADE) handbook [19] and Oxford Centre for Evidence Based Medicine levels of evidence [20] were predominant approaches in Croatian HPGs (53.8% of HPGs), but were less frequent in the European HPGs set (16.7% of HPGs). European HPGs mostly mentioned adapted Infectious Diseases Society of America–United States Public Health Service Grading System [21] (45.8% of HPGs). Other grading systems, such as American College of Chest Physicians (ACCP) modified GRADE system [22] or Scottish Intercollegiate Guidelines Network (SIGN) guide [23], were mentioned in 7.7% of Croatian and 12.50% of European HPGs. Some HPGs were unclear about the grading systems, mentioning the levels of evidence but not referencing any grading systems (23.1% of Croatian and 16.7% of European HPGs). Finally, 15.4% of Croatian and 8.3% of European HPGs did not mention or imply the use of any grading systems for levels of evidence.

Our study showed that both national and transnational HPGs were poorly reported in journals, with adequate presentation for fewer than 50% of necessary reporting items. This is, to the best of our knowledge, the first evaluation of HPGs in Europe using the RIGHT reporting checklist, developed for guidance in reporting guidelines in healthcare, including clinical and public health areas. It is important to keep in mind that the RIGHT checklist is not a guide for development nor a tool for assessment of methodological quality of HPGs, but a tool for checking the clarity of presentation of the HPGs to a wide audience, including both developers and readers, so that HPGs could be adequately understood and implemented in practice [9].

Our finding of inadequate reporting of HPGs matches previous observations of poor methodological quality of HPGs produced by specialty societies [24, 25]. There was no difference between the quality of reporting of national HPGs, which are directly implemented in practice in individual health care systems, and those produced by transnational societies, which put emphasis on multidisciplinary collaboration and have standard operating procedures for guideline development [26].

The relatively small sample size is the most important limitation of our study. The sample size was determined by the cohort of available Croatian HPGs. We did not include HPGs published earlier than 2014 as they were not numerous and their presentation was of poor quality (our unpublished observation). The small sample size was also the reason why we based the interpretation of results on the results of Bayes and not frequentist statistics. The advantages of coherency, independency of the intention with which data are collected, and minimum bias toward null hypothesis made Bayesian inference valid for small sample sizes [27]. It is also important to keep in mind that we analyzed national HPGs in a single country, so the results may not be generalizable to the whole population of nationally produced guidelines. However, the similarity of findings for Croatian and matching transnational guidelines indicates that the level of reporting quality is similar across professional societies in Europe. Finally, it is important to keep in mind that the RIGHT checklist does not evaluate the validity of the actual recommendations in the HPGs, but only the quality of their publicly available presentation.

The analysis of compliance with individual reporting items demonstrated that there was an overall insufficient emphasis on providing clear aims and a summary of recommendations. Guideline developers failed to provide clear instructions on how the guidelines are intended to be used and by whom. In this way, basic principles of evidence-based medicine had not been addressed because there was little information on the users of HPGs, the purpose of HPGs, and acknowledging barriers and constraints that may limit their implementation [28].

Both national and transnational sets of HPGs adequately reported on strengths of recommendations and certainty of supporting evidence, but details on how the evidence was searched for and evaluated were unsatisfactory. Established grading systems, such as GRADE and Oxford EBM, were used in half or less than a half of the analyzed HPGs, and 22% of HPGs were unclear about grading evidence. This means that current HPGs do not satisfy the preference of clinicians for recommendations that have clearly presented processes behind them and provide evidence summaries [29, 30]. Clear and transparent reporting should also help address problems such as relying on lower levels of evidence than stated in HPGs [31], relying on low levels of evidence to form recommendations [32], and inadequate presentation of evidence grading [33].

In national HPGs, reporting on the inclusion of patient values and preferences in the making of recommendations was poor, and reporting cost and resource considerations was almost non-existent. This reflects generally unsatisfactory involvement of the patients in guideline development [34], as well as integration of evidence on patient preferences [35], even though the involvement of patients and other stakeholders is especially important for development and adaptation of HPGs oriented at lower- and middle-income populations, which often have limited resources and different patient experiences and priorities [36, 37].

We also found that reporting of conflicts of interest (COI) was worse in Croatian than in European HPGs, but that both groups of guidelines showed low adherence to providing declarations on the conflicts of interest and elaborating policies regarding their management. Available data show that the prevalence of conflicts of interest in guideline development is high [38], and that conflicts of interest can influence interpretation and recommendations of treatments [39], as well as definitions of diseases [40]. Croatia is a country that follows the practice of the European Federation of Pharmaceutical Industries and Associations (EFPIA) Code for disclosure of financial transfers [41], but does not have a national legislation that requires disclosure on a central platform, such as Denmark [42]. Even in systems with mandatory conflict of interest declaration, like Denmark, there is a discrepancy between disclosures in the national register and self-reporting in publications [42]. In Croatia, only 11% of physicians made their individual data available in 2016, when the first report from the Innovative Pharmaceutical Initiative, an association that gathers 33 pharmaceutical companies that cover 60% of Croatian pharmaceutical market, was published [43].

Our study provides evidence to recommend the use of reporting checklists to guideline developers both at national and transnational levels. A variety of tools have been developed to help translation of evidence to reliable and high-quality recommendations, including GRADE system for rating quality of evidence [44], AGREE tools for guideline methodology [10], checklists for development, implementation, evaluation, and quality [45, 46]. Without complete and transparent reporting, there is no way of knowing how good and how appropriate HPGs are for use in specific clinical circumstances. In times of overwhelming production of guidelines, which are supposed to help the end-user in finding and summarizing the best evidence [47], it is particularly important to achieve clarity and enable informed decisions. In future research, the usefulness of RIGHT reporting guideline should be tested in actual health practice settings and they should be compared with similar reporting guidelines, such as AGREE Reporting Checklist [8]. Endorsement of reporting guidelines for HPGs, such as RIGHT, by professional associations, health websites, and journals would contribute to the effort of transparency for developers and to the aim of adequate implementation for users of guidelines. Only then can we hope to achieve the goal of full transparency, comprehension, and availability of information that concerns a variety of stakeholders.