Strategies for eliciting and synthesizing evidence for guidelines in rare diseases

We piloted the suggested framework in three specific guideline development processes undertaken in three different rare conditions. All three guidelines were coauthored and developed by authors of this report. All three guidelines also explicitly aimed to achieve a double goal: 1) produce a guideline to be used in clinical practice; and 2) implement our framework to overcome expected barriers in guideline issuing in rare diseases. The individuals most focused on the latter goal where the guideline methodologists, whilst the clinical panel members acted as end users trying out and experiencing the impact of our framework.

The first guideline focused on hemophilia, an X-linked bleeding disorder caused by the hereditary lack of a coagulation factor. Blood clotting is severely impaired in hemophilia, resulting in serious bleeding with minimal provocation. Hemophilia A (deficiency of coagulation factor VIII) and hemophilia B (deficiency of coagulation factor IX) are both rare; hemophilia A affects fewer than 1 in 10,000 people and hemophilia B affects fewer than 1 in 50,000 people. In 2016, the National Hemophilia Foundation in the United States partnered with McMaster University in Canada to publish a guideline that explored the optimal model of care delivery, and the optimal composition of the care team, in this rare disease. The guideline used the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) system and the GIN-McMaster guideline development checklist, and adhered to the principles set out by the Institute of Medicine in their 2011 document, “Guidelines We Can Trust.” [19‐21] The guideline and its associated studies have been published in the Journal “Haemophilia” and are included in the National Guideline Clearinghouse [22‐26].

The second guideline focused on catastrophic antiphospholipid syndrome (CAPS), also known as Asherson’s syndrome. CAPS is a clinical syndrome characterized by the rapid onset of multifocal thrombosis associated with multi-organ failure in patients meeting the serological criteria for antiphospholipid syndrome [27]. The mortality rate for CAPS is about 50%. CAPS is extremely rare, with approximately 500 individuals identified in the published literature since the disorder was first described in 1992 [28, 29]. In 2015, the European RARE-Bestpractices project group partnered with McMaster University to produce guidelines that explored therapeutic and diagnostic questions in this rare disease. The guidelines adhered to GRADE methodology and the GIN-McMaster Guideline Development Checklist [20, 21]. The guideline has been submitted for publication in a peer reviewed journal.

The third guideline focused on sickle cell disease (SCD), an inherited condition in which a mutated form of hemoglobin distorts red blood cells in low oxygen conditions. These crescent shaped “sickle” cells can obstruct the flow of blood in blood vessels, so that oxygen cannot reach nearby tissues. SCD results in anemia, acute painful crises, chronic pain, strokes, and damage to vital organs. The overall prevalence of SCD per 10,000 live births is approximately 30 in African Americans, 0.1 in Caucasian Americans, and 0.3 in Hispanic Americans [30]. In 2016, the European RARE-Bestpractices project (funded by the European Union Commission) organized a group that partnered with the GRADE Centers from American University of Beirut and McMaster Univerisity to produce guidelines exploring diagnostic and therapeutic questions in sickle cell disease [31]. This project’s methods were based on the GRADE and GRADE Adolopment approaches [32, 33]. The project resulted in a manuscript reporting the guideline development process and the resulting recommendations. This manuscript is in the process of journal submission.

All three guidelines were developed using an overarching approach proposed by the GRADE working group. GRADE is a methodological approach that helps guideline developers present summaries of evidence in a structured and standardized way, make transparent judgments about the quality of evidence, and then systematically move towards developing recommendations [34]. GRADE has been adopted by over 100 organizations worldwide, including the World Health Organization [35, 36]. It encompasses not only guideline development for questions about treatment, but also for diagnosis [37, 38]. Most of the applications of GRADE to date have been outside the field of rare diseases, and it is unproven if GRADE can be efficiently applied to rare diseases. For this study, we utilized the RARE-Bestpractice Working Group’s pilot framework for applying GRADE to rare disease guidelines, applying elements of it to all three guidelines and assessing its feasibility and practicability [18]. These rare disease specific elements were as follows:

In all three guidelines, elements of the RARE-Bestpractice Working Group’s pilot framework were successfully used. The specific elements, and our experience with their use, are described below.

Rare diseases pose a real challenge to patients, providers, payers and policy makers. Many of these conditions have no satisfactory therapeutic options, while others have very expensive therapeutic options with relatively little evidence to support them [2, 40]. Health care practitioners and policy makers often have little experience with rare conditions. Thus, clinical practice guidelines which summarize the available evidence and provide recommendations developed by experts in a structured and transparent manner are potentially helpful in supporting decision making at various levels.

Our study – in which we applied a framework for evidence synthesis, allowing for the flexible and transparent use of indirect, expert-based, and ad hoc newly generated evidence – successfully addressed some of the “evidence gaps” encountered when developing three target rare disease guidelines. These “gaps” are common to guideline development for many rare diseases. Guidelines in common diseases frequently rely on comparative evidence of treatment effects (generated when two or more interventions, or an intervention and a placebo, are compared). This reliance on comparative studies is often not practical for rare disease guidelines. Indeed, there is often only one treatment for any given rare disease, or there is only very low certainty evidence for the effect of different treatments.

A major strength of our study is that it shows how, in the absence of disease-specific comparative studies, valuable evidence can still be retrieved and successfully used. Systematic reviews of evidence from similar, more common diseases can provide valuable comparative information to Guideline Panels. Qualitative research has proven to be a viable data source for values and preferences, acceptability, feasibility, and equity [41]. These important aspects are underreported in both rare and common diseases. Our study suggests that patient registries are a feasible source of information for guideline developers as well. Registries can capture long term longitudinal data that are difficult to capture in conventional experimental study designs due to ethical, regulatory and financial constraints. Registries may be of particular use in rare diseases, many of which are chronic, and affect individuals over the life span. Registries are considered key instruments for developing rare disease clinical research, enhancing patient care and health planning, and improving social, economic and quality-of-life outcomes. Indeed, it is usually the case that no single institution, and in many cases no single country, has sufficient data on a given rare disease that can be applied broadly to clinical and translational research. However, the fragmentation and heterogeneity of existing registries, which are often the result of spontaneous initiatives, limit the general applicability of their observations [42]. The use of registries in guideline development is subject to other limitations. Ideally, registries must be designed to collect information on patient-important outcomes. Registries must also be set up to capture data that are amenable to pooling in order to achieve a sufficient sample size for key outcomes. This requires significant foresight as the registry is launched, ongoing efforts to maintain recruitment, and assurance that the technology to securely share data is maintained. In the CAPS guideline, many analyses could not be performed because of missing variables that would have controlled for confounding. The guideline development group could therefore provide only crude estimates and unadjusted analyses. Another concern with using registry data is duplication of data from published studies, since populations with rare disease populations are small. For example, data from published studies of CAPS patients were likely incorporated into the CAPS Registry, as part of the registry recruitment strategy is periodic canvassing of the literature to identify and recruit cases. In these circumstances, it may be important to inform the Panel that analyses from the registry could potentially be based on duplicate data already presented in the studies retrieved from the systematic review. Furthermore, use of registries almost inevitably requires involvement of the creators of the registry, which can lead to important intellectual conflicts of interest.

Another strength of our study is its confirmation that unpublished non-experimental data (including observational data and qualitative evidence) can be elicited and collected in such a way to be usable by Guideline Panels. Systematically collected evidence from field experts can provide this information; further, “going to the source” may overcome dissemination bias, albeit at the risk of introducing selection bias (which must be carefully monitored).

Either of the two approaches indicated above can overcome the tendency of guideline developers to focus only on randomized controlled trials and large observational studies to inform their work, which may not be possible outside the field of common diseases. As users demand high-quality guidelines for rare diseases, it will be important to incorporate non-experimental and non-comparative data collected in a systematic and transparent fashion.

There were clear limitations to our work, and to the RARE-Bestpractice Working Group’s framework. Qualitative research did provide valuable information in the NHF-McMaster guideline. However, in general, published qualitative data can be difficult to find in rare diseases – either because these studies are not conducted, or because they are not published. Guideline developers can conduct their own qualitative studies (as the NHF-McMaster group did), but this takes time and resources. Qualitative research may have complemented the limited quantitative evidence base in the CAPS guideline. In particular, it may have provided insight into patient perspectives, which are not reported in the literature. However, identifying patients to interview was a major limitation to conducting a qualitative study of CAPS; the number of patients diagnosed with this rare condition every year is low, the presentation is precipitous and often fulminant, and the mortality rate is close to 50%. In such circumstances, patient organizations may be able to assist with identifying patient perspectives. Partnering with a patient organization ultimately helped the CAPS guideline development team recruit a patient representative for the Guideline Panel. When published qualitative studies are available, there is still little consensus in how to rate the quality of evidence. GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research), a GRADE project group, has proposed that an assessment of confidence for individual review findings from qualitative evidence syntheses be based on four components: methodological limitations of the incorporated studies, relevance of the studies to the review question, coherence of the review finding, and adequacy of data supporting the review finding [43]. This is an active area of study, but is currently a major constraint on the utility of qualitative evidence in guideline development.

Finally, our study identified limitations to the use of expert-based evidence. Guideline developers have been advised to help Panel Members identify the evidence underlying their opinions, and judge the quality of that evidence [44]. However, expert-based evidence can be challenging to incorporate into guidelines. We maintain that it should be collected transparently and systematically, and subjected to the same level of Guideline Panel appraisal as other evidence. This evidence can be subject to bias if it tilts from objectively collected data into subjectively developed opinion. Experts should be encouraged to provide statements in advance, after systematically having reviewed cases they have managed, to provide the most objective information. This would provide Panels with unique data gained from years of clinical experience. Experts should also be required to sign off on their statements, which may increase its trustworthiness.

Further research into how novel methods of evidence creation would support the guideline update process is also needed. For example, if a guideline for a rare disease relies heavily on evidence from a more common disease with a rapidly evolving knowledge base, the rare disease guideline may require more frequent updating through reviews of indirect evidence. Guideline developers must also be able to efficiently respond to changes in the knowledge base that may impact guideline recommendations, but can only be detected by ongoing qualitative research (for example, shifts in patient values over time). Finally, it is clear that non-experimental and non-comparative forms of evidence can play a valuable role in guideline creation. Several groups, including AID (Appraising and Including Different Knowledge in Guideline Development) and GRADE-CERQual are actively exploring this concept.

After theoretically exploring the barriers to issuing guidelines on rare diseases, we proposed an operational approach consistent with GRADE [18]. We have now applied this approach in three specific rare conditions [22]. Our work confirms that priority setting, question generation and prioritization, and outcome identification and ranking, all pursued with extensive involvement of patients and experts are possible in the development of rare disease guidelines, and the process can parallel that in more common diseases. Furthermore, we have come to a number of conclusions: We have concluded that intentional and systematic gathering of (unpublished) observational evidence from experts is a more efficient and transparent alternative to informal “around the table discussion” by the Guideline Panel. We have also concluded that it is feasible to use indirect evidence from other diseases and access a patient registry of the target disease to complement published low certainty evidence. Finally, we have established the feasibility of qualitative research as an important way to glean information about values and preferences, acceptability, feasibility, and health equity – all key factors to consider when making practice recommendations.

All three rare disease guidelines in this study maintained a clear link between quality of evidence and the strength of the ultimate recommendations, as specified by the GRADE Working Group framework. Rigorous clinical practice guidelines are needed to improve the care of the millions of people worldwide who suffer from rare diseases. We recommend that systematic processes must be used to create evidence-based practice guidelines that are useful to patients, clinicians, researchers, industry, and policy makers in the rare disease community. However, these processes must also be dynamic and responsive to the unique challenge presented by rare diseases: a dearth of high certainty evidence. Innovative evidence elicitation and synthesis methods will benefit not only the rare disease community, but also individuals with common diseases who have rare presentations, suffer rare complications, or require nascent therapies. Further refinement and improved uptake of these innovative methods should lead to higher quality clinical practice guidelines in rare diseases.