What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids

The Cochrane Risk of Bias tool was used to evaluate the articles related to the DAs across six domains: generation of random allocation sequence, concealment of allocation sequence, blinding, incomplete outcome data, selective outcome reporting and other biases [18]. Articles were assessed for bias by one reviewer with any ambiguity resolved through discussion with a second reviewer.

The short form of the International Patient Decision Aid Standards Instrument (IPDASi-SF) was used to assess the quality of the DAs [19, 20]. The IPDASi-SF comprises 16 items across seven dimensions, including: information, probabilities, values, development, disclosure, decision support technologies evaluation, and evidence. DAs were assessed by one reviewer with a second reviewer evaluating a random sample of 30% of DAs. Disagreements were resolved through discussion between the two reviewers.

As no tool existed to complete this task, a novel classification tool (Additional file 2) was developed by the authors to analyse the various types of evidence obtained through data extraction. The purpose of this tool was to determine the extent to which research, practice, patient and contextual evidence were incorporated within DA content, the methodological rigour used to obtain this evidence and the clarity of evidence sources within the DA. Supporting documentation (articles) was used to clarify/establish information where necessary. The extraction tool consisted of a hierarchy of classification criteria for each form of evidence. The criteria for research evidence was categorised according to an eight-point rubric based on evidence level and transparency ranging from A1 (systematic reviews of randomised clinical trials (RCTs), meta-analyses or multiple RCTs with sources available in DA or associated documents) to D2 (use of research evidence not evident). The criteria were based on the levels of evidence from the European Society of Cardiology guidelines [21] and aligned to the hierarchy of evidence outlined in the OCEBM Levels of Evidence tables [22]. Practice and patient evidence consisted of a seven-point rubric ranging from A1 (expert or clinical consensus of ≥15 participants obtained directly or through published evidence with evidence sources available in DA or associated documents) to D (no evidence included). Factors considered in classification related to the number of participants, the strength of methodology employed in obtaining the information and the clarity of evidence sources within the DA. Direction for a representative sample size was guided by recommendations for powering Delphi consensus, which suggests a minimum of 15 respondents [23]. Thus, the grading system awarded a higher score for studies including 15 or more participants for practice or patient evidence. Criteria for contextual factors were simplified to a three-point scale based on quantity ranging from A (includes multiple contextual factors e.g. time, cost, resources for each treatment option) to C (does not include any contextual factors). The tool was tested by three reviewers and went through an iterative process (five iterations) until the authors were satisfied that the criteria accurately represented the types of evidence and specific factors to be addressed. Data extraction and grading was completed by three reviewers independent from each other. One reviewer conducted extraction and grading for all DAs while two other reviewers evaluated a 50% sample each. Disagreements were discussed and resolved between the reviewing team.

Research evidence was well represented in most DAs evaluated (Table 6). The majority (7/10) incorporated a range of research evidence including articles of high methodological rigour such as systematic reviews, meta-analyses or randomised controlled trials. One DA was categorised as an A2 as it used a combination of national clinical guidelines and a cohort study to populate content [30]. In some cases, it was unclear as to the type and rigour of research evidence used to inform DA content and this was reflected in categorisation. Mott et al. [32] for example, referred to research findings but did not provide citations to studies used [32]. Vandemheen (2010) reported that the DA content was based on a literature review, patient interviews and data from a national registry [35] but did not delineate the sources of all DA information.

Use of practice evidence to populate DA content was less evident being incorporated into only 3/10 DAs (Table 6). Two DAs, produced by the same organisation, were categorised as B1 on the grading rubric as they directly incorporated clinicians’ opinions through interview methods [24, 34]. Despite this, both DAs recruited less than 15 clinicians/experts and therefore did not receive the maximum score. One DA [27, 31] was graded as C2 despite incorporating practice evidence as it utilised an unknown number of members of the research team to retrieve this evidence which was also not presented explicitly within the aid and used mainly to supplement low level research evidence about the specific topic [37].

A third (3/10) of DAs integrated patient evidence as a source of information (Table 6). Two DAs, produced by the same organisation scored A1 in this category interviewing 30 people considered to represent the target audience [24, 34]. The methodology, number of participants and explicit presentation of patient evidence justified the highest grade for this category. The other DA graded as B2 as it incorporated hypothetical patient stories played by actors but did not report on how the script was developed [26].

Contextual factors were generally addressed to some degree by all DAs (Table 6). The majority (9/10) typically presented 1–2 contextual factors per treatment option. One DA presented more than two [35]. Issues addressed by DAs aids included: daily routine (6/10) including dose and particulars such as how to take medication, cost (5/10), length of hospital stay, if any (3/10), time to execute treatment including consultation length and/or number of weekly sessions involved (2/10), and other practical issues e.g. self-monitoring (1/10), waiting time for procedure (1/10) and extra testing or doctor visits (1/10) as a result of the treatment.

This novel review found that the information used to inform the content of the ten DAs examined is primarily founded on research evidence and contextual factors. Research evidence was well represented across the DAs evaluated, the majority using high quality research evidence categorised according to the A descriptor. This compares to the findings of a systematic review by Montori et al. [9], which found that half of the DAs examined used high quality evidence [9]. Furthermore, for the DAs examined the research evidence was explicit as citations for research evidence were provided in almost all aids. This finding compares favourably to previous research, which identified between a fifth (21%) to half (50%) of DAs provided citations for research evidence [9, 10]. The difference in findings may be explained by a number of factors, specifically the particular nature (treatment) of the DAs included in each study, the relative recency of those in the current study which reflect on the application of the 2006 IPDAS standards [12] and the bias that may result from including only DAs which had supporting documentation. While this was necessary to examine DA content in depth, it also limited inclusion and therefore may not represent all available DAs, implying that Montori et al’s [9] and Feldman-Stewart et al’s [10] findings may be more representative of the general pool of DAs. Topics regarding the medical condition, treatment options, expected outcomes and side effects were generally informed by research evidence reflecting standards advocated by International Standards for Patient Decision Aids Collaboration [12]. DAs which scored lower in this domain did so either because of lack of availability of high quality evidence or by not providing sufficient information to enable an evaluation to be made of the quality and sources of evidence. One DA reported using treatment manuals and a national website dedicated to the condition for treatment information [32] raising issues of information bias and comprehensiveness. In order to facilitate fully informed decision making by patients, up-to-date and complete scientific evidence should be used in DA content [10] which can be achieved via a systematic process of reviewing the literature and explicitly reporting scientific uncertainty where applicable [1]. Notably, most of the randomised controlled trials evaluating the effectiveness of the DAs were mainly of unclear bias, demonstrating that even when the appropriate study design is implemented it does not guarantee the quality of evidence within the DA itself.

As with research evidence, contextual evidence was reported in all aids with a wide range of factors being addressed and each DA containing at least one such factor. The inclusion of contextual information, particularly those representing treatment burden such as out-of-pocket costs and treatment intensity, can be seen as highly patient-centric given that such factors have the potential to dominate patient decision making [38]. DAs which include such content are likely to be more patient relevant and user friendly. Furthermore, complementing research evidence with matters related to the specific sociocultural context of the patient may serve to improve patient engagement with information [39].

Practice and patient evidence were conversely not well represented in DA content with only a small subset of DAs incorporating this type of evidence. Further, the quality of such evidence varied and the sourcing of this information was not always transparent suggesting less rigorous rules for obtaining and using such evidence. Practice evidence was least commonly used to populate DAs with the majority (70%) not incorporating any clinical or expert evidence. This absence exists despite findings which demonstrate that patients value the opinion of health professionals when making decisions [40] and often regard clinical opinion as more important than information on intervention risks and benefits [41]. This relative lack of inclusion of practice evidence in DAs may be due to a number of factors including the perception of clinical expertise as a less reliable or significant form of evidence [22, 42]. It may also emanate from a perception that asking clinicians to comment on the aid during the design phase (rather than contribute collectively to the content) constitutes practice evidence. Additionally, the patient-clinician conversation itself when using DAs may be interpreted as utilising practice evidence suggesting that, unlike with research evidence, a singular clinical opinion may be considered sufficient to represent this form of evidence. Of the DAs in this study that incorporated practice evidence, two used clinicians to present factual information, such as describing treatment features, as well as report their clinical opinions [24, 34]. There was a lack of transparency with regards to how such experts were selected. Another DA supplemented weak research evidence about medication side-effects with clinical knowledge from the design team. This was retrieved through discussion and it is not known how many clinicians were consulted in the process [37]. Thus, in both cases issues with methodology and rigour in attaining practice evidence were present. The differences observed in retrieval and use between practice and research evidence suggests different value systems for different types of evidence. Practice evidence when used appears not to be rigorously obtained through formal methodologies, rather being frequently obtained through members (often few in number and unspecified) of the DA design team. Moreover, it appears primarily used as a supplementary source of information (especially when research evidence is lacking) rather than in its own right. As for research evidence, it is no less important to understand how and where from practice evidence is retrieved in order to ensure unbiased and systematically obtained information is delivered to patients [1].

The use of patient evidence in DAs is a contentious area with lack of consensus regarding its optimal use [39, 43]. This is reflected in the limited use of patient evidence in the DAs examined with the majority, (70%) not incorporating patient experiences and opinions. This is analogous to findings regarding practice evidence suggesting a corresponding low valuing for this type of information. It is in contradiction to patient evidence having been shown to facilitate patient understanding, coping with illness and adjustment to treatment [44‐46], it being a useful means of providing insight into a particular experience about a condition. Research suggests that patients experiencing a range of health conditions have expressed positive responses to inclusion of patient evidence in health information and would like it to be integrated in health education [44]. There are clear suggestions of the relative valuing of such information by patients: they regard narrative information to be as credible as research evidence, and; patient narratives are suggested to be more persuasive than research evidence with patients often basing their treatment decision on the experiences of others above factual representations [43, 47, 48]. Thus, decision aid developers may purposely limit the focus on patient evidence in DA content both because of lowly valuing of such information and due to concerns around its optimal use for promoting unbiased, informed decisions [43]. These concerns may be balanced by involving patients in DA development from the offset for example, by providing input on the range of outcomes that patients consider critical in decision making (in addition to researcher-driven outcomes) and incorporating user panels comprising patients and caregivers (in addition to clinicians, experts and researchers) in an iterative design process focused on representing and balancing all forms of evidence [13]. Inclusion of patient evidence in DAs would address the real life experiences of decision aid users [49], thus empowering patients to make decisions based on a range of information that considers their individual context, goals, values and preferences.

The reduced practice and patient content in DAs may further reflect under researching (and thus lack of availability) of these forms of evidence. Thus, there is a requirement for both forms of evidence to be acquired using established research methods. Such methodologically rigorous research would help improve the reliability and validity of such evidence in addition to strengthening the relevant content used in DAs. It would also be useful when evaluating DAs which have been developed with the totality of evidence in mind, that studies be undertaken to examine how patients’ decisions are influenced by such individual forms of evidence. Further, as Violette et al. [50] point out, the content of DAs needs to remain current [50] to reflect additional and evolving evidence and policy developments.

These recommendations reflect a global move toward patient inclusion in their health care in a variety of ways specifically regarding fully informed shared decision making. As for other decision support tools, the development of DAs is a positive step in this regard and their importance is highlighted in policy and legal developments such as The Patient Protection and Affordable Care Act (2010) in the United States of which Section 3506 calls for the creation of a certification and quality process for patient decision aids and for best practices which include the sharing of developed tools [51]. Such quality processes should include the inclusion of multiple forms of evidence, rigorously acquired evidence irrespective of type and explicit sourcing of information used in DA content. To further progress this constructive movement, it is important moving forward that information representativeness, transparency and selectivity in DAs are targeted in order to best and fairly meet patient needs.