Five shared decision-making tools in 5 months: use of rapid reviews to develop decision boxes for seniors living with dementia and their caregivers

This is a descriptive study of the steps to implement a rapid review approach as part of the creative process of developing five decision boxes (henceforth DB).

We defined the successive project steps and asked reviewers to self-report the time required to complete each of these steps, at the end of each working day, in a Google Drive worksheet (Google sheets) [23]. The first author (MAL) collected this data for each DB and from all reviewers. We used a Dropbox platform to share project documents.

We used four specific strategies to shorten the review process. The first strategy involved having a single reviewer conduct literature searches and study selection. The second strategy consisted of searching a limited number of databases selected for content relevance with regard to each of the studied decisions. The third strategy consisted of using an “overview of reviews” approach, by searching for secondary literature sources first. The fourth strategy consisted of engaging several reviewers in data extraction and having them mutually review their results as part of the process to minimize the risk of one reviewer misinterpreting the results extracted. The following sections provide more details on the strategies used.

The results from the five knowledge syntheses were reported in DB templates, which are designed in such a way as to help weigh the benefits/harms of all options in light of the patient’s individual health status, as previously published [16, 17, 30]. The DB is tailored to the needs of each type of user, as it includes a version designed for clinicians (C-DB) and a lay version for patients/caregivers (P-DB). The C-DB is designed as a continuing education activity. It provides clinicians with scientific information that they can review carefully prior to patient consultations. It is more succinct than the P-DB and allows clinicians to appraise the evidence critically by describing the design and participants of included studies and by synthesizing study limitations using the GRADE approach [31]. The P-DB, on the other hand, is designed for use during consultations to encourage discussion and to be left with the patient and caregiver for review afterwards. It is distinct from the C-DB as it presents the information in complete sentences, in non-technical language, features larger font sizes, and comprises a value clarification exercise and an instrument to screen for decisional conflict. The P-DB design is inspired by the Ottawa Decision Support Framework [32] as well as current international standards for decision aids [21].

Once the content was adapted to fit into each template, we sent it to a graphic designer who prepared printable versions of each decision box.

The number of extracted health outcomes was highest for DB1 (n = 62), followed by DB2 (n = 58).

Overall, we completed 124 GRADE quality assessments, with numbers of assessments ranging from 9 to 47, depending on the DB (Table 3). For 53 of the outcomes, we did not complete any GRADE assessment, either because the outcomes were reported in qualitative research studies (17 outcomes) or in non-scientific literature (20 outcomes), or because the outcome was not supported by any evidence (e.g., time required to attend psychotherapy) (16 outcomes).

We separated the activities leading to the creation of the prototypes into two: (a) the rapid review process, and (b) decision box prototypes design. We defined the rapid review process as the period starting when we initiated the preliminary search of the literature to the end of the integration of experts’ inputs. We defined the decision box prototype design as a stage that included adapting content (e.g., reading level) and tailoring graphic design to each type of user (clinicians and patient/caregivers). The entire project required a total of 3300 h. Of these, 2800 were required for the rapid review process and 500 for the DB prototypes design.

Depending on the DB, the rapid review process required an equivalent of 7 to 31 weeks for a single reviewer working full-time (Table 4). On average, an equivalent of 19 weeks (SD = 10) full-time was required to complete a single rapid review, representing approximately 4.8 months. These estimates comprise the time required for question refinement, inclusion criteria selection, search strategy development, but do not comprise the time of the PI, biostatistician, and information specialist. The final data extraction step was the most time-consuming, taking an average of 8 weeks for a single DB (SD = 6.8). The rapid review in support of DB5 (advanced directives) required the least time (7 weeks) of the five. The rapid reviews for DB1 (support for caregivers) and DB2 (agitation, aggression, and psychosis) required the most time (30 and 31 weeks, respectively). Depending on the DB, the experts’ comments were incorporated into the DB within 0.8 to 1.6 weeks. Based on these times, and excluding the time spent by the PI and biostatistician, we estimated that the cost of a single review ranged from C$1925 to C$30,130 (mean ± SD; C$11,646 ± 10,914).

During the decision box prototype phase, a senior research associate initially adapted the reading level and content to each type of user, which required 4.5 to 25.5 h per DB (mean ± SD, 18 ± 8 h). A graphic designer then created the printable color version of each DB, requiring on average 25 h/DB (SD: 14 h) (range: 11 to 47 h/DB). In total, we estimate that an average of 20 weeks (SD: 11) was required to complete both the review and the prototype design processes for each DB (range: 8 to 32 weeks, depending on the DB) (Table 4). Based on these times, we estimate that the cost of developing a prototype based on the rapid review results ranged from C$1194 to C$3304 (mean ± SD; C$2188 ± 982).

Overall, the costs of the whole of the activities comprising the rapid review process and decision box prototypes design ranged from C$3581 to C$27,335 (mean ± SD; C$13,429 ± 9024).

Ten reviewers were responsible for reviewing a specific DB, often in teams of two for a single DB (Table 5). They conducted literature searches, study selection and extraction, and drafted the DB content. To ensure the tools would be relevant to users, three of the reviewers were potential users of the DB, as they were undergraduate medical students. The reviewers were supervised by the PI and the project coordinator (BV) who holds a Doctorate in epidemiology (BV).

We asked content experts (physician-geriatrician, nurse, informal caregivers, managers of community-based or institutional organizations, pharmacist, and social worker) to review and provide feedback on the content of each DB before beginning work on the graphic design (Table 5). The content of each DB was reviewed by between 3 and 5 experts (Table 3). They made various comments, including suggestions about new references to consider; information to add about the evidence presented, such as drug doses and length of interventions; additional interventions; benefits or adverse effects of an intervention; and the possibility of transforming the probabilities of certain benefits and harms. For DB2 (agitation, aggression, and psychotic symptoms), three experts had serious concerns about the evidence presented relative to medications and asked that the increased risk of death in patients taking atypical antipsychotics be added. One of these experts also asked that we include a section about the conditions that need to be verified before starting any pharmacological treatment for behavioral and psychological symptoms of dementia (BPSD). The experts’ concerns regarding the current inappropriate drug prescription for BPSD in primary care led us to remove the evidence on the impact of these options from the DB. In DB4 (quality of life), two experts suggested including a number of additional treatment/intervention options. Other minor comments by the experts consisted of grammar corrections, rewording of some sentences to improve understanding, and removal of information they considered less relevant (e.g., they suggested removing the use of a therapeutic robot to improve quality of life as it is not yet available in Canada for community-based seniors with dementia).

During the course of the study, the reviewers who lacked training in the conduct of systematic reviews attended some formal training of the Cochrane for systematic review author, and team workshops led by the PI on GRADE study quality assessment. These basic training sessions were subsequently complemented with informal coaching by the PI and study coordinator (BV). At the start of each new stage of the review (e.g., selection, extraction), each reviewer’s work was revised iteratively by the PI and the coordinator, in the presence of the reviewer, ensuring it met the team’s internal standards of quality, so as to enhance the team’s expertise and the quality of their work.

To facilitate understanding of the probabilities presented by users, we presented numeric estimates of the effect of an intervention into absolute risks whenever possible. A number of the AR were already available in some of the included studies, however, we had to calculate the SMD or absolute risk differences for others. We did not transform 107 outcomes (between 1 and 37 per DB), either because the results were not statistically significant or because they were non-quantifiable. This last aspect is particularly important for DB5, which presents several legal aspects that have not been reported on in scientific studies and could thus not be quantified with regard to risks and benefits. Since some of the probabilities could not be transformed, we reported data in absolute risk reduction for 70 outcomes (25 for DB1, 22 for DB2, 10 for DB3, 8 for DB4, and 5 for DB5) out of the 177 reported.

In this study, we describe the methodology of a rapid review approach, with the aim of furthering the research on systematic reviews and on the development of shared decision-making tools. We address some of the limitations often reported with regard to rapid reviews, particularly that they lack transparency and use inappropriate reporting [7, 47, 48]. As recommended [49], we report the methodological details of our approach, to allow a careful examination of the choices we have made and reproduction of these results. This may provide a foundation for future rapid review teams, for example, when planning the resources they need [50]. Another strength of this report is the detailed documentation of each step and the careful reporting of review outcomes (time, resources).

The proposed rapid review approach. Firstly, the information specialist was involved as a consultant and did not conduct the searches, which may have affected the validity of the searches. Secondly, the experts were not blinded and so they might have refrained from making certain observations. Thirdly, the lack of quality appraisal for the qualitative studies included represents a significant limitation, especially considering the relatively important number of such studies. Fourthly, we did not create a flow diagram for each DB, to describe the number of references from each source and the reasons of exclusion.

Additional limitations concern the methods used to describe this work. First, we did not collect data to assess how the proposed approaches to accelerate review processes affected the quality of the reviews. A subsequent study could thus be conducted to compare our results to those obtained using a traditional systematic review methodology. One of the challenges of such a study would be that the current approach consists of a review of several interventions that require consideration during the course of decision-making, and this, in itself, diverges considerably from traditional systematic reviews. A second limitation is that this work relied on the reviewers’ self-reporting of the time required at each step, with risks of bias from misclassification and recall. It is uncertain to what extent such bias might impact results. The use of an online reviewing software allowing the automatic recording of the time at each review step may have been more reliable. Such reviewing software might also improve efficiency and limit the errors in data extraction, which can reach 30% in a single extraction [51]. While conversions for dichotomous outcomes were fairly straightforward, we did encounter more difficulties converting standardized mean differences to absolute risk differences. The methodology used was simple but could lead to some biases in favor of the treatment. Because of this, we believe greater care and time should be planned on these conversions, involving a biostatistician and a clinician well versed in the particular scale used.