Development and usability testing of Understanding Stroke, a tailored life-sustaining treatment decision support tool for stroke surrogate decision makers

A Patient and Family Advisory Council (PFAC) was formed to provide continuous input on the development of the tool. The Council included stroke survivors and family members of survivors recruited from the patient population of the study center. The Council met at the beginning of the study in December 2017 to share their experiences with stroke with the study team, and also give feedback on specific proposed elements of the decision support tool. These tool elements included graphical displays of prognostic information (pie charts, bar charts, and pictographs/icon arrays); simplified descriptions of a standardized stroke outcome scale (modified Rankin scale) [14‐17]; and a list of valued life activities to include in a values clarification exercise [18]. Subsequently, members of the PFAC provided ongoing feedback to the study team on draft versions of the materials and participated in up to two usability testing sessions during the final stages of development.

Focus groups of key stakeholders were recruited between December 2017 and April 2018. Groups were segmented intentionally to encourage candid responses. The four groups were as follows: nurses, therapists, and social workers with expertise in stroke or palliative care; physicians with expertise in stroke; stroke survivors and family members; and family members without stroke patients present. Provider participants were selected through a convenience sample from two hospital systems. Stroke survivors and their families, as well as bereaved family members, were recruited by contacting subjects from past research studies who agreed to be contacted for future research, as well as by sending out invitations to the Michigan Medicine Patient and Family Centered Care (PFCC) Program. Each focus group lasted approximately 2 h, and was primarily facilitated by author CA, a cardiovascular nurse researcher with extensive experience in conducting focus group sessions, and assisted by authors DZ, the principal investigator and a stroke physician, and WN, an expert in human computer interaction. Providers were asked about their experiences communicating stroke prognoses with patients and families, whether they use numbers or percentages and why. Stroke survivors and family members were asked about their experiences talking with the doctors and nurses early in the hospitalization about prognosis. Specifically, they were asked who discussed this with them, how the information was communicated and whether it was helpful, and what they think patients and families should have been told early on in the hospitalization. All groups provided feedback on draft elements on the tool, including graphical displays, descriptions of prognostic information, the language used to describe stroke outcome categories from the modified Rankin scale, and the list of valued life activities for the values clarification exercise. Focus group conversations were audio recorded and key themes were summarized by the study team. Demographic data for the focus group participants are listed in Additional file 1.

After the focus groups were complete, the study team began developing the tool content. A content map was developed by a multi-disciplinary team including experts in vascular neurology, cardiovascular nursing, psychology, human computer interaction, and tailored behavioral change and health education interventions. The tool content was divided into 6 main sections – About Stroke (general introduction), Stroke Outcomes (levels of disability and prognostic calculator), In the Hospital (common treatments), Making Decisions (goals of treatment and values clarification), After the Hospital (rehabilitation options), and Asking Questions. Developing the content map was an iterative process where the organization of topics was rearranged to achieve the best flow, with the final content map shown in Fig. 1. Introductory descriptions of ischemic stroke and intracerebral hemorrhage were adapted from existing materials [19, 20]. A behavioral scientist with expertise in developing tailored health communication interventions revised the content to fit the goals of the decision support tool and ensured that it was written in plain language. Illustrations demonstrating ischemic stroke and intracerebral hemorrhage were also developed. The unique elements of the tool (e.g. prognostic calculator and values clarification), were written in more detail than the informational components, such as what rehabilitation options are available after leaving the hospital. Key information was then summarized into a printer-friendly summary report (see Additional file 2) that the surrogate decision maker could use to facilitate their communication with the treating health care team.

Ordinal prognostic models for Understanding Stroke were developed separately for ischemic stroke (IS) and intracerebral hemorrhage (ICH). We chose to develop a novel model rather than use an existing model as most existing models predict a binary outcome rather than a full ordinal range of outcomes after stroke [21, 22]. Detailed statistical methods for model development will be reported separately [23]. Briefly, data for the model development was taken from the Virtual International Stroke Trials Archive Plus (VISTA-Plus) [24] which has 90-day mRS data for 9904 IS patients and 890 ICH patients. The mRS was selected as the outcome based on its broad availability in stroke studies and the large number of cases with data available in VISTA-Plus. Common predictor variables for both IS and ICH include age, NIH Stroke Scale (NIHSS), history of atrial fibrillation, diabetes, and prior stroke. For ischemic cases, other predictors include the NIHSS Level of Consciousness score (item 1a from the NIHSS) and whether the patient received intravenous tissue plasminogen activator (IV tPA), while the ICH model included sex as a predictor variable. When a surrogate is enrolled, the study team manually entered predictor variables into the website’s study team dashboard, which then calculated the results and fed the information to the Understanding Stroke website. As the direct display of prognostic calculator information to surrogates was a novel aspect of this tool, it was designed with an option to disable the display of calculator information at the discretion of the investigators. This option was created to guard against the possibility of concerns about model accuracy for individual patients with unusual features at presentation, or in the event of negative feedback from early enrollees.

Medical decision making experts generally recommend the inclusion of values clarification methods in decision support tools to help elicit values relevant to the decision [18, 25]. We conceptualized the values clarification exercise in this setting as helping the surrogate to think about which abilities are essential for the patient’s quality of life.

To begin developing the values clarification exercise, a preliminary list of important activities of daily living and functional abilities was created by adapting items from other tools [26, 27] and by adding individual items suggested by the focus group participants. Duplicative or similar items were merged, and the study team added a few items common in post-stroke patients (e.g., “Could not move one side of their body”) to make a comprehensive list. To assess the relative importance of each item and help to assure readability, the comprehensive list was made into a card sorting exercise. A convenience sample of volunteer participants was recruited from outpatient stroke and cardiovascular clinic waiting areas. Participants were shown, in random order, a list of the 14 items that may be important for people to think about when making medical decisions for a family member. The participant was then asked to rank the items on a board in order from most important to least important and add any additional items they felt were missing from the pool. Any item that did not play a role in decision making was asked to be left off the board. Therefore, the number of items ranked varied for each participant. We recorded their responses and scored the items according to their rankings.

Finally, two rounds of usability testing were completed with volunteers from the PFAC, focus group participants, and health care providers. In these sessions, participants were first asked to locate specific pieces of information or complete certain tasks within the tool. Then they were asked a few questions from a modified Systems Usability Scale (SUS) [28] regarding the level of difficulty and complexity of using the tool. Lastly, they were asked a series of open-ended questions about what they liked the most, what could be improved, and whether they felt anything was missing from the website. Participants in the first round of usability testing also participated in the card sorting exercise mentioned above.