Development of two shortened systematic review formats for clinicians

To describe the development process of two shortened formats for a systematic review intended for use by primary care physicians as an information tool for clinical decision-making.

We chose a full-length systematic review to be used for developing prototypes by having four generalist clinicians select from a list of systematic reviews that were drawn from 120 medical journals published in the last five years on topics relevant to primary care [14]. These physicians are chosen from a pool of more than 4,000 physicians who have received formal training in rating articles to identify those that are pertinent to practice as part of a larger program in evidence-based health informatics at McMaster University, Canada [14]. The clinicians were asked to rate the articles that they believed would be important to practicing primary care clinicians using the McMaster PLUS (Premium Literature Service) 7-point Likert scale, where 1 indicates that the article is definitely not relevant and 7 indicates that it is directly and highly relevant. The PLUS scale is used by the Health Information Unit at McMaster University to identify articles for inclusion in a secondary journal (ACP Journal Club) and BMJ Updates [14]. The Health Information Unit supplied a list of 927 systematic reviews that scored 6 or better (out of 7) on the McMaster PLUS scale. Initially, two physicians (one internal medicine physician and one family physician) reviewed all the systematic reviews supplied and independently voted on the three most relevant to generalist physicians. The final review was selected by a third family physician independently. The systematic review that was selected for this study was: ‘Systematic review of rosacea treatments.’ van Zuuren EJ, Gupta AK, Gover MD, Graber M, Hollis S. Journal of the American Academy of Dermatology. 2007 Jan;56(1):107–15.

Two shortened formats were developed in collaboration with a human factors engineer using the selected systematic review. Human factors is the application of what is known about human capabilities and limitations to the design of tools in order to enable more effective use [15]. Guiding principles for user-centered design were employed, which focuses on making tools that are usable, useful and accessible for the development of the prototypes [16]. The initial prototypes were designed to be one page in length (front and back), giving them the flexibility to be viewed online (as a PDF document) or be printable. Once this is finalized, our future plans include optimizing them for handheld environments. The decision for this sequencing was based on the lack of evidence that increased availability and advances in electronic health technology affect the use of evidence in practice [17]. The first format used a case study to present contextualized information (case-based format), and the second format integrated evidence and clinical expertise (evidence-expertise format). The case-based format was designed to provide evidence within the context of a specific situation, presenting a real-world example of how the evidence could be used in decision-making. This format was chosen since text is easier to understand when it has personalized elements including examples, such as case studies [18‐22]. Personalized texts prompt readers to recall more information [21, 22], as well as allow instructions and information to be embedded more succinctly [23]. The evidence-expertise format was guided by David Sackett’s definition of evidence-based medicine, highlighting the integration of clinical expertise and the best external evidence [24]. El Dib and colleagues [25] analyzed 1,016 randomly selected systematic reviews covering a wide variety of clinical topics and found that approximately half reported results that neither supported nor refuted the intervention tested. Similarly, less than 5% of 2,535 Cochrane systematic reviews explicitly state that no more research is needed or feasible [26]. Primary care physicians expressed the need to have an explicit statement about where the evidence was absent and how clinical expertise could bridge this gap when describing their preferences for the presentation of evidence [27]. These findings indicate that supplementing the review with clinical expertise may be useful, since finding a systematic review relating to a clinical question does not assure guidance for a clinical decision. Content was developed for the case study in the case-based format, and information was obtained specifically to present an expert interpretation for the evidence-expertise format. All other information presented in the shortened formats was drawn directly from the original full-text systematic review.

Heuristic evaluation is a usability inspection method that involves the examination of the prototypes by comparing them to recognized usability principles (the ‘heuristics’) [16]. It is used to identify major usability problems of a product or tool in a timely manner with reasonable cost [35‐38]. Having a number of heuristic evaluators will identify more usability problems; however, it is recommended that a cost-benefit consideration be employed to determine the number of evaluators appropriate for an individual project [39]. Since the prototypes for this study were undergoing a multi-step development and evaluation process, we decided to use one heuristic evaluator. The consultant who carried out the evaluation had no involvement in the study. She was selected to conduct this phase as she has a PhD in mechanical and industrial engineering, and conducts research related to the science and technologies of human factors [40]. A modified set of heuristics applicable to the analysis of printed materials (based on the tool provided by Nielsen [41]) was used for the heuristic evaluation (Additional file 2). As per the heuristic evaluation methodology by Nielsen [42], the errors are first identified, then classified by severity, i.e., cosmetic, minor, moderate, major, or catastrophic. The severity estimates are based on frequency, impact, and persistence of errors.

Clinical content was reviewed by a family physician (NP). The role of the clinical content reviewer was to ensure that the information was transferred from the original document to the shortened versions accurately (and not to evaluate the accuracy or quality of the information) [43]. The clinical content reviewer is an independently licensed and active family physician with three years of clinical experience. He was selected based on clinical knowledge and willingness to volunteer time to this study. One reviewer was sufficient, as the function of the exercise was to identify obvious errors, and this was done with the knowledge that the next step in development would be to assess the prototypes using iterative focus groups with end-users (not described in this paper).

We developed and refined a summary of a systematic review on rosacea in two formats, case-based and evidence-expertise, which addressed many obstacles clinicians encounter while searching for evidence-based answers to questions. As reported in the Methods section, we selected a systematic review of rosacea treatments and developed summaries in case-based and evidence-expertise formats.

The heuristic evaluation indicated that there were no major usability problems. Several moderate usability issues were identified, including wording that could potentially confuse readers, the placement of information (e.g., an evidence rating appearing in different columns of tables), or omissions that could potentially confuse readers (e.g., no evidence ratings for some treatments). Minor issues concerned the small size of text and layout for the case-based format. We used all feedback to modify the prototype formats.

The clinical content review revealed that the evidence-expertise format accurately reflected the information in the full-length review. One issue was detected in the case-based prototype, and the reviewer recommended modifications to the case that included not focusing on iatrogenic rosacea, removing references to prednicarbate, and using the term ‘family physician’ instead of ‘general practitioner.’ All of these changes were made to the case-based prototype. Additional file 4 and Additional file 5 provide the prototypes before and after the mapping exercise, heuristic evaluation, and clinical content review.

The development process for the prototypes described in this paper needs to be considered within the context of certain limitations. It may not be possible for all groups to collaborate directly with a human factors engineer when developing information tools. One consideration is to hire consultants for this expertise and include this cost into the budget of research grants. Alternatively, online resources can be used to provide guidance [16]. A single reviewer was used for both the heuristic evaluation and the clinical content review. Although using more than one reviewer has the potential to identify more problems, a pragmatic approach was taken, and cost-benefit considerations guided this decision. For the clinical content review, using one reviewer was also influenced by the fact that the full-length systematic review came from a peer-reviewed journal, which meant that the clinical content had already gone through peer review. We made these decisions with the knowledge that this process was the first step in a multi-step strategy in which the prototypes will be tested by end-users in a series of focus groups.

In the next step in the development of these prototypes, we plan to conduct focus groups with primary care physicians to gain their input on the format, presentation and layout of the revised prototypes after all revisions had been made. The purpose of the focus groups is to generate essential components of the shortened systematic reviews, and to seek reactions to these prototypes and their potential for clinical decision making. This activity will provide the opportunity to hear from users about their requirements when using such tools, as well as to make changes and correct problems as they emerge. Iterative focus groups allow the chance to take results and quickly incorporate them into the new design. This is an important step of the Knowledge-to-Action cycle that facilitates the tailoring of information tools to the needs of potential users [13]. Following this, we will complete usability testing. Finally, we will test the prototypes in a randomized trial to determine their impact on knowledge and ability to apply the evidence to clinical scenarios.