Active Involvement of End-Users in an EHR Procurement Process: a Usability Walkthrough Feasibility Case Study

Government usability standards—including those from the US Office of the National Coordinator (ONC) of Health Information Technology (HIT)—emphasize the importance of user-centered design to maximize electronic health record (EHR) usability.^[1] Published studies^[2,3] and best-practice recommendations^[4,5] show that improving user experience can promote technology adoption, increase system efficiency^[6,7], improve patient safety^[8‐10], and reduce clinician burnout.^[11] Nevertheless, clinicians cite EHR design as one of the most common causes of clinical errors, long work hours, and dissatisfaction.^[12,13] Furthermore, not all vendors—including those headquartered in the US—integrate user-centered design principles into the product lifecycle.^[2,14] It is incumbent upon healthcare organizational leaders and clinical champions to be actively engaged throughout implementation from technology procurement, through system configuration, to eventual deployment and training.

How a clinician uses an EHR depends on the system's work context and socio-technical characteristics (i.e., technology, organizational climate, and user).^[15‐17] The specialty and practice settings are critically important to consider. For example, an emergency physician will not use the EHR like a geriatrician—each will have specific needs within their workflow. It is, therefore, a best practice to include end-users in the EHR procurement process.^[18‐21]

The methods organizational leaders use to engage end-users can influence the selection process outcome.^[22‐25] While executives often only invite end-users to EHR demonstrations,^[24] there are practical ways for users to evaluate products during demonstrations. Project managers can administer the Usability Questionnaire for Demonstrations in Procurement (DPUQ)^[26] to gather end-user perceptions. Usability experts can then supplement this with a heuristic evaluation during the demonstration (HED).^[27] The HED is a low-cost method where an expert rates a product against a set of usability heuristics.^[28‐30]

These methods notwithstanding, data gathered during vendor demonstrations may not be the best quality since users rarely interact with the EHR, and sales representatives can hide product weaknesses.^[31] Prospective customers can gather more predictive data by conducting usability tests with clinical information processing scenarios (CLIPS). CLIPS are scripts representing clinical situations with tasks for users to complete while experts observe and collect data.^[23,31,32] Stakeholders, however, may be apprehensive about the time and cost of simulation testing of multiple EHRs with CLIPS.^[33,34] Heuristic evaluations are typically more cost-effective but do not involve end-users and can miss important issues.^[22,23,28] We see a need to combine these methods into a single protocol.

Schumacher and colleagues described a pragmatic approach to involve end-users in the procurement process: the usability walkthrough.^[34] During a usability walkthrough, end-users complete CLIPS with the EHR and classify usability problems using a set of heuristics. After the walkthrough, participants rate effectiveness, efficiency, and satisfaction.

A usability walkthrough is like simulation testing in that it involves end-users. In user simulation testing, experts identify problems while watching users test the technology, whereas, in a usability walkthrough, users identify the problems.^[28,34,35] The usability walkthrough is also like a cognitive walkthrough—both methods require users to think and talk through a clinical scenario. However, the cognitive walkthrough evaluates product learnability by asking standardized questions about interface intuitiveness and the ability to guide users through tasks.^[35,36] The usability walkthrough, by contrast, measures multiple usability dimensions by replacing standardized questions with heuristics. To conduct a usability walkthrough correctly, usability professionals train the end-users to apply the same heuristics experts use.

To summarize, usability walkthroughs permit users to compare EHRs without vendor interference and allow clinicians to champion their needs and preferences.^[34] Yet, no studies have formally investigated the feasibility of conducting a usability walkthrough during an EHR procurement. We sought to close this knowledge gap by asking four questions: (1) Are end-users able to detect, describe, and prioritize usability problems? (2) Does the usability walkthrough method help identify problems only detectable by clinical experts? (3) How satisfied are end-users with the usability walkthrough process? (4) What are the challenges of implementing a usability walkthrough during EHR procurement? To answer these questions, we conducted an implementation study of a usability walkthrough as a hospital transitioned to a new commercial EHR. In this article, we report on the method's feasibility. The results of the EHR evaluation are published elsewhere.^[37]

Leadership at a private, non-profit, 1000-bed teaching hospital in Lille, France, issued a request for proposals (RFP) and organized a procurement process to select a replacement for their current commercial EHR. The process included three steps: (1) a vendor demonstration; (2) a usability walkthrough with each candidate EHR; and (3) technical and economic comparisons of EHRs selected during the walkthrough. The comparisons focused on “back-end” functionality (e.g., data interoperability). In this manuscript, we describe the second step of the process (i.e., usability walkthrough).

To conserve resources and adhere to a timeline, it was necessary to quickly thin the pool of EHR candidates for later technical evaluation. We did not set out to exhaustively safety test products or generate summative statistics during the second phase. Therefore, we limited the number of users recruited.

The project manager (AP) and four usability experts (RM, SG, JS, SP) designed a usability instructional session and a usability workshop that included structured exercises and evaluation instruments. We planned to conduct two instructional sessions and five workshops over three weeks from September to October 2020 (Fig. 1).

Participants reported 361 usability problems. We excluded 21 issues (5.82%) using our eligibility criteria. After deduplication, the final list consisted of 265 problems: 258 detected by end-users (97.36%) and 7 detected only by usability experts (2.64%) (Table 2). Each end-user within a professional role detected between 26.82 and 70.37% of all problems identified by that group (mean = 42.92%; SD = 14.11). On average, 59.83% of the problems were detected by one participant, 23.73% by two, and 12.75% by three.

End-users assigned a criterion to 218 of the 258 problems (84.49%); 157 matched those assigned by usability experts (72%; Krippendorff’s α = 0.66 [0.59; 0.74], Table 3). The largest discrepancies were in three categories: “guidance,” “workload,” and “compatibility.” Participants assigned a severity level to 217 of the 258 problems identified (84.10%): 165 matched those assigned by experts (76%; Krippendorff’s α = 0.75 [0.68; 0.82], Table 4). End-users more often rated problems as “light” or "major,” whereas usability experts more often rated problems as “minor.”

Thirty-two of the 258 problems (12%) required clinical expertise to detect (Appendix E). In one instance, a pharmacist using the medication review module could not determine how to make a drug substitution, and in another, a physician did not receive an error message after entering the same treatment order twice. We include a complete list of issues in Appendix E. We classified these problems into 5 novel categories: patient identifiers (n=4), information availability and visibility (n=12), EHR configurations (n=9), access to EHR functions by users’ roles (n=2), and work and cognitive load problems (n=5).

There were instances when patient identifiers were not visible while performing tasks, increasing the risk of wrong patient selection errors. End-users encountered screens missing critical information, including orders, decision support, care plan actions, and medication changes. There were readability issues associated with typography and inappropriate “hard stops” (e.g., health insurance information queries blocking access to other functions). We found a mismatch between user EHR permissions and real-world scope of practice (e.g., nurses were granted access to prescribe when not permitted in clinical practice). Finally, the EHRs tended to increase workload and cognitive load. For example, some features increased the number of actions per task, failed to provide the user with feedback, or made selections confusing (e.g., a mismatch between medication type and dosing units).

Initial selection or transition to a new EHR can have seismic consequences on health system outcomes, patient safety, and clinician well-being.^[40] Choosing one that does not meet organizational needs can destabilize the work system and undermine patient safety. To our knowledge, this is the first study to evaluate the feasibility of a usability walkthrough involving end-users in EHR procurement. After a short training session, clinicians could identify, and risk stratify real EHR usability problems. The procurement team used our results to guide selection decisions and screen candidate EHRs for the third phase of the evaluation.

Our findings match studies of usability testing showing that testing products with as few as 3–5 users can identify up to 85% of interface problems affecting similar users.^[41] In our study, end-users within the same profile did not detect all the same problems. It is, therefore, crucial to recruit at least 3–5 evaluators per profile.^[42] While the assignment of criteria and scores by end-users and usability experts were consistent, end-users sometimes struggled to disambiguate “guidance,” “workload,” and “compatibility” criteria.^[43] Clinicians also tended to assign higher severity scores compared to usability experts. We hypothesize that the consensus method our experts used when assigning severity levels had a moderating effect on scores. Overall, our findings suggest that the usability walkthrough method—and the inclusion of end-users—is valid for identifying usability issues, is generalizable across settings, and may be extensible to other technologies.

End-users identified 32 usability problems that required clinical expertise to detect. This is consistent with studies showing that end-users with domain expertise identify problems that usability specialists may overlook.^[44] Some of the problems identified by clinicians could have had severe patient safety consequences. Patient identifier issues are known to increase the risk of wrong-patient prescribing^[45], missing patient data can affect medical decision-making, and inappropriate “hard stops” can delay patient care.^[8,46] Therefore, our findings reinforce the axiom that end-users must be engaged throughout the HIT lifecycle—including the procurement process.

Participants said they would use the walkthrough and scoring system for similar projects despite the learning curve. The information helped them objectively evaluate features and compare products systematically and argue preferences and advocate for their constituency. Data collected without vendor interference may generate more reliable information and more safety concerns than traditional product demonstrations, enabling business leaders to base decisions on technology elements or objective measures of human-computer interaction.^[31]

While usability walkthrough methods do not require extensive training, there are opportunities to improve orientation and testing activities.^[47] Sending the CLIPS and sample patient data to the vendors could have given them an unfair advantage and influenced how they configured their EHRs. In practice, however, the vendors were unprepared for testing. We could not complete several test scripts because vendors did not include the minimum data required for each simulation. This may have biased our selection decisions. Before implementing the usability walkthrough, usability experts should complete a “cross-check” of the EHR setup. We hypothesize that France’s lack of incentives for usability work and evaluation could explain vendors’ lack of awareness surrounding prerequisites for usability evaluation (e.g., missing mock patient data in a test patient record). Usability requirements included in the ONC certification program and other federal research funding programs may have incentivized vendors to collaborate more effectively with US healthcare organizations and create better test environments.^[48]

Readers should interpret the findings from our case study with caution. Our results are based on the behaviors and observations of a small sample of users recruited at a French hospital during an EHR procurement. The findings could differ across users, settings, products, or time points. Nevertheless, this case study shows how to gather usability data and stakeholder sentiments at the pace of healthcare operations.

Usability research adds up-front costs for the customer.^[49] The main costs include time spent creating test scenarios, recruiting participants, conducting usability evaluation, and retaining usability experts to proctor sessions, facilitate debriefs, and analyze the results.^[50] Executives must be convinced that the method generates a return on investment if they are to budget evaluation resources during procurement. The cost of testing must be weighed against the downstream costs of a poorly designed EHR related to training, workflow, efficiency losses, clinician burnout, staff disenfranchisement, patient endangerment, and legal actions.^[19]