Healthcare information systems: the cognitive challenge

In large part, the responsibility for the design, development, implementation and modification of healthcare information systems is left to software engineers and information technologists whose appreciation of the scope and complexity of healthcare work-as-done is necessarily limited [4, 10]. Experienced administrators and clinicians who select from commercially available systems typically have little to no expertise in device evaluation [13]. Opportunities for healthcare professionals to influence the design are fragmented at best and typically amount to little more than an incomplete list of functional requirements as developed by a small, non-representative, albeit knowledgeable group of healthcare specialists [10]. Nothing about this process approximates a design strategy that could lead to a coherent, robust and effective information system to support the diverse and complex demands of healthcare work.

Although healthcare information systems are focused on the cognitive dimensions of the work [10], they reveal no sensitivity to an insight from the field of situated cognition; that workers converge naturally on robust and powerful ways of doing work that differ markedly from the formal strategies that emerge from the technocratic world view [14, 15]. Nor do they reveal any sensitivity to the insight from Naturalistic Decision Making that experts in the field, who are making critical decisions under high workload and time pressure, do not follow a rational strategy of options analysis but rather, recognize and act spontaneously, choosing a different strategy only if the first proves to be unsatisfactory [16].

Absent sensitivity to these insights, healthcare information systems will, at best, fail to support the informal but powerful strategies used in patient care. At worst, healthcare information systems will disrupt and block these strategies, thereby inducing new and unanticipated systems errors and forcing those involved in patient care into work patterns and work arounds that are fragile, error inducing, and labor intensive.

There has been a recent call for healthcare to embrace the principles of high-reliability organizing [17, 18]. Despite undertaking complex and risky work, high-reliability organizations achieve exemplary levels of safety [19]. They do so in part by remaining mindful of the subtle and complex details of operational work [20, 21]. In that respect, procedural constraints embedded in technology should not interfere with operational work and institutional demands should not be prioritized over operational demands [20]. Technology is a crucial element of today’s high-reliability organizations [19], but those who design computerized information systems for healthcare impose constraints on how work is done with meager understanding of the operational work at anything more than a superficial level and with little sensitivity to its operational demands. Unless that changes, the push for high-reliability organizing in healthcare will never be completely satisfied.

User-Centered Design is offered by many as an intervention that will lead to the development of systems that will provide better clinical support [22]. It is questionable, however, whether a well-developed User-Centered Design process involves substantive design activity, where the term design refers specifically to those activities used to formulate the design solution as distinct from activities of analysis and assessment. Ellsworth, Dziadzko, O’Horo, et al. [23], although critical of the tendency within User-Centered Design to enter the development cycle late, nevertheless view User-Centered Design as directed at assessment of already-designed systems.

A systematic and comprehensive process of design proceeds through the stages of defining the problem, analyzing the work, generating solutions, and assessing prototypes and fielded systems; a nonlinear process that involves considerable iteration over adjacent and nonadjacent stages. LeRouge and Wickramasinghe [24] identified six stages for User-Centered Design (planning and feasibility, requirements, design, implementation, test and measure, and post-release) that correspond approximately to this process. Of fourteen activities identified for the design stage, only two suggest any form of design activity, the remainder being either analytic or assessment activities. Possibly, because of healthcare’s reliance on this limited view of design, clumsy and labor-intensive features such as data entry windows, drop-down menus, lists with check boxes, and alarms dominate as design features.

Decision-Centered Design [25] emerged from research into Naturalistic Decision Making, the study of how people make decisions under time pressure and uncertainty [26]. The focus in Naturalistic Decision Making is on operational work that has meaningful consequences as undertaken by experienced (often expert) practitioners. Notably, decisions made in this context may have ambiguous or poorly-defined goals. Naturalistic deciding came to be viewed as a macro-cognitive process and the insights generated through its study prompted an extension of the naturalistic method of investigation to other macro-cognitive processes [26].

According to Crandall et al. [27], the term macrocognition refers to the collection of cognitive processes and functions that characterize how people think in natural settings. The designation as macro signifies that these cognitive processes relate directly to work goals. Cognitive processes such as situation assessment, diagnosing, deciding, planning, communicating, managing, directing, and collaborating are viewed as macro-cognitive versus, for example, micro-cognitive processes such as noticing, managing attention, accessing information, or assessing options. In this conceptual scheme, the micro-cognitive processes support the macro-cognitive processes when deliberately designed to do so.

Decision-Centered Design progresses through five phases; preparation (development of domain understanding for the research group), knowledge elicitation (identification of essential work-related cognition), analysis (isolation of leverage points for supporting work-related cognition), design (development of a design concept), and evaluation (impact estimate of the proposed design). It relies on context specific, incident-based narratives to isolate leverage points for supporting the macrocognition involved in challenging situations [25]. Decision-Centered Design identifies the key cognitive challenges and key elements of expertise involved in cognitive activity as a basis for generating design ideas that can support challenging cognitive work [28]. Design solutions may incorporate one or more of technological innovations, work process enhancements, or cognitive skills training [29].

The creative design work undertaken within the framework of Decision-Centered Design is informed by views that align with Rasmussen’s problem-solving theory of cognitive modes [30]. A cognitive mode is a style of cognitive processing used to undertake cognitive work. Although Rasmussen does not refer to macrocognition as such [30, 31], the types of cognitive processes he refers to align with those identified as macro-cognitive by those who promote Decision-Centered Design [25‐27].

Rasmussen offers three modes of cognition; skill-based, rule-based and knowledge-based [30, 31]. The skill-based mode has no conscious processing between perception and action, the rule-based mode is guided by sets of procedural instructions that specify sequences of actions, and the knowledge-based mode is grounded in conscious and explicit reasoning. Identification of the modes used in any cognitive work guides the design of supports for that work, although a cognitive modes analysis needs to identify not only modes in use but also the information that supports the work and the type of action to be taken (Table 1).

Cognitive work such as diagnosis might employ any one of the cognitive modes on their own or two or three in combination. A clinician who recognizes sepsis at a glance is working in the skill-based mode. One who consults a checklist is working in the rule-based mode. One who notices the signs but must consult a text book to resolve what they mean is working in the knowledge-based mode. It is also of value to consider why different workers use different modes for the same task. For this sepsis illustration, an experienced practitioner may prefer the skill-based mode, an inexperienced practitioner may prefer the rule-based mode, and a student may prefer the knowledge-based mode. A cognitive support must conform to the cognitive mode or modes preferred by those who will use the system. Watson and Sanderson [32], in their development of sonic displays for anesthesiology, have observed that the cognitive modes supported by the monitoring technology do not align well with those preferred by anesthesiologists.

Several of the papers we cite in our background review note that clinicians often express concern about the impact information technology has on their workflow (the series of activities necessary to complete a task). As will become evident in our subsequent discussion, clinicians establish workflow patterns in part to support macro-cognitive processing. For example, a clinician who is concerned with maintaining their situation awareness within a clinical setting is likely to develop a workflow that will help them access the information critical for sensemaking in the desired sequence and at the right time.

An automated scheduling system was developed for the U.S. military to relieve healthcare staff of the task of scheduling patients for evacuation from first-point-of-care facilities [34]. For large-scale problems, the new system produced better schedules with less effort. However, the scheduling problem was dynamic in that staff could be confronted with an unscheduled evacuation request such as immediate transport of a seriously ill patient who required emergency medical treatment at a specified facility [35]. With diversion of an aircraft and crew to fill this urgent requirement, evacuation schedules for other patients could be disrupted so that the schedule would have to be adjusted.

Although common, schedule adjustment in response to dynamically unfolding needs had always been a challenge. However, in the process of manual scheduling, staff had implicitly developed an appreciation of potential resource options and conflicts, which they could use to adjust a schedule as consistent with new demands. In macro-cognitive terms, they had developed a useful level of situation awareness relating to available resources and potential conflicts via an implicit process of sensemaking. However, constraints imposed by the new automated scheduling system blocked staff from building that appreciation of potential options and conflicts so that staff were then ill-prepared to adjust schedules when needed.

Resource scheduling is a ubiquitous challenge in modern hospitals. For example, it can often be difficult in a large hospital to satisfy demands for in intensive care beds [36]. This is a problem that seems ideal for computerized support, but there is an ever-present concern that those who develop such a system will ignore many of the subtle cognitive processes that are critical to a satisfactory outcome.

A new, highly integrated, microprocessor-based physiological monitoring system for cardiac anesthesia was introduced into a cardiothoracic surgery unit to replace the functions of four single-sensor devices [37]. By centralizing the sensor data and the patient-monitoring functions in a single computer-based system, designers provided anesthesiologists with options for reorganizing windows on the screen and for viewing different representations of the same information. The most obvious interface difference from the previous assembly of discrete devices was the multi-layer menu structure that was activated via a touch screen.

Cardiac surgical patients are susceptible to rapid and profound hemodynamic changes, many of which can be life-threatening. However, the new system’s default numeric displays limited an anesthesiologist’s ability to assess the magnitude of rapid changes in blood pressure. This became an issue when the surgeon lifted the heart to feel the coronary blood vessels, an action that could cause blood pressure to fall rapidly. During such an event, the surgeon depends on the anesthesiologist to announce the correct blood pressures. These could be inferred readily from the default waveform representation of the old system, but the default numeric configuration of the new system encouraged a direct reading of numbers that changed too slowly to track blood pressure accurately. Although anesthesiologists learned to compensate by extrapolating the digital values, inexperienced residents sometimes failed to do so, which resulted in complaints from surgeons.

After considerable thought and experimentation, anesthesiologists developed a fixed-scale analog window that showed all blood pressures. Although it served the need when visible, this new window configuration had to be set up with a complex series of steps at the beginning of each case and, even then, was not entirely stable. An automatic window-management function could hide this new blood-pressures window as the anesthesiologist performed other tasks. That problem was largely resolved when an anesthesiologist discovered that the preferred screen configuration could be maintained by reserving screen space with modules that contained no useful information. Once this solution was known, the necessary window management could be completed during the low-workload period of system initialization.

Nevertheless, window management continued to be a problem when the anesthesiologist needed to measure cardiac output. Although cardiac output can be measured in 10 to 30 s, the result can be unreliable, and anesthesiologists often measure cardiac output two or three times in rapid succession to improve the estimate. With the new computer system, cardiac output was viewed on a special window brought to the screen by activating a screen label, but activation of this window had the side effect of removing the blood-pressures window, thereby degrading practitioner ability to detect rapid changes in blood pressure. This did not occur in the old system because the discrete devices displayed the data in parallel. Problematically, the time that measurement of cardiac output was most frequent coincided with the time that rapid changes in blood pressure were of most concern.

One consequence of providing multiple functions in a single device is that the control of these functions becomes more complicated. For example, a blood pressure channel was reset on the old system by pressing a physical switch on the front of a panel. With the new system, that channel reset required a series of screen activations. The most frequently used menu function of the computer system was measurement of cardiac output, a process requiring at least three menu activations. For the old, discrete device, that same function was activated with a single press of a mechanical button. Furthermore, errors were common. For example, an unintended menu activation could switch the system to a seldom-used region of the menu space. To recover, the anesthesiologist would typically escape back to the highest-level of the menu and then navigate through the menu tree back to the desired location. Although workable, this sort of disruption was clearly undesirable.

Evident in this illustration is that much of an anesthesiologist’s workflow is organized to support sensemaking. Workflows established for that purpose in the new system, which revolved largely around window management, contained many more steps and were more fragile than in the old system. Sensemaking was disrupted when displays either obscured information or provided it in formats that could not be interpreted at a glance. In addition, there was a flow-through to the macro-cognitive process of collaboration between anesthesiologist and surgeon, which was jeopardized when the anesthesiologist found it difficult to transform display readings as required by the surgeon.

The problems experienced in use of the systems described in our first two case studies [34, 37] resulted primarily from the failure to adopt a cognitively-relevant systems approach in design and implementation. A techno-centric mindset took the benefits of computerization for granted, directing attention away from the cognitive demands of healthcare work. Obvious functionality was assigned by default and subtle but critical functionality ignored.

More generally, these design projects appear to have been dominated by a technological imperative with no apparent concern for the cognitive challenges posed by the work.

These were failures of neglect, ones that could have been avoided with the right sort of analysis and design expertise. The need to design computerized systems so that they do not disrupt cognitive work and do not add onerous cognitive tasks may be challenging, but the design efforts behind the two systems of our case studies show no evidence that this challenge was even recognized.

The analysis of cognitive work explores the way workers pursue work goals and resolve cognitive challenges of that work, often in novel, subtle, and creative ways, and it seeks to uncover the strategies and tactics employed by experts as they engage in non-routine or challenging incidents [28]. The analysis of cognitive work requires structured methods because, on the one hand, healthcare professionals have expertise in their own specialty, but they cannot always articulate the subtle aspects of their expertise as it applies to their work. Nor can they frame even the knowledge they can articulate in terms that software engineers and information technology specialists can translate into design specifications. Finally, even knowledgeable healthcare professionals have only limited understanding of healthcare complexities beyond the domain of their own expertise. The design of a healthcare support system requires considerable specificity but inevitably, healthcare professionals will be able to make only general recommendations outside their own specialty.

On the other hand, software engineers and information technologists who build healthcare systems have only a limited appreciation of the demands and complexities of healthcare work. They are possibly induced to believe they know enough about healthcare because they do interact with the healthcare system as patients. While an understanding of work-as-imagined is useful, cognitive design requires a deeper analysis of work-as-done. If information technologists think at all about cognition, they will more likely be guided by common fallacies [38] than by scientifically sound principles.

Our first two case studies [34, 37] illustrate how an in-depth, cognitively oriented work analysis can uncover the cognitive complexity and cognitive subtlety of the work undertaken by healthcare practitioners, and they illustrate the character of the cognitive issues that designers of information technology support systems need to address. However, neither of those studies proceeded to the step of designing a solution for the cognitive issues they identified.

Tufte [39] observes that to be effective as decision support tools, information displays need to reflect the cognitive structure of the problem. Among his numerous illustrations of this principle, Tufte refers to an innovation by Dr. John Snow who identified the source of a cholera epidemic in London in 1854 by visually mapping cases against location to reveal that cases were clustered around one fresh water pump. Snow brought the epidemic under control by removing the handle from that pump. Notably, Snow’s cognitively-oriented solution to this problem was technologically simple, low-cost and remarkably effective. This principle, that displays should reflect the cognitive nature of the problem, can be restated as a call to represent the affordances [32, 40, 41] or the functionality [42] of the work and is one of the central design goals of cognitive engineering.

Powsner and Tufte [43] were guided by this cognitive principle in their redesign of the daily hospital record typically located at the end of a patient’s bed. Powsner and Tufte argued that the traditional form of the hospital record serves well as an archive but is not organized in a way that supports problem solving for patient care. They developed a detailed, one-page graphical representation that showed the time course of important physiological parameters, symptoms, and treatments in an array of small, high-resolution graphs with identical formats. Distinctive labeling allowed healthcare practitioners to converge readily on the information they were seeking on any occasion; for example, a single piece of information such as when the patient had commenced treatment with a specific drug, or a cluster of information such as the progression of a group of symptoms over days or weeks. The result was a summary picture of patient status that enabled the healthcare practitioner to examine relations between findings and treatments and to assess alternative diagnostic and management strategies. This work by Powsner and Tufte illustrates how it is possible to design a cognitive support tool that goes beyond the standard suite of design options of window hierarchies, drop down menus, text boxes and check lists or the common rule-based strategies that rely on written procedures, alarms and algorithms for clinical decision support.

Although managing and tracking colorectal cancer screening appears to be a simple problem readily resolved with rule-based reminders, screening rates remain well short of the 80% target established by The National Colorectal Cancer Roundtable [44]. Militello, et al. [45] developed a colorectal Screening and Surveillance App that was designed to work with the US Veterans Health Administration’s Computerized Patient Record System. For their development, they used Decision-Centered Design to identify the challenging decisions faced by clinicians in managing and tracking colorectal cancer screening and then design a cognitive support system for that work.

The development progressed through four design iterations. Each design iteration was different but, as required by the five-phase Decision-Centered Design strategy, each involved reviews of documents, discussions with or observations of healthcare practitioners, identification of leverage points, development of a design concept, and evaluation. This cognitively-focused strategy identified three important macro-cognitive processes that were troublesome within the current system and the micro-cognitive processes that posed challenges to healthcare professionals attempting to execute those macro-cognitive processes.

Sensemaking was troublesome because it was difficult to locate and integrate the information needed to construct a useful narrative about the patient. Details of a patient’s colorectal cancer screening history were stored in disparate places, again demonstrating how electronic information systems can be so poorly configured that they impede rather than facilitate the work.

Problem detection was troublesome because the electronic health record did not help healthcare professionals notice anomalies that might require non-routine action. Sometimes, the electronic record would indicate erroneously that the patient was due for screening. Because the clinical reminder provided no rationale for the recommendation, the primary-care provider would typically order the test, which would then be rejected by the testing clinic for no reason apparent to the primary-care provider. Nor did the electronic record clearly show what happened after a test was ordered. Primary-care providers found it difficult to ascertain whether support staff and patients followed up as needed to complete the order.

A third macro-cognitive challenge involved collaboration between primary-care providers and patients. Screening rates are affected by patient attitudes and it is incumbent on primary-care providers to engage effectively with patients to help them reach an informed decision. Patients will typically proceed with screening only if they are aware that it is beneficial and accessible. Providers reported that patient education in relation to colorectal cancer screening is time consuming; a challenge that is exacerbated when the primary-care provider is pressed for time and the patient has limited communication skills.

Militello, et al. [45] configured their colorectal Screening and Surveillance App to support these three macro-cognitive processes. The App encouraged a mix of skill-, rule-, and knowledge-based processing by use of an information representation that helped the primary-care provider navigate skillfully to key information that would support knowledge-based processing in service of complex macro-cognitive processes such as sensemaking, problem detection, and collaborative decision making. Rule-based processing was supported by a salient display of reminders.

Militello, et al. [45] also developed a one-page educational brochure designed to help a primary-care provider discuss screening options for, and common misconceptions about colorectal cancer with their patients. Although the Center for Disease Control already had a two-page brochure on this topic, this new one-page brochure was designed to be easier to view on a screen, easier to print, and less daunting to read [46].

The evaluation phase in the fourth design iteration showed that, given the support of the prototype Colorectal Screening and Surveillance App, primary-care providers answered questions about patients more accurately and found relevant information more quickly compared to those using only the Computerized Patient Record System [45]. Primary-care providers also reported reduced mental effort, assessed subjectively on a nine-point scale from extreme to none [47], and rated the Screening and Surveillance App positively for usability [45]. A separate evaluation showed that the one-page educational brochure improved knowledge of and openness to colorectal cancer screening to a degree equivalent to that achieved with the more cumbersome and detailed two-page brochure from the Center for Disease Control [46].

Klein and Lippa [48] studied the challenges posed by patient self-management of Type II diabetes. They interviewed diabetes patients to explore how well they understood their disorder and its management demands. The interviews identified the explicit knowledge, macro-cognitive skills, and mental models used to structure glucose self-management activities and decisions. To supplement the results of their interviews, Klein and Lippa reviewed relevant documents and websites, surveyed educational interventions, and observed diabetes management training classes. They also reviewed discussions in an American Diabetes Association chat room. By use of these methods, they built up a comprehensive picture of the macrocognition that impacted the effectiveness of patient self-management for Type II diabetes.

Their analyses of this information indicated that the explicit knowledge provided to patients from many sources could be useful, but the complexity of diabetes self-management could be overwhelming. Many patients found it difficult to deal with changes in routine that accompany events such as illness, stress, and travel. Even people with considerable explicit knowledge encountered challenges because that knowledge was not linked effectively to the situational constraints imposed by non-routine events. For example, a diet that worked well could be disrupted by the social expectations that accompany holidays or other communal interactions.

From analysis of their data, Klein and Lippa [48] concluded that education for diabetes self-management emphasizes rules and procedures that are overly complex and that do not respond well to the dynamic challenges that patients face in managing diabetes. They proposed that glucose-level management is analogous to the regulation of a complex, dynamic system (see Table 2 for the distinction between rule-based and dynamic control). Although, Klein and Lippa [48] acknowledge the value of rules and procedures, they argue that no rule-set can be adequate and that patients need an instructional program that can sensitize them to the situated knowledge associated with non-routine events, and can help them build skill with the macro-cognitive processes that support dynamic control [48]. Subsequently, they established that patients who relied on dynamic control (detecting vital information, responding to feedback, anticipating perceived trends) rather than rules are better able to maintain healthy blood glucose levels [49].