Results
Introduction
We interviewed 82 subjects representing clinical, technical, and administrative disciplines. Table
2 indicates the number and roles of those interviewed and observed. We observed 105 clinicians for a total of 194 person-hours and conducted observations in most areas of the hospitals. We visited and observed clinicians working in 41 different clinics. Data analysis, which took place during 90 team meetings, revealed ten general themes.
Table 2
Details about interviews and observations at each site
Interviews and field surveys
| | | | | | | | |
Roles of Subjects
| | | | | | | | |
Admin-Managerial
|
5
|
5
|
2
|
1
|
3
|
1
|
3
|
20
|
Bridger-Clinical*
|
8
|
3
|
4
|
6
|
6
|
3
|
1
|
31
|
Clinical User
|
13
|
12
|
6
|
4
|
0
|
3
|
0
|
38
|
Technical
|
1
|
2
|
2
|
1
|
2
|
3
|
5
|
16
|
Site total
|
27
|
22
|
14
|
12
|
11
|
10
|
9
|
105
|
Observations
| | | | | | | | |
Hours observing
|
36
|
26
|
37
|
20
|
25
|
33
|
17
|
194
|
Individuals
observed
|
10
|
12
|
17
|
16
|
17
|
27
|
6
|
105
|
Number of clinics
|
NA
|
NA
|
9
|
6
|
5
|
9
|
12
|
41
|
The Multiple Perspectives model was used to help select our subjects, frame our questions and observations, and remain cognizant of relationships and dependencies among our four components of CDS: users, content, technology, and governance. It forced us as researchers to use different lenses and to gather data from users as they viewed components through the three different lenses: the technical, organizational, and personal, which also overlap and blend at times. Although the themes and patterns arose directly from our data, each is more closely aligned with one system component than others, so the four ovals depicting the CDS components within the dotted oval in the Figure
1 model will serve as an organizing scheme for the following discussion. In addition, because several of the themes that arose directly from the data did not fit into one of the four components of the CDS "system" outlined in the framework, we conducted further analysis which has resulted in our proposing a modification to the framework and a new theoretical construct.
CDS Fieldwork Themes
Please see Table
3 for a listing of themes, subthemes, and recommended practices. Recommended practices will be described in the discussion section. Table
4 provides illustrative quotes for each of the themes.
Table 3
Themes, subthemes, and recommended practices
User Component-- Theme 1: Workflow
|
| | Assess workflow early |
| | Start with simple inline CDS |
| | Plan to customize products to fit workflow and vice versa |
Content Component-- Theme 2: Knowledge Management
|
| Knowledge creation | Plan early and allocate sufficient resources |
| Content library management | Catalog and monitor CDS from the beginning |
Technology Component-- Theme 3: Data as a foundation
|
| Having enough information about the patient | Develop interfaces to external systems |
| Quality of the data | Participate in HIE efforts |
| Sharing the data | Educate clinicians about reason and importance of good data |
| Varied uses of data | Promote standards |
Technology Component-- Theme 4: User computer interaction
|
| Customization | Solicit feedback |
| Usefulness | Test new CDS on users |
Technology Component-- Theme 5: Measurement and metrics
|
| Administrative needs | Identify reporting goals |
| Monitoring CDS | Plan measures early |
| | Refine CDS based on measures |
Governance Component-- Theme 6: Governance
|
| Environmental factors | Identify existing structures to repurpose as many as possible |
| Setting priorities and resource management | Establish decision making structures early |
| Governance structure | Plan to reassess structures regularly |
| Relations with vendors | Involve clinicians continuously |
New Construct, Translational Interaction--Theme 7: Translation for collaboration
|
| Collaboration for development | Promote collaboration |
| Translation for vendor collaboration | Speak language of collaborators |
| Translation between users and IT | Spend time with users as they work |
| Collaboration among clinical orgs. | Create a culture of interaction |
New Construct, Translational Interaction--Theme 8: The meaning of CDS
|
| Multiple meanings | Understand the user view |
| Informatics philosophy | View CDS broadly |
New Construct, Translational Interaction--Theme 9: Roles of special essential people
|
| As previously defined | Create and formalize roles |
| Newly found roles | Educate the workforce |
New Construct, Translational Interaction--Theme 10: Communication, training, support
|
| Communication, training, support | Involve users, give resources, over-communicate |
Table 4
Representative quotes from the fieldwork
"People practice in very different ways. Some physicians look at the screen once before they see the patients, and then they don't really touch the computer [again] until they have to write prescriptions. So, the opportunities to interact with the computer and receive decision support can be limited for those practitioners." "Now they need to turn the alerts back on, condition by condition. They plan on customizing the alerts before they turn them back on." "She uses the point and click charting templates to complete her review of systems [and] history and physical very quickly." | "We moved to the EMR because we felt it would standardize or help quality." "I think the underlying drive, perhaps not surprisingly, it's the recognition that we need to distinguish ourselves as an organization from amongst the competitors." "We sort of have a very tight knit connection with [our vendor]. So, I think everyone sort of collaborates with them and cross-communicates with them on practically everything." |
Theme 2: Knowledge management
"We assess new drugs as they are introduced as possible candidates for CDS." |
Theme 7: Translation for collaboration
"She is a 'development analyst' and the team leader for similar analysts. They write specs, test, and modify and they serve as liaisons between the users and IT. There are other analysts who are implementation and support analysts." "A lot of her job and a lot of my job is working with [the vendor] to make sure things are running correctly." |
Theme 3: Data as a foundation for CDS
"The patient we were looking at had an LDL reminder but the patient had actually had the LDL done. The reminder didn't work correctly since he didn't have lab results in the EMR (so it thought the test hadn't been done when it had been)." |
Theme 8: The meaning of CDS
"Sometimes the best decision support is not to give them [physicians] the decision [and to design a nursing protocol instead]." |
Theme 4: User computer interaction
"We have to build custom-like orders, we have to build for the practice, medications, problems, custom lists for each practice and when they log into the system, it automatically defaults to their custom list." "It flags it in red, so it's a visual cue to the physicians that it's a little bit outside of the range and if you're ordering something with a narrow therapeutic index, you need to be aware." |
Theme 9: Roles of special essential people
"We do have people who are practicing clinicians who are helping create the rules. You definitely need someone who knows the technical side of the equation." "I know how challenging it is for clinicians to take time to address these important issues. That has been compensated." |
Theme 5: Measurement and metrics
"Most physicians will use the reminders because they get report cards on their completion rates. . . if you go down into the clinics you'll see graphs that compare clinics to one another as a form of competition." "We are now better able to track the timeliness and the labor required to meet those maintenance obligations." |
Theme 10: Communication, training, support
"There are always new features that come up and I think we still completely suck at letting people know about these new features." "We actually make it very easy [for clinicians] to write patients a letter describing their test results in a patient-friendly format." "We have to understand what the physician is going to be doing. Are they going to be dictating, typing their notes? So, really trying to gear the training around workflow." |
Component One: Users as a component of the CDS "system"
The end users of CDS are those whose workflow is most affected by it. Users are constantly adjusting their work because of the system and the systems are ideally constantly changing to better facilitate users' work.
Theme 1: Workflow
We were consistently told that any system should fit the workflow of its users as closely as possible. The locally developed systems were designed to fit into the work done at a particular site, but since users differ in their work habits, even these systems needed some customization to match individual workflows. Concomitantly, users must generally adapt their workflows to better fit the system. Those using commercial systems are continuously individualizing or customizing aspects of the system to better fit their ways of doing things, or adapting to the system's requirements. There are limits to what buyers of commercial systems are allowed to customize, however, which is often why workflow must be adapted.
Reengineering the workflow
The sites using commercial systems had all conducted workflow analyses in each clinic prior to implementation. The sites with locally developed systems seem to be in a perpetual state of workflow engineering. A researcher wrote in fieldnotes: "I speak to the workflow fellow who calls himself an EMR Workflow Engineer. He observes how the staff uses the computer system and helps them to trouble shoot workflow problems. His team observes the lean manufacturing/production philosophy. He uses time/motion studies and asks the practice about what needs they have." At one site we were told "So now what we're doing is we're sort of going back to all these sites and saying okay, we're going to start from scratch with you. We'll go over all of your workflows and all the ways that you document and make your decisions and we'll show you how to do this in the EMR now."
In-line applications and CDS that fit the workflow
By in-line applications we mean computer-enabled help that seamlessly fits the workflow, that does not interrupt the clinician, and that is nearly invisible. Applications are in-line if they provide needed information at the appropriate time in the encounter. Templates are an excellent example of an in-line application providing decision support. These were especially useful in the ambulatory setting when clinicians used the system during the patient encounter. One researcher's fieldnotes said: "She uses the point and click charting templates to complete her review of systems [and] history and physical very quickly." Another noted: "[The provider] uses templates and occasionally brings notes forward. I asked him whether he did this because it made it faster or because it helped him remember. He said mainly because it made it faster, occasionally for remembering." Some users were critical of the documentation generated by use of the templates, so they entered free text into the template instead of or in addition to filling in the fields. Some clinicians would not use a computer in the exam room because they thought it would hamper physician-patient interactions. Some, however, were observed to be remarkably facile, brought the patient into the encounter skilfully, and enjoyed using the templates. Often, these were the clinicians who had taken the time to modify templates to their liking.
Most clinicians who had e-prescribing available praised its ability to help them. One researcher said in fieldnotes: "If he prescribes a med, he does it in the room on the computer. It [then] prints out [so he can] hand it to the patient or to fax, or may fax directly. The app is populated by a list of pharmacies. The patient's usual choice is there as the default value."
Variability of workflow
We were told that the prime reason why workflow analysis is needed prior to implementation or on an ongoing basis is that each physician has developed his or her own way of doing things. One interviewee said: "People practice in very different ways. Some physicians look at the screen once before they see the patients, and then they don't really touch the computer [again] until they have to write prescriptions. So, the opportunities to interact with the computer and receive decision support can be limited for those practitioners." Others carry laptops or tablet computers with them at all times and have multiple opportunities to receive CDS.
Location of the encounter
We observed that CDS usefulness depends a great deal on where the physician opts to use the computer. Clinicians who use templates during the patient encounter receive timely, helpful, welcome, seamless decision support: "Her process is to use her laptop in the exam room, filling out the smart form [template] for her note. She further edits the smart form in her office." On the other hand, clinicians who waited to use the templates, often until after the patient had left, missed an opportunity to be reminded of important issues.
Temporal issues
Timing of the CDS presentation, especially alerts, is important to users. We heard complaints about alerts firing at the wrong time, both too early in the encounter and too late. Clinicians wanted them at "the point in time during the encounter where it's really going to be most helpful and most actionable." Time pressures had an impact as well. None of the outpatient sites we studied had many alerts aimed at physicians. We were told: "They're overwhelmed, they're too busy, they have too many demands on their time." An informatician noted about alerts: "We thought that it was a much bigger downside to frustrating people by constantly interrupting their workflow than missing the alerts." One site turned them all off and a representative told us: "Now they need to turn the alerts back on, condition by condition. They plan on customizing the alerts before they turn them back on; having the task force review the logic before they turn them back on; turning them on clinic by clinic."
Component Two: Content as a component of the CDS "system"
Content issues include development or purchase and management of CDS.
Theme 2: Knowledge Management
By knowledge management, we mean the entire process of developing and translating pieces of knowledge so that they are available in the system. Knowledge management also includes acquiring, tracking, evaluating, and maintaining knowledge, just as libraries gather, catalog, and maintain library collections.
Knowledge creation
The sites we studied that had locally developed systems also had locally developed CDS, which, because of the way it was developed over many years by innovative individuals, is hard to track. However, they are making progress in developing ways to monitor their CDS. Individuals at these sites continue to develop new CDS, which now tends to be more carefully managed. Close ties with the pharmacy and therapeutics committees and quality assurance staff members yield ideas about "new drugs as they are introduced as possible candidates for CDS." Although our study sites that used commercial systems did not develop CDS de novo, they followed many of the same processes when customizing content they obtained from others, including content vendors. All of our sites with commercial systems had informaticians in leadership roles and CDS analysts who could modify CDS content.
Content library management
We use this term because we see an analogy between traditional library functions such as acquisitions, cataloging, maintenance, provision of access, and "weeding" of materials and the functions that appear to be needed for CDS content management. The acquisition phase includes either development or modification of CDS, described above. Once an organization acquires a certain amount of CDS, it starts to lose track of what it has, so an inventory is wise if it has not been conducted from the beginning. Following the inventory, a means of cataloging, or indexing, is needed so that analysts can search to find out what exists. Model sites conduct cyclical reviews for curation and maintenance and have mechanisms for scanning the environment to keep up to date about new evidence. Organizations with commercial systems can take advantage of software offered by vendors to help manage this process. Unlike libraries, holders of CDS do not often share their locally developed or modified CDS. We asked interviewees about their willingness to share CDS. The reasons for not sharing included lack of a technical ability to do so and a hesitation to share without remuneration what they had spent time developing. There are also legal issues that inhibit sharing. On the other hand, there was interest in sharing among sites that have a particular vendor-based system and also when an organization wants new CDS. Organizations would like to be on the receiving end but not the giving end of the exchange.
Component Three: Technology as a component of the CDS "system"
Several of our themes relate to this component: data as a foundation for CDS; user computer interaction; and measurement and metrics.
Theme 3: Data as a foundation for CDS
Many types of CDS require that data about individual patients reside in the system. For example, before a reminder that a mammogram should be scheduled can be generated, the system needs to know the age and gender of the patient, when her last mammogram was performed, whether they have a mammogram already scheduled, and finally, if they are status post bilateral mastectomy or in a hospice program. We were told that if decision support is to be highly sensitive and patient-specific, then accurate, complete, structured information about the patient must already exist in the system.
None of our study sites has truly complete data about its patients because patients receive care from many different organizations. Even VA patients sometimes get care outside of the VA system. Some data, such as those in medication lists, are especially hard to keep accurate and up to date. Other data must come from sources such as laboratories and agreements as well as technical interoperability are needed if these data are to be shared. For example, one interviewee noted: "The patient we were looking at had an LDL reminder but the patient had actually had the LDL done. The reminder didn't work correctly since he didn't have lab results in the EMR (so it thought the test hadn't been done when it had been)." We often heard remarks such as: "We're working on getting university radiology, a radiology site to send us their results electronically, because right now they come over as paper and we have to scan them." Clinicians uniformly desire having all the right information, but not too much information, at the point of care.
Quality
Often our subjects worried about the accuracy of data that had to be entered manually. One said "Even my own partner doesn't really, you know, capture or do the data. I mean a lot of it is just getting the work done at that moment in time." A nurse confided that "at times the nurses will simply cut and paste medication profile information from [the system] into the medication reconciliation document without properly verifying all of the medications on the list."
Sharing Data
Data are often stored in separate silos, with laboratory and radiology information in separate systems that cannot share information with an EMR. The extent of the ability to share data within and across sites varies a good deal. One site, which included a group of outpatient clinics, shares nothing but demographic data between clinics. The VA shares nationally, Partners sites share some information such as allergy information, and the Wishard clinics using the Regenstrief system are part of a statewide network sharing some patient-specific clinical information.
Varied uses for these data
Administrators and informaticians told us they value data availability not only as a basis for patient specific CDS but also for quality measures reported after the fact. One informant said "It is frustrating that we have not been able to get any quality indicators out" because data were not being entered by all clinicians. Another use is for research purposes, and both accurate and complete data are needed. Others would like population-based data.
Theme 4: User computer interaction
We think of user computer interaction as ease of use of the system, including the equipment, the screen layout, the number of clicks needed to accomplish a task, the cognitive energy needed to figure out what to do next, and the speed of use. There are two major sub-themes that emerged: customization, which can take place on many different levels, and usefulness.
Customization
All of the systems we studied could be customized and these successful sites all devoted considerable staff time to this endeavor. Even sites with locally developed systems were constantly providing further customization: "We sat down with the [EMR] team and they had to change the user interface of how pediatricians would order medications, because now we're doing it through weight based dosing vs. flat dosing." An analyst at a site with an EMR that provides templates noted: "We can edit the forms and customize them for the practice. We have to build custom-like orders, we have to build for the practice, medications, problems, custom lists for each practice and when they log into the system, it automatically defaults to their custom list." Analysts also make changes to simplify use of the system. As one analyst noted: "I look at it and say no, this will never fly. Ten clicks to get here, forget it, we've got to simplify this."
Usefulness
During observations, we found that presentation of the CDS was of utmost importance. Where CDS was "in-line" with workflow, we often observed that simple presentation decisions could be extremely powerful. For example, the use of color draws attention to data without changing workflow: "It flags it in red, so it's a visual cue to the physicians that it's a little bit outside of the range and if you're ordering something with a narrow therapeutic index, you need to be aware." Actionability was likewise critical. If reminders are "actionable," meaning that the clinician can respond to the reminder without needing to access another part of the system, usually with one click, they minimize impact on workflow and tend to be used. At one site, most decision support is provided this way, and the positive outcome is that reminders tend to be voluntarily viewed and acted on. In addition, structured data are collected and reports generated about responses to reminders. Reliability is very much valued by clinicians. We were told that CDS cannot be useful if clinicians avoid the system because it is not available at times. Finally, correctness and applicability to the patient are important. There are times when the system simply is not correct. One clinician noted: "When you order inhalers, it often rejects the dose that it suggested you use!"
Theme 5: Measurement and metrics
We were told that patient-specific, accurate, and complete information that already exists in the system is needed to measure both the effect of clinical decision support and the use of it. Also, metrics need to be established so that the impact of the EMR can be measured over time. For example, it is useful to know how often alerts are being overridden and why.
Administrative needs; quality reporting
With increasing pressure to be accountable for quality, the sites we studied either already take advantage of measures that can be extracted from the system based on CDS interventions, or they are planning to conduct ongoing measurement once the system is fully implemented across all clinics. We were told at several sites that provide performance feedback to clinicians that such feedback is welcomed. One site with a "dashboard" provides direct feedback to clinicians. An interviewee noted: "So, it's at the clinician level, sort of their performance on key indicators compared to their peers and compared to some external benchmarks." Some sites provide incentives for meeting performance goals: "we are reporting on things that ultimately become these accountability metrics. . . people are either going to get bonuses if they do certain things and if they don't, they don't." At another site we were told: "most physicians will use the reminders because they get report cards on their completion rates. . . they are attended to as opposed to other systems in other organizations where there's not this tracking reporting type system. If you go down into the clinics you'll see graphs that compare clinics to one another as a form of competition."
Monitoring and control of CDS
Some of the study sites monitor how effective some CDS is: "How usable is our decision support such that for example we are now putting in routine efforts to track override rates." They might also monitor the effort put into maintaining CDS: "we are now better able track the timeliness and the labor required to meet those maintenance obligations." Sites that do not monitor CDS at present are planning to do it soon.
Component four: Governance as a component of the CDS "system"
All of our study sites had formal governance structures for managing CDS.
Theme 6: Governance
Governance includes formal and informal mechanisms for making decisions about the system and about CDS in particular; four subthemes emerged from our data.
Environmental factors/Motivation
We were told in interviews that while the ultimate motivator for implementation of CDS is the desire to improve patient care, there are other intermediary factors pressing for it. These include increasing attention to rewarding patient safety and healthcare quality by accrediting bodies, payors, and professional societies. As one quality assurance director noted: "We moved to the EMR because we felt it would standardize or help quality and would standardize our, some of our practice." Another motivator is competition in the health care sector. As one interviewee stated: "I think the underlying drive, perhaps not surprisingly, it's the recognition that we need to distinguish ourselves as an organization from amongst the competitors in terms of safety and quality."
Setting priorities and resource management
We were told that one of the most difficult aspects of CDS governance is the setting of priorities. With outside pressures to meet certain measures and internal pressures to decrease costs or improve specific local outcomes, organizations must decide where to put their energy. One of our sites has a committee that developed a list: "A top ten list of what we thought should be standardized across the enterprise." Each site has a somewhat different approach to setting priorities, but all have multidisciplinary committees that provide oversight and make ultimate decisions.
Governance structure
We found three aspects of governance structure related to CDS, which our study sites consider crucial: committees, process, and feedback to the governance system. Committees play a vital role in governance. The more mature CDS sites have several layers of committees, with higher-level decisions made by higher-level committees. These are generally multidisciplinary, with a mix of clinicians, administrators, and technology representatives. One was described by an interviewee: "It's called the EMR IT Advisory Group. The physicians, some of the IT staff, some of the clinical staff, and the analysts." All tend to gather task forces of clinical experts when needed. Each site has a process for discovering new evidence or environmental changes that impact CDS. One example of this is a process for learning about changes made by the Pharmacy and Therapeutics Committee. Finally, mechanisms for feedback to the governance system are imperative. Each site also has a process for reviewing requests from users. At one site, "the clinical content committee reviews requests that come up from the user base and they are funnelled to and from the [information system] management team about decision support."
Relationships with vendors
The sites with commercial systems must depend a great deal on decisions about CDS that are made by their EMR vendors. Therefore, they must be in close contact with the vendor. We heard at one of these sites: "We sort of have a very tight knit connection with [our vendor]. So, I think everyone sort of collaborates with them and cross-communicates with them on practically everything. We really can't do very much on our own here without [the vendor]." The EMR vendors generally purchase content for CDS from content vendors. Sites with locally developed EMRs often purchase directly from content vendors, especially for medication information.
Proposed New Construct, Translational Interaction
Several themes emerged from our analysis of the data that did not easily fit into the framework originally proposed, which included components for content, users, governance, and technology. Instead, the additional themes all included aspects of "translation," which we define as communicating meaning through language.
Theme 7: Translation for Collaboration
For groups to collaborate effectively, they must understand the cultures of the different involved groups. Culture implies a shared system of meaning and language. The different groups for which collaboration is necessary include: the developers and analysts, IT staff, clinic staff, the vendors, clinicians, and administration.
Collaboration for development
At sites that build new CDS, the development process involves a development analyst who facilitates discussions among clinical specialists, knowledge engineers, and programmers. A researcher explained in fieldnotes: "She is a 'development analyst' and the team leader for similar analysts. They write specs, test, and modify and they serve as liaisons between the users and IT. There are other analysts who are implementation and support analysts." The development analysts and knowledge engineers exist at the interface of the clinical and information technology worlds and are familiar with the vocabularies of both.
Translation for vendor collaboration
At the sites with commercial systems, analysts modify CDS content provided by vendors and to do so, they must often work with members of the vendor's staff. As one analyst stated it: "A lot of her job and a lot of my job is working with [the vendor] to make sure things are running correctly." Often, those within the purchasing organization feel that they are not sufficiently supported by vendors, although it is usually incumbent on them to purchase services in order to receive them.
Translation between users and IT
Both analysts and training and support staff translate or explain the clinical culture to information technology staff. One analyst noted: "Our local IT people said 'oh, that can't be done. . . then I realized it wasn't that it couldn't be done. The IT people we were working with didn't understand what the clinicians were asking to get done. I realized then that there needs to be some kind of an intermediary who understands the IT world and the clinical side."
Collaboration among clinical organizations
The cultures of outpatient clinics which house physicians in private practice and the cultures of the hospitals to which those physicians refer patients are different enough that information systems are impacted. The business models are different, of course, and some vendors do not have products for both or will not sell to both except under certain circumstances. In addition, there is sometimes competition for patients if the hospital performs outpatient procedures. We were told that collaboration between the organizations that purchase content and EMRs and the vendors is essential.
Theme 8: The Meaning of CDS
We asked each interviewee how he or she would define CDS, primarily because we wanted insight into different perspectives. We quickly learned that CDS generally means something quite different to the informatics experts than it does to the clinical users. It is important to note that meaning goes beyond definition: it is making sense of a phenomenon. Much of what we learned about these mental models our subjects hold came from observations and informal comments. Their view surprised us: often they did not know what "clinical decision support" is, so we had to explore the idea by asking how the computer helps them make clinical decisions. The users see CDS as an opportunity for the system to help them get through their day. They focus on the help and assistance the EMR can offer. Experts usually describe CDS in terms of sophisticated alerts or reminders. Interestingly, CDS implementers at each site seem to have a unique philosophy that guides their CDS efforts, a shared mental model or organizational meaning. We divide this theme into two sub-themes: the multiple meanings of CDS and different informatics philosophies of CDS.
Multiple meanings of CDS
Clinical decision support means different things to different disciplines and to individuals within those disciplines. One of our sites has a position called "Coordinator of Clinical Decision Support" which deals exclusively with administrative data in the form of reports about clinicians' actions and not at all with how clinicians make clinical decisions. We heard similar definitions echoed by quality assurance staff. When asked for a definition, informaticians usually offered a very broad definition such as "presenting information to somebody in a way that's going to help them to make decisions or take actions." However, those individuals often went on to describe alerts and reminders, most likely because these forms of CDS are most interesting to them. On the other hand, most of the practicing clinicians we observed thought of CDS as anything that could help them finish their work in a timely manner. Any information in either the clinical information system or the office practice system that assists the clinician's workflow constitutes CDS in their view. Some clinicians described talking with or e-mailing other clinicians or even reading another clinician's notes as decision support. They make little distinction between clinical information and other types of information such as demographic or scheduling information. To them, all of this information helps them take care of their patients.
Experts at several of our sites expressed philosophies that guided their organization's development of CDS. One of the study sites held a philosophy "we're not trying to tell the physicians what to do, we're trying to give them the information." Informants at another site used the terms "guardrails" and "helping the clinician to do the right thing." One informatics professional noted: "I've seen a lot of decision support done as forcing people down this path or that path, always has been the carrot or the stick. . . what is the grade of the ground? You can get that mule pulling that cart, are you whipping them, are you enticing them, but the truth is that if you just make it easier to go down one path. . . To me that's the ideal decision support is when the person doesn't even realize that it is happening." Elsewhere we heard "giving vaccinations when patients were in the hospital and the most effective way to do it was to give it to nursing and make it part of their protocol; and take it out of the decision tree of the doctor. Sometimes the best decision support is not to give them the decision." Administrators at one site clearly saw CDS as an "enabler of standardization."
Theme 9: Roles of Special, Essential People for CDS
In prior studies, we have identified and described typical "special essential people" roles for CPOE implementation and maintenance [
35]. These roles include administrative leaders, clinical leaders, champions, opinion leaders, and bridgers of different types who span the gap between the clinical world and the technology world and generally provide support and training. This study confirmed that these roles are critical for effective CDS as well. However, this study also identified several new and emerging roles directly related to CDS, which will become increasingly important. We arrived at this list through analysis of statements our informants made during interviews and through field observations.
Essential people as previously defined
We found the same types of essential people at these sites that we described after visiting five organizations for a prior study about CPOE success factors [
35]: champions, who are clinicians in the forefront of information technology; opinion leaders, who are clinicians well respected for their clinical expertise who are spokespeople for systems; administrative leaders, who are not clinicians, but who hold a vision of what CIS can do; clinical leaders, who are clinicians by background but hold administrative positions; and "bridgers," who are usually clinically trained but who have enough IT expertise so they can train and support users or serve as analysts who develop or modify systems. These designations are not mutually exclusive, since clinical champions may also be administrators, for example. In addition to these roles, we discovered in this CDS study a number of variations of the roles we previously identified. For example, two sites that have commercial systems have on-site analysts who actually work for the vendor and not for the hospital or practice. This arrangement has both advantages and disadvantages and seems to work best when the analysts have experience working for the organization they serve. One analyst of this type described how difficult it had been for outsiders hired as analysts because "They had to get used to the flow. . . they had to get used to that because this is a completely different environment for them. So it took them a while to kind of figure that out, whereas I already knew, that so that was an advantage for me."
Newly found essential roles for CDS
These include knowledge engineers, subject matter experts, outpatient clinic champions, pharmacy informaticians, and ambulatory clinic chief medical information officers.
Knowledge engineers and analysts
The sites we studied that have non-commercial systems were unique in that they each have knowledge engineers who are clinicians, usually physicians, who have developed and evaluated decision support through grant funding. These knowledge engineers help to develop the content for CDS and are skilled facilitators who seek to gain consensus from clinicians. They translate human readable content into a form that the system can use, so they are technically as well as clinically astute. Their role was well described by one informant: "We do have people who are practicing clinicians who are helping create the rules. You definitely need someone who knows the technical side of the equation." Analysts are generally clinically trained as well and they perform many of the same tasks of knowledge engineers, often modifying content available through commercial systems.
Clinical CDS Subject Matter Experts (SMEs)
Each organization has a cadre of clinicians who assist with development or modification of decision support. Organizations seem to have difficulty motivating SMEs as time goes on. By SME motivation, we mean that once these individuals are identified and they are taking their SME roles seriously, they need to be nurtured and continuously updated and motivated to continue being SMEs. These people are clinicians who are interested in information technology and the potential of computerized CDS. When systems are new, they seem naturally motivated by the challenge of implementation. However, as CDS is continuously rolled out, these SMEs grow weary. Some of the sites compensate these experts: "I know how challenging it is for clinicians to take time to address these important issues. That has been compensated and I think that's another whole, you know, another whole dimension."
Outpatient clinic champions, often non-clinical
In office practice settings, there is usually someone who serves as office champion. This go-to person is sometimes one of the clinicians who has an aptitude for computer systems, but more often it is one of the office administrative staff members who devotes part time to working with the system. Having such a point person who is both knowledgeable and personable seems to be important for success.
Pharmacists who are Pharmacy and Therapeutics (P and T) Committee connectors
Hospitals have Pharmacy and Therapeutics committees, which oversee medication use. Pharmacists who bridge the gap between these P and T committees and CDS developers are uniquely capable of assisting with medication related CDS development and maintenance. They play a critical role in communicating between the committee and the developers so that the developers are well informed about P and T priorities.
Ambulatory clinic CMIOs
Each of the two freestanding clinic organizations we studied hired physicians with informatics training to fill the role of a chief medical information or information systems officer. Each used a commercial system, so these individuals played a key role in expressing the clinic's needs to the vendor, in facilitating the work of the CDS analysts, and in communicating with users.
Theme 10: Communication, Training and Support
Communication, training, and support are critical success factors for any clinical information system, but there are unique issues when the focus is CDS. This theme includes communication and training about new CDS and CDS modifications as well as ongoing support efforts.
Communication about CDS takes many forms, and staff members at our study sites feel it is exceedingly difficult. Like training, communication is hard to do when busy clinicians are the target audience. One informatician said: "there are always new features that come up and I think we still completely suck at letting people know about new features." Types of communication include e-mail, on site meetings and presentations, "lunch and learns," use of feedback buttons, and personal contacts with CDS analysts or super users. Communication between clinicians and patients, we found, can be enhanced through CDS, both through the use of reminders about health maintenance schedules and other forms of communication. For example, one informatician noted: "We actually make it very easy [for clinicians] to write patients a letter describing their test results in a patient-friendly format."
Training, primarily conducted one-on-one at the sites we studied, needs to be ongoing, especially as more decision support is added, and organizations find it exceedingly difficult. It is hard to separate training from communication and support or to differentiate it from education or efforts to motivate clinicians. As one physician developer noted: "asking them [providers] to do something with decision support, it's just, you know, to really make that behavior change requires lining up more than the reminder. You've got to line up education and incentives and a whole bunch of things and generally we don't do that for too many [providers]."
Initial training needs to show the user how a particular CDS type might fit that user's individual workflow. One trainer noted: "We have to understand what the physician is going to be doing. Are they going to be dictating, are they going to be typing their notes?. . . So, really trying to gear the training around workflow." Ongoing training is especially necessary as new CDS is added. However, organizations were somewhat apologetic about their inability to do this well. We found that users rarely felt their knowledge about changes to CDS was up to date. One physician said "A lot of it you learn by trial and error," and a researcher's fieldnotes noted someone "had developed some work-arounds that seemed valuable, but that required him to do many inefficient actions within [the system]."
By support, we mean providing help to users at the time of need. Support related to CDS involves continuous feedback to and from users, generally by phone to a help desk or through e-mail. One clinician's quote is characteristic of what we heard at all of our study sites: "whenever we e-mail them [IT support] or have a problem with them or feel like things should come up differently or pop up differently or whatever, I mean, they're great."
Acknowledgements
We are grateful to the participants who allowed us to observe and interview them during our research, and to Drs. Marc Overhage, Eric Pifer, Greg Fraser, Frank Sonnenberg, and Jason Saleem for their assistance with site visits. We thank the 17 experts who took part in our fifth Menucha conference: John Dulcey, Tom Payne, Mike Shabot, Jim Carpenter, Victor Lee, Scott Evans, Brian Patty, Eric Poon, Chuck Tucinda, Julie McGowan, Paul Nichol, Brian Churchill, Ken Guappone, Joanie Kapusnik, Dick Gibson, Gil Kuperman, and Randy Miller.
This work was supported by AHRQ contract #HHSA290200810010, NLM Research Grant 563 R56-LM006942 and Training Grant 2-T15-LM007088. AHRQ and NLM had no role in the design or execution of this study, nor in the decision to publish.
Finally, we would like to dedicate this paper to POET team member Richard H Dykstra, MD, MS, who was instrumental in developing the new theoretical framework but who unfortunately passed away before the paper was accepted for publication.
Authors' contributions
JSA, DFS, AW, and CM designed and conducted the study, analyzed the data, and wrote the paper. KPG, RHD, JR, JC, AB and MS assisted in the design of the study, gathered and analyzed the data, and reviewed versions of the manuscript. AB also organized the data and contributed significantly to writing the paper. BM, with JSA, obtained funding for the study and reviewed and revised drafts of the manuscript. All authors except RHD, now deceased, have read and approved the final manuscript.