Background
Methods
Gathering and structuring usability design principles
Matching usability design principles to known usability flaws
Results
Gathering and structuring usability design principles
First author | Year | Focus | Method used to provide the principles | |
---|---|---|---|---|
[2] | Bates DW | 2003 | Design, implementation, monitoring | Lessons learned |
[19] | Kuperman GJ | 2007 | Design, implementation | Expert consensus |
[4] | Sittig DF | 2008 | Design, implementation, research | Lessons learned / expert consensus |
[20] | Phansalkar S | 2010 | Usability | Targeted review |
[21] | Pelayo S | 2011 | Usability | Targeted review & analysis of cognitive and collaborative tasks |
[22] | Zachariah M | 2011 | Development of a usability evaluation instrument | Phansalkar et al.s’ review and feedback from a preliminary evaluation |
[1] | Horsky J | 2012 | Usability | Targeted review |
[23] | Horsky J | 2013 | Usability | Targeted review |
[24] | Payne T | 2015 | Usability | Expert consensus |
Usability design principles | Summary of corresponding flaws |
---|---|
Meta-principle A:
#1 Improve the signal-to-noise ratio by improving the sensitivity and the specificity of the alerting system in order to decrease the number of irrelevant alerts [1, 4, 19, 20, 23] (e.g., system (non-medical) alerts, alerts with little evidence, low clinical relevance or redundant alerts, alerts that require no action). | |
/ | |
#5 Consider temporal dimensions: interval between drugs’ administration [23]: distinguish “now”, “standing”, and “future” orders, evolution of the unsafe event: increase the severity of the unsafe event if it gets worse [21], time lab tests are overdue [1, 19], interval between the appearance of the unsafe event and the administration of drugs. | |
#6 Consider patient clinical context [1, 19, 23, 24]: besides the specific drug regimen(s) (e.g. dose, route, duration of therapy, sequence of initiating co-therapy, timing of co-administration), add patient and laboratory data into the expected interaction (e.g. age, gender, body weight, mitigating circumstances, predisposing risks factors, drug serum level, renal function, co-morbidity, and previous experiences). Consider the point during patient’s stay at which the alert is presented. | Medication order checking is not patient tailored [26, 35]: alerts may be valid but not applicable to patient clinical context [10, 31, 36]: e.g. pregnancy alerts for male patients and women of non-child-bearing age [32], no distinction between true allergies and side effects [10, 27]. An alert that is supposed to appear the last day of the stay (which is unforeseeable) appears every day [34]. |
Alerts appear while the corrective or monitoring actions have already been taken by the physicians [29]. Some corrective actions that are clinically relevant are not accepted by the system [36]. In some specialties, adverse events are intentional [29, 32] (e.g. psychiatry [32]); in other, drugs are used off-label (e.g. pediatrics [32]). Clinicians already know the alerts [25, 29]. | |
#8 Include fuzzy logic-based algorithms, multi-attribute utility model and filters into the triggering model to change alerts’ activation when certain conditions apply [1, 19, 20]. Define appropriately thresholds to trigger the alerts and to prioritize the alerts according to the patient’s clinical context and the severity of the unsafe event [1, 4, 20, 23] (see #28). | |
/ | |
/ | |
Alerts are not grouped according to their severity [22]. | |
/ | |
/ | |
/ | |
Meta-principle B:
| The system does not inform physicians whether pharmacists review their justification of override and / or find them useful [10]. |
/ | |
/ | |
Justifications and comments are displayed to no one [37] | |
#21 Display consistently the basic alert content, i.e. the main elements of the alert, amongst all clinicians [21, 24]. Nonetheless, the detailed presentation may differ based on clinicians’ expertise (e.g. pharmacists may need more pharmacological data) [1, 23], on their role (privileges, responsibilities) and on the context of use [24]. Details may be presented upon request [21]. | The way data are displayed is not adequate for all clinicians’ types [10]. |
/ | |
Meta-principle C:
| Mental model implemented in the system does not fit users' one [12]. |
Alerts appearance is delayed [12]: lags/down-times of 8 sec. [28] up to 15 sec. [10]. Clinicians must explore several parts of the alerts to get all relevant data [9]: they must scroll [10, 11, 22], explore several tabs [9, 40], or find information in tooltips [26] because short versions of alerts are not sufficiently informative [31]. | |
#27 Streamline users’ workflow in response to alerts [22]. Make the resolution of alerts quick and easy (few steps) through screen operations [1, 22, 24]: cancel or reset alerts in response to the appropriate corrective action, do not require acknowledgment before a corrective action [20]. After the alert is resolved, resume the workflow [19] (see #49). | / |
Alerts of different severity levels and of different types are not distinguished [22]. | |
#29 Adapt alert’s format (e.g. color, symbol) and location on the screen [20, 24]. More severe alerts must be placed within the focal region of the user’s visual field in order of importance while non severe alerts must be placed in side regions [1, 20, 23]. Distinguish system vs. medication alert messages [20]. | |
#30 Adapt alert’s intrusiveness [1, 4, 19, 21, 23 24]. Interruptive alerts should be reserved for high severity warning and used judiciously: they should require an explicit response. Less important alerts must be displayed less intrusively (e.g. on-demand) as messages not requiring any actions. Do not use pop-up alerts for system messages. | |
#33 Present the most critical information on the top-level of the alert: the unsafe event, its causes and its severity [1, 23, 24]. Then display on-demand (linked) information on background and secondary considerations (contextual information, mechanism of interaction and evidence[19, 24]. The suggestion of action could be presented either at the top level or on-demand [20, 24] (see #36). | |
/ | |
Meta-principle D:
| / |
Alert does not provide information on how the causes of the unsafe event conduct to the unsafe event [9]. | |
#41 Relevant data supporting the decision-making process and the suggestions of action [1, 4, 23, 24]: e.g. contextual information, modifying and predisposing factors (e.g. co-morbidity or lab-values). Provide a link to a summary of patient clinical data [23]. Display data necessary to interpret values provided (e.g. thresholds). | The alert does not provide essential patient information for the prescriber [10]. User can link to outside sources of information from elsewhere in the system, but there is no link within the alert [22]. Even when alerts provide patient biological results, the thresholds to interpret them are not presented [9]. |
#42 Suggest, do not impose. Make the system a clinician’s partner [21]: provide clinically appropriate suggestions of action for mitigating the potential harm, do not impose [1, 19, 20, 23, 24]. Present possible ancillary orders such as monitoring/surveillance actions, drug alternative (incl. Dose and frequency) and / or order modification or cancellation [1, 2, 23, 24]. Make the suggestions actionable [24] (see meta-principle F). In case of multiple suggestions, prioritize them and present their conditions of application [24]. Justify suggestions [21, 24]: check locally suggestions [24] and include link to institution-specific guidelines [19], make consensual suggestions [1, 19]. Monitor whether or not users followed through with the suggested action they started; if users fail notify them they did not finish the action they started [22]. | |
Meta-principle E:
| |
#46 A description of the characteristics of the tools included in the alert (e.g. duration of activation). | |
The system does not explain the levels of severity that are used [22]. | |
#48 The events that are checked by the alerting system [20] and the type and format of data (e.g., free-text, origin of the data (other hospital data), name of drugs vs. Anatomical Therapeutic Chemical-ATC codes) | |
Meta-principle F:
| |
#50 Modify the order being entered (or its dose) [1, 22‐24]. For instance, propose formulary drugs lists [19] for formulary drug alerts. Allow ordering a drug suggested [23] or a new drug: in this case, clearly state that the existing drug will be discontinued if the new one is finalized [23], open a pre-populated ordering screen for the new drug [23]. | The system does not guide users for switching a medication [41]. |
The system does not guide user for discontinuing a medication [41]. | |
/ | |
/ | |
#54 Provide patient education [24]. | / |
#55 Delay the alert for a predetermined amount of time (“snooze” function) [24], allow users to get the alert again. | |
#56 Send the alert to another clinician [24]. | The system does not support the transmission of alerts to others clinicians [28]. |
#57 Send the alert into the clinical note template | |
#58 Override the alert (meaning continue ordering, ignoring the alert) [22‐24]. Most severe unsafe events must be more difficult to override (e.g. require a second confirmation, or even no possibility of overriding [23]) than less severe ones. Alerts must require the reason for override [24] (especially the most critical ones, optional otherwise [23]. Avoid text entries, propose a list of 3–4 (max 5 items) selectable coded reasons; reasons must be 1–2 word long [19, 23, 24]. | The system does not provide appropriate options for justifying overrides [28]. The system is not explicit about the necessity to enter a justification [37]; moreover, free-text entries are not effective in the override justification logic [10] and entering data to justify overrides is seen as time burden [10, 36]. |
The system does not provide users the possibility to remove irrelevant alerts that therefore continue to appear [28]. | |
The system asks the users to enter data in the alert and then in the patient record, leading to wasting time and double documentation [28]. |