MPA CDSS

The Open Source Clinical Application Resource (OSCAR) electronic medical record is the platform for the MPA CDSS. The MPA is a suite of electronic forms that are integrated with the workspace, workflow and databases contained within the electronic medical record. The MPA includes ‘encounter guides’ that actively and passively assist a clinician through a patient visit for low back pain, neuropathic pain and opioid management based on clinical practice guideline recommendations. The MPA provides data fields for recording assessment information during the encounter; automatic filling of data fields where information exists elsewhere in the patient’s record; automatic extraction from data fields (such as scores from questionnaires) into a tracking table and graphic presentation; web links to educational materials for both clinicians and patients; and brief video presentations to assist clinicians with difficult discussions with patients.

When this study was initiated, all of these functions were available at either partial or full level of function. The initial development requirements of the MPA functionality were based on focus group needs, assessments of inter-professional end-user clinicians, as well as an expert panel consisting of e-health academics, programmers, pain clinicians and knowledge translation experts.

Sample and Setting

An academic family medicine centre with 2 clinics, 30 physicians, 8 nurse practitioners, 70 family medicine residents and a diverse allied health group. A convenience sample of potential users of the MPA was used and included family physicians, family medicine residents and nurse practitioners. The target sample size was 5–8 participants in each iteration, based on the literature that suggests that 80% of deficiencies will be found in samples of 4–5 participants.3

Data Collection

Two iterations of usability testing were performed, 2 months apart, with an identical format and different participants. A demonstration version of the CDSS was used for testing, and participants received no training in its use prior to working through a scenario. Three different patient case scenarios were provided: medication renewal, diagnosis of low back pain and monitoring of neuropathic pain. Each study participant was observed separately, and sessions were facilitated by one of the authors (RM) and observed in person or on camera by a team that included co-investigators, research staff, business analyst and the system software developer. Screencast video and audio were recorded and transcribed, and all the observers wrote field notes about what they observed and what the participant reported during the exercise.

The System Usability Survey (SUS) was completed by each participant immediately following the exercise. The SUS consists of 10 items, each scored on a 5-point Likert scale that ranges from 1 – strongly disagree to 5 – strongly agree and provides an overall score of system usability.3^,4 Scoring involves subtracting 1 from all odd items, and subtracting all even numbered item responses from 5, which scales each item from 0 to 4. The total is multiplied by 2.5 to provide a score out of 100, which is interpreted as a percentile ranking and not as a percentage. SUS can also provide sub-scores for learnability and usability.

Data Analysis and Use in Development

For each iteration, all the observer notes and SUS scores were discussed by observers and then by the research team. Directed content analysis was used, and the number of barriers and facilitators of usability was determined and used to generate a priority list for system revision. The mean, SD and p-value between the two iterations were calculated for SUS scores. The MPA was modified following round 1 of testing and the revised version used in iteration 2. Further modifications followed the second round of testing.

Participants

Seven clinicians (three family physicians, three nurse practitioners and one family medicine resident) participated in round 1 testing. Six new clinicians (three family physicians, two nurse practitioners and one family medicine resident) participated in round 2.

Qualitative Data

Transcribed audio recordings and observer field notes were analysed and resulted in the following categories of usability barriers and facilitators: system, UI, content and technical (see Table 1). Most identified barriers were related to the UI in both rounds, and this and technical were the areas that were modified most extensively between testing rounds. For example, titles of resources were identified as ambiguous in round 1. Between rounds, these were modified to be clearer, and round 2 demonstrated that this issue was largely resolved.

Most identified facilitators were related to content, and many were related to UI. For example, participants felt that the added feature of a timeline that correlated medication to pain screening scores could help track changes in pain scores based on medications taken. Changes between round 1 and 2 testing were not analysed for statistical significance.

SUS Score

The overall SUS score decreased from 81.10 (±12.02) out of a possible 100 points in round 1, to 70.40 (±6.78) in round 2. This decrease in score was not statistically significant (p = 0.86; Table 2). Subscale scores were calculated for learnability (SUS items 1 and 4) with round 1 mean score 3.57 (SD = 1.72) and round 2 3.16 (SD = 1.34 and p = 0.78). Subscale scores were also calculated for usability (SUS remaining items) with round 1 mean score 3.16 (SD = 1.40) and round 2 2.73 (SD = 1.06 and p = 0.69).