2 Methods
In order to explore to what degree the STRIP Assistant is usable for aiding GPs and pharmacists with performing medication reviews, an experiment was conducted.
2.1 Participants
The experiment was aimed at GPs and pharmacists. Fifty-two respondents were selected through opportunity sampling, as the researchers lacked the resources to guarantee participants’ cooperation through reimbursement. All participants were required to be either GPs or pharmacists in Dutch primary care and had to fully complete both parts of the experiment to warrant inclusion. Of the 52 responses, nine had to be discarded because of corruptions in the data: three participants did not fill out the unassisted first part of the experiment, five did not assign drugs to diseases or did not respond to advice during the assisted part, and one record was a duplicate. Finally, 43 participants’ results were eligible for inclusion in the data analysis.
Respondents were recruited through the researchers’ personal networks (i.e. symposia, conferences and [training] conventions). They were briefly informed about the experiment’s goal and assured that their anonymity would be guaranteed. As an incentive, respondents were offered 3 months’ use of the software application for their own patients, free of charge.
2.2 Study Design
The experiment took the form of a pre-experiment with a one-group pre-test post-test design, as described by ‘t Hart et al. [
39]. Respondents were placed in a single research group; an initial test was performed, after which a stimulus was applied and the test was repeated.
In the test, the medical records of two polypharmacy patients, which had been selected from the geriatric ward of an academic medical centre for the study by Drenth-van Maanen et al. [
14], were used; they were actualized (i.e. drugs that were no longer available were replaced by their contemporary counterparts) and confirmed to be of comparable difficulty by an expert panel of geriatricians specializing in clinical pharmacology (PJ and WK). During the experiment, respondents were asked to optimize the first case in their usual manner and the second one using the STRIP Assistant.
The three usability aspects of effectiveness, efficiency and user satisfaction were operationalized in the experiment as follows: effectiveness was measured by recording the respondents’ medicine prescriptions, after their optimization. The decisions made by the respondents were then compared with the medication list that the aforementioned expert panel of two geriatrician–pharmacologists prepared. They reached consensus on the pharmacotherapeutic changes that should be made in the medical records that were optimized by the respondents, and classified the decisions as correct, neutral or potentially harmful. Efficiency was operationalized by recording the time that respondents took to optimize the two cases. Finally, user satisfaction was measured through a standardized questionnaire—the System Usability Scale (SUS)—consisting of ten statements with which respondents had to indicate their agreement or disagreement on a Likert scale.
All data were gathered between November 2013 and June 2014. During that time, no changes of any kind were made to the software.
2.3 Outcome Measures
The main outcome measure was the difference in the percentage of appropriate decisions made by the participants without and with use of the STRIP Assistant. Secondary outcome measures were the difference in the number of inappropriate decisions taken by participants without and with use of the STRIP Assistant, the difference in the time needed to perform the medication review without and with use of the STRIP Assistant, and the extent to which participants experienced their use of the STRIP Assistant as satisfactory.
2.4 Instrument
The STRIP Assistant has been designed as a stand-alone web application, which aims to assist GPs and pharmacists with pharmacotherapeutic analysis of patients’ medical records. The user interface accommodates the six phases of the STRIP medication review (i.e. drugs–disease assignment, undertreatment, overtreatment, side effects–drugs assignment, clinical interactions and dosage frequency). In most phases, users are shown advice on missing, superfluous or incompatible drugs. The items of advice are patient specific, incorporating their diseases, drugs, side effects and users’ actions up to that point. The STRIP Assistant’s rule base consists of a combination of well-established clinical rule databases and specific implementations of the START and STOPP criteria.
For the experiment, the user interface was enhanced to first display one of the patient cases in a bulleted list, summing up his/her diseases, drugs, side effects, complaints, measurements and laboratory test results.
2.5 Procedure
Respondents were asked to optimize the first case in their usual manner, specifying in an adjacent text field which drugs should be added or removed for optimal treatment. They were then shown a 1.5-min video explaining the use of the STRIP Assistant, after which they were presented with the second patient case in the STRIP Assistant user interface. Respondents were asked to optimize this case through the STRIP process, reacting to the advice generated by the application. Each screen contained a help button explaining what was expected of the respondents.
After optimizing the second case, respondents were presented with the SUS, consisting of ten statements with which they had to indicate their agreement or disagreement on a Likert scale. Finally, information on the respondents’ demographic characteristics (age and sex) was collected, alongside their experience with medication reviews and CPOE systems. In a text field, respondents could optionally leave their comments.
2.6 Statistical Analysis
In all cases, an expert panel determined the correctness of the decisions made by the participants. Slight corrections to the data had to be made to account for the differences in the potential number of appropriate decisions that respondents could make in each case: 17 in the unassisted case and 20 in the assisted one. Similar corrections were applied to account for differences in the possible number of inappropriate decisions: 30 in the unassisted case and 40 in the assisted one. Paired t tests were used to analyse the data pertaining to appropriateness and inappropriateness of decisions, and the differences in time spent.
The results of the SUS were formatted in the manner described by Brooke [
40]: for the odd questions, 1 was subtracted from the values; for the even questions, the values were subtracted from 5 to get the corrected scores. The sum of all questions was multiplied by 2.5 to calculate the final score ranging from 0 to 100.
3 Results
3.1 Descriptive Statistics
Of the 43 respondents whose answers were valid, all but four filled out the questions pertaining to their personal characteristics (Table
1). The majority of these were female (62.8 %). Most respondents were in their fifties (32.5 %) or forties (18.7 %). Seven participants (16.3 %) were aged between 31 and 40 years, and five (11.6 %) were in their twenties. Five (11.6 %) were over 60 years of age. Most were either GPs (72.1 %) or pharmacists (9.3 %). Two were dispensing GPs (4.7 %) and two were GPs in training (4.7 %). Most were experienced with performing medication reviews: 18 participants (41.9 %) did not use STRIP for their reviews, while 12 (27.9 %) did. Nine (20.9 %) had no experience performing medication reviews at all.
Table 1
Overview of participants’ characteristics
Sex |
Male | 12 | 27.9 |
Female | 27 | 62.8 |
No data | 4 | 9.3 |
Age |
≤30 years | 5 | 11.6 |
31–40 years | 7 | 16.3 |
41–50 years | 8 | 18.7 |
51–60 years | 14 | 32.5 |
≥61 years | 5 | 11.6 |
No data | 4 | 9.3 |
Function |
GP | 31 | 72.1 |
Pharmacist | 4 | 9.3 |
Dispensing GP | 2 | 4.7 |
GP in training | 2 | 4.7 |
No data | 4 | 9.3 |
Experience with medication reviews |
STRIP | 12 | 27.9 |
Other medication review method | 18 | 41.9 |
None | 9 | 20.9 |
No data | 4 | 9.3 |
3.2 Usability Hypotheses
In total, 86 medication reviews were performed by the participants; in half of these cases, they used their usual care methods to perform the optimization, in the other half they were aided by the software application. An overview of all tested hypotheses is shown in Table
2. On average, the participants prescribed eight drugs for the unassisted case and 14 for the assisted one.
Table 2
Overview of the tested hypotheses and their statistical outcomes
The STRIP Assistant positively influences the number of appropriate decisions made in a medication review: accepted | 418 (58 %; mean 11.44; SD 2.63) | 656 (76 %; mean 15.26; SD 2.05) | Paired t test: t(42) = 8.80; p < 0.0001 |
The STRIP Assistant negatively influences the number of inappropriate decisions made in a medication review: accepted | 302 (42 %; mean 9.36; SD 2.53) | 210 (24 %; mean 4.88; SD 2.23) | Paired t test: t(42) = 8.93; p < 0.0001 |
The STRIP Assistant negatively influences the time taken to perform a medication review: rejected | 13 min (mean 0.94; SD 0.40) | 24 min (mean 1.34; SD 0.20) | Paired t test: t(42) = 7.07; p < 0.0001 |
Users perceive using the STRIP Assistant as satisfactory: rejected | | SUS score 63.25 | Quality consensus test: 63.25 (<70) |
A paired t test showed a statistical difference in the appropriateness of the decisions made without the STRIP Assistant [mean 11.44; standard deviation (SD) 2.63] and with the STRIP Assistant [mean 15.26; SD 2.05; t(42) = 8.80; p < 0.0001]. A Wilcoxon signed-rank test showed similar results (Z = −5.40; p < 0.0001). From totals of 418 unassisted correct decisions and 656 aided ones, over decision totals of 720 and 866, respectively, it follows that the proportion of appropriate decisions increased from 58 % without help to 76 % with the STRIP Assistant.
A paired t test showed a statistical difference in the inappropriateness of the decisions made without the STRIP Assistant (mean 9.36; SD 2.53) and with the STRIP Assistant [mean 4.88; SD 2.23; t(42) = 8.93; p < 0.0001]. The percentage of inappropriate decisions decreased from 42 % in the unassisted case to 24 % in the assisted one.
On average, participants took 13 min to complete the unassisted part of the experiment and 24 min to complete the assisted medication review. A paired t test of the base 10 logarithm of these values showed a statistical difference in the time taken without the STRIP Assistant (mean 0.94; SD 0.40) and with the STRIP Assistant [mean 1.34; SD 0.20; t(42) = 7.07; p < 0.0001]. This indicates that participants spent significantly more time optimizing medication with the STRIP Assistant.
On average, the respondents assigned the STRIP Assistant an SUS score of 63.25 out of a possible maximum of 100. This value is lower than the quality threshold of 70 arrived at by Bangor et al. [
41] and corresponds to a marginal acceptance rate in a later paper by the same authors [
42].
4 Discussion
4.1 Effectiveness
This study has shown that a decision support system can make GPs and pharmacists perform better medication reviews, albeit in an experimental setting with preselected patient cases. This is in line with the consensus on the effectiveness of health recommendation systems in the literature [
27]. More specifically, the results indicate that the choice for a recommender based on a predetermined explicit knowledge base yields viable results in a complex domain with potentially far-reaching implications. Rather than relying solely on collaborative or content-based filtering, a knowledge base guarantees a minimal quality level when recommendations are generated [
43].
Even though the medication reviews performed with the STRIP Assistant were significantly better than those performed without assistance, a non-negligible number of mistakes the respondents made (15 %) could be attributed to software suggestions. In this experiment, each START advice was presented as an alphabetically ordered list of medicines that users had the possibility to prescribe. In practice, many users picked the first item in the list, resulting in an overabundance of suboptimal choices; when adding a vitamin D supplement, for example, many users picked alfacalcidol instead of cholecalciferol, even though the former has fewer and more specific indications. Few publications have touched upon the subject of decision support systems generating incorrect recommendations; consequently, strategies to prevent them are lacking [
44‐
46]. Hybrid recommendation systems, combining an explicit knowledge base with content-based or collaborative filtering, have been shown to outperform their simpler counterparts [
43]. As long as the risk associated with automatic learning systems in a precarious domain such as health care is accounted for, a hybrid approach may prove beneficial in improving recommenders’ effectiveness.
4.2 Efficiency
Contrary to our assumption, performing medication reviews with the STRIP Assistant was less efficient (i.e. it took more time) than optimizing drugs manually. Traditionally, the three aspects of usability are assumed to be positively correlated [
47]. However, a different perspective viewing effectiveness and efficiency as conflicting requirements in a project has been proposed by Nilsson and Følstad [
48]. In an experiment such as the one in this study, where respondents either use their habitual approach or have to learn a new structured method, a drop in efficiency can be reasonably attributed to effectiveness and efficiency conflicting. Because of the experimental setting, unfamiliarity with the method and the user interface is likely to play a role as well.
Conducting experiments in which more gradual changes in the method are applied may result in improvements in both effectiveness and efficiency; in a study related to this one, a paper version of the earlier POM was tested in an experiment [
14]. It, too, proved to be less efficient than performing a medication review manually. However, the software-aided reviews performed in this study took less time than the paper-based ones in the previous study. This lends credibility to the assumption that gradual changes may improve all aspects of usability simultaneously.
4.3 User Satisfaction
Respondents perceived using the STRIP Assistant as only marginally acceptable. The average SUS score of 63.25 was lower than the commonly accepted quality indicator of 70 [
41,
42]. This aspect, too, can be understood by viewing the usability aspects as conflicting requirements [
48]. The suboptimal prototypical design of the software’s user interface, and the respondents’ unfamiliarity with the application, may explain this inconsistency with the consensus in the literature.
4.4 Clinical Relevance
Methods for medication review have proven to be valuable in improving prescriptions for polypharmacy patients. The POM, which served as a foundation for the STRIP method, has led to improvements in appropriate decisions in medication reviews [
14]. The START and STOPP criteria, which constitute a major part of the STRIP Assistant’s knowledge base, have been shown to be associated with improvements in medication appropriateness, reductions in adverse drug reactions and decreases in drug use and costs [
17,
49].
The two patient cases used in this experiment were comparable in their complexity and number of medicines, but there could, for reasons of validity, not be a complete overlap of diseases and drugs. This makes it difficult to determine the clinical relevance of the intervention. Nevertheless, the most noticeable improvements in the adequate prescribing of drugs were the treatment of osteoporosis with bisphosphonates, calcium and vitamin D; and the treatment of systolic heart failure with angiotensin-converting enzyme (ACE) inhibitors. The most important improvement relating to stopping medicine use was discontinuation of digoxin when atrial fibrillation was adequately treated with beta blockers. These interventions correspond to the guidelines of the START and STOPP criteria [
17].
Thus, the results in this study confirm the results of previous studies—namely, that structured methods for medication review significantly improve the medication appropriateness of prescriptions.
4.5 Limitations
When these results are interpreted, the experimental nature of the method should be taken into account. The STRIP Assistant’s usability has been tested and validated with real patient cases in a controlled environment, but it has not been validated in practice with users reviewing their own patients. While the results lend credibility to the STRIP method being useful in practice, this study does not prove its clinical relevance.
When the results of this study are generalized, the limited number of participants should be considered, as well as the sampling method. Forty-two GPs and pharmacists participated voluntarily, raising the possibility that they were positively biased towards use of a clinical decision support system to aid them with medication reviews.
4.6 Further Research
A randomized controlled trial incorporating a large representative sample should be conducted to conclude the STRIP Assistant’s effectiveness, efficiency and user satisfaction. Further research should focus on its usability through evaluation in a real-life setting over a longer period of time, exploring to what extent experience influences users’ effectiveness and efficiency in working with the software. Furthermore, longitudinal research could show if the STRIP Assistant is clinically relevant in practice and could evaluate its impact on adverse effects and medicine costs.
5 Conclusion
In this study, a clinical decision support system (the STRIP Assistant) designed to aid GPs and pharmacists with conducting medication reviews was validated in an experimental setting. The results showed that use of the STRIP Assistant positively influenced the number of appropriate decisions made in a medication review of elderly polypharmacy patients and decreased the number of inappropriate choices. Contrary to our assumptions, users spent more time optimizing prescribing with the STRIP Assistant than without it. The users perceived the experience of using the software as only marginally acceptable. Further research is needed to determine whether optimization of polypharmacy with the help of the STRIP Assistant is clinically beneficial.
Acknowledgments
We would like to thank the participating GPs and pharmacists for their contributions to this study.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.