Assessment tool for hospital admissions related to medications: development and validation in older patients

The first part of the study was an unstructured review of the available literature to find existing methods or tools used to identify MRAs. Relevant articles related to MRAs were obtained using the Medline database in February 2016. Articles in languages other than English or Swedish and abstract-only articles were excluded. The literature search identified several articles using various methods to identify MRAs [3, 4, 6, 11, 13, 25, 28‐40]. Some studies included a variety of MRPs [3, 4, 11, 25, 31, 32, 34, 35, 37‐39] as causes of MRAs, often referring to existing MRP classification systems like the eight categories by Strand et al. [2]. Some studies only included ADRs [6, 13, 29, 30, 33, 36, 40]. When assessing causality, one study [28] used completely implicit methodology while all the others [3, 4, 6, 11, 13, 26, 29‐38] used established criteria such as the Naranjo probability scale [21], the Hallas criteria [22] or the Kramer algorithm [20] to guide an expert panel (consisting of physicians or pharmacists with various degrees of seniority).

The results of the literature search were used to develop the preliminary version of our tool. We defined an MRA as a hospital admission of which an MRP is either the main cause for admission or a significantly contributing cause for admission (i.e. without the MRP, the patient would not have been admitted). Elements from established tools and previously published studies were listed to include all relevant categories of MRPs with the potential to cause, or contribute to, a hospital admission, and to include reasons for classifying admissions as non-medication-related [2, 5, 11, 19‐21, 25, 41‐46]. Overlapping elements were combined and reformulated. The tool was designed in the form of a questionnaire with yes/no answers, and it consisted of ten questions in the final version (Table 1). Explicit lists with medication-specific triggers or clinical rules were excluded to make the tool less time-consuming. The rationale for choosing the questionnaire format was that it would be easy to use. Questions 1–3 in the tool are used to identify admissions that are unlikely to be medication-related (U1–3), while questions 4–10 are used to identify possibly MRAs (P4–10). The assessment is finished as soon as the answer is “yes” to any of the questions. Only if all the questions are answered “no”, the assessment is indecisive and should still be examined by an expert panel. The terms unlikely and possibly are in line with the causality terminology of the UMC-WHO system [23]. The reason for not distinguishing between other degrees of certainty or preventability was to make the assessment not too complex and time-consuming. For the same reason, we decided that only a limited amount of data would be available for the assessments (admission notes, medication list upon admission, laboratory data during hospital stay, and discharge summary). Instructions for use, including examples, were developed for the tool (“Supplementary material”).

The concept of content validity relies on the assumption that a tool is intrinsically valid if all relevant aspects and no irrelevant aspects are included [17]. The content validity of the preliminary version of the tool was assessed using a questionnaire to score each question for relevance, understandability and completeness. Seven clinical pharmacists were asked to fill in the questionnaire and were encouraged to suggest additions or modifications. The questions that gained low scores were deleted or changed. Two questions (P9–10; Table 1) were added and seven questions were rephrased. The number of questions was eventually set at ten in the final version: Assessment Tool for identifying Hospital Admissions Related to Medications (AT-HARM10).

One hundred admissions of patients aged 65 years or older, discharged from two internal medicine wards at Uppsala University Hospital between January and April 2016, were randomly selected to test the content validity of the final version of the tool. Data were obtained from the hospital’s electronic medical record system (Cosmic^®). Seven pharmacy students (five final-year undergraduates, one doctoral student and one postgraduate clinical pharmacy student) received information about the tool (“Supplementary material”) and a 1-h training by two researchers (YA-S and UG) on how to use the tool. One of the researchers (UG) was an experienced clinical pharmacist and researcher. The students then applied the tool to five admissions (none of which were subsequently used in the study). The results of the assessments were discussed in plenum. Each student assessed 50 or 100 of these admissions (see “Inter-rater reliability” section), resulting in a total of 400 assessments. The number of the specific question within AT-HARM10, that was used to classify each admission as unlikely to be or possibly medication-related, was recorded for these 400 assessments, to determine whether all questions were relevant and which questions were used the most.

An assessment tool must be practical and it should be possible to use it within a reasonable time frame [48]. Fifteen clinical pharmacists each applied the tool to 10 randomly selected admissions of patients aged 65 years or older, discharged from one internal medicine ward and one geriatric ward at Uppsala University Hospital (Sweden) between December 2015 and January 2016. Data were obtained from the hospital’s electronic medical record system (Cosmic^®). The pharmacists evaluated whether the limited patient data provided for the assessments were sufficient to satisfactorily answer the questions. They also discussed the tool’s appropriateness and user-friendliness. Time spent on assessing the admissions was measured and averaged to determine the acceptability of the time taken.

The inter-rater reliability (IRR) refers to the degree of consistency among the assessors when assessing the same set of samples [15]. The seven participants of the content validity test (five undergraduate and two postgraduate pharmacy students) were divided into four pairs (one student was part of both pair 1 and 4 to enable the formation of four pairs). The pairs consisted of either only postgraduate students or only undergraduate students (Table 2).

The tool, instructions for use and examples (“Supplementary material”), and anonymized patient data from 50 admissions (either 1–50 or 51–100 of the 100 admissions used for the content validity test, see “Content validity” section) were sent to each assessor. These patient data consisted of copies of the electronic medical records: admission notes, medication list upon admission, laboratory data during hospital stay and discharge summary.

Each assessor then independently applied AT-HARM10 to their assigned 50 hospital admissions, classifying them as either unlikely to be or possibly MRAs. After assessing the admissions separately, each pair of assessors discussed the admissions that they disagreed on to reach consensus. Cohen’s kappa within each couple and Fleiss’ kappa between pairs assessing the same admissions were calculated to determine the IRR. Cohen’s kappa measures the agreement between two assessors and Fleiss’ kappa is used in cases with more than two assessors [15, 49]. The kappa values were then interpreted according to Hammond et al. [15] (Text Box 1).

The traditional definition of criterion-related validity (CRV) is a measure of the validity of a tool by correlating the results with those from some other measure, ideally a gold standard, which has been used and accepted in the field [17]. The sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), as defined in Text Box 2, were used to measure the CRV.

We defined the gold standard as an expert panel of experienced clinicians. One consultant physician (geriatrics and primary care specialist) and one senior clinical pharmacist and researcher (UG) assessed the same 100 admissions as the four pairs of study assessors, see “Inter-rater reliability” section. The experts had access to all patient data in the electronic medical journal. These data included medical notes from all healthcare professionals, medication histories, and laboratory results from both hospital and primary care facilities. They did not use AT-HARM10 or any other tool for the evaluations. The experts assessed the admissions individually, classifying them as either unlikely to be or possibly an MRA. They then discussed the cases on which they disagreed to reach consensus and hence created the gold standard for the 100 admissions. In cases where consensus could not be reached, a third expert was available for a decisive vote.

The sensitivity, specificity, PPV and NPV were calculated according to the formula in Table 3. The primary outcomes for the CRV were the consensus results of the AT-HARM10 assessments of all 100 patient admissions, performed by couples 1 + 3 and 2 + 4, compared to the gold standard.

After the development phase resulting in the final tool, AT-HARM10, the question that was used most often when assessing the admissions was question U1 (148/400), followed by questions P8 (75/400), P4 (68/400) and P5 (47/400), see Fig. 1. All the questions were used at least twice. In all cases, at least one question was answered with “Yes”, hence no expert panel was needed.

According to the fifteen assessing clinical pharmacists, AT-HARM10 was sufficiently relevant and user-friendly and was easy to use; no further changes were suggested. The limited data obtained from the example patients were deemed sufficient to answer the questions in the tool. The time used to assess each admission was on average 5.7 (range 2.5–14) min.

The strength of agreement was substantial for one pair of study assessors (Cohen’s kappa 0.75) and moderate for the other three (Cohen’s kappa 0.45–0.57), see Table 4. The strength of agreement among all assessors was moderate (Fleiss’ kappa 0.58 and 0.46 for the two batches of data).

The gold standard experts reached consensus for all 100 assessments, which resulted in 50% of the 100 admissions being classified as unlikely to be and 50% as possibly a MRA. Pairs 1 + 3 and 2 + 4 assigned 52% and 42%, respectively, to unlikely, and 48% and 58%, respectively, to possibly. The sensitivity was 70% and 86% for pairs 1 + 3 and 2 + 4, respectively. The specificity was 70% and 74% for 1 + 3 and 2 + 4, respectively. The PPV and NPV were 73%/74% for 1 + 3/2 + 4 and 71%/83% for 1 + 3/2 + 4, respectively (Table 5).