Health timeline: an insight-based study of a timeline visualization of clinical data

The focus of this study is on time-based visualizations of clinical data (EHR) and the assessment methodology used to validate them. Time-based visualizations are graphical representations of data collected over time. The research literature has a large number of data visualization techniques that vary in their strategies. However, we refined our search to include only visualization tools based on time, and the longitudinal nature of the data. The search was narrowed down further to only include those techniques that were used in the context of clinical data. We were particularly interested in the assessment methodology used to evaluate these visualizations.

LifeLines is a computerized tool that displays clinical data [13] using dots positioned along horizontal lines [14]. A study showed that participants responded 50% faster to a “post-experimental memory test” (p<0.004) [15]. LifeLines was extended in a second version with support for aggregation of temporal events [16]. The focus was on emphasizing prevalence and temporal order of the clinical data. A study revealed that the clinicians were able to confirm hypotheses on the hospital length of stay of patients. LifeLines [13] and LifeLines2 [16] are Java software applications, thus they must be installed in a Java-capable device. These tools provide data filters to narrow down the data exploration and enable the user to focus on certain aspects of the timeline.

TimeLine is a visualization software that displays EHRs chronologically [17]. The data is grouped by categories and displayed along a visual timeline. No assessment of the software was reported in the article. Timeline [17] is the closest application to Health Timeline. It features web support, EHR interfacing and a timeline representation of data with a focus on oncology. Timeline also has a large number of features such as causal models, imaging files, data search, disease progression visualization and data category toggling. It is probable that using Timeline involves a learning curve as it offers several features that would require the user to become acquainted with.

LifeFlow is a visualization tool that summarizes data in sequences using temporal spacing of events [18]. One physician took part in a briefing interview about the visualization of patient transportation data. EventFlow is a drug prescription pattern visualization [19]. A study on the use of asthma drugs was conducted to identify patterns that complied with regulations. LifeFlow and EventFlow are also software applications that require installation. The data are not visualized in a timeline but instead the representation is chronologically ordered as a series of events and outcomes. These visualizations are optimized for understanding the causes and outcomes of patient admissions to hospital.

Bertini and colleagues [20] made a strong case for the objective assessment of visualization tools. A literature review on assessment methods for information visualization reports a number of practical cases and proposes a classification of these methods. The Visual Data Analysis and Reasoning (VDAR) classification group is relevant to our study because it emphasizes the decision-making process, knowledge discovery and visual data analysis. We found that no assessments of this kind have been conducted using clinical data and medical experts.

Time-based visualizations have been found helpful in several use cases. However, without a systematic assessment method, it is difficult to demonstrate how they improve the understanding of the data. To provide a contribution towards the good practice of assessment methods for clinical data visualizations, we conducted and documented the assessment of the Health Timeline using the insight-based methodology. This methodology has been previously used in bioinformatics [8, 21‐23] and well-being data analysis [9, 24].

The VDAR classification group proposed by Lam and colleagues [25] is relevant to this study due to the nature of the clinical data. The purpose of our study is to measure the degree to which a time-based visualization can assist the data analysis and the reasoning process of a clinician when analyzing the clinical history of a patient. In the context of clinical data, we found in the research literature that studies tend to focus on the usability of the interactions with the visualization system. Other studies conduct briefing interviews and questionnaires to try to assess the participants’ understanding of the data. Nevertheless, a consistent evaluation methodology is still missing.

The purpose of our study is to measure the degree to which a time-based visualization can assist the data analysis and reasoning process of a clinician when analyzing the clinical history of a patient. In the context of clinical data, we found in the research literature that studies tend to focus on the usability of the interactions with the visualization system. Other studies conduct briefing interviews and questionnaires to try to assess the participants’ understanding of the data. Nevertheless, a consistent evaluation methodology is still missing.

The insight-based methodology provides an assessment method focusing on the insights generated by a visualization [7]. In this approach, the insights formulated by the visualization audience are assessed in a Likert scale from 1 to 5. To contextualize the insight-based methodology to the clinical data domain, we consulted with two subject matter experts to assist us in establishing the assessment criteria of the insights.

The insights gained in this study are analogous to those observed in the bioinformatics context because they demonstrate different degrees of completeness and accuracy in the understanding of the data represented by the visualization. In the bioinformatics context, predetermined information deemed as correct and complete, was used to compare the insights obtained in the study [8, 21, 22]. The same is true for our study, since two professional psychiatrists reviewed the evaluation criteria. Due to their experience, these psychiatrists were qualified to evaluate the degree of completeness and accuracy of the insights. These experts were also acquainted with the clinical data used in our study.

The insight-based methodology could be used in future studies to quantify the effectiveness of clinical data visualizations. This study provides details on the application of the methodology, including the process required to establish criteria to assess the insight value. The insight value is a key component to analyze and interpret the results of the evaluation. We hope to provide enough documentation about the steps followed to apply the insight-based methodology. The purpose is to provide a case study on evaluating visualizations using the insight-based methodology contextualized to the healthcare domain.

The events in the timeline are grouped in adjacent rows. Each event is depicted as a box. The boxes are grouped in rows with labels on the left edge of the timeline. The events are shown at the corresponding start date position displayed on the horizontal axis. The width of the event box is adjusted to the corresponding end date. The group labels correspond to domains that are clinically relevant, for example medical treatments, visits to the general practitioner, psychiatric consultations and laboratory tests. Users can interact with the timeline by adjusting the displayed time frame (start and end dates). This is done by zooming in and out or panning left and right. Figure 1 shows an example of the Health Timeline portraying the clinical history of a patient.

The rationale behind the design of Health Timeline is to provide a web-based (usable with any device with a web browser) visualization tool for clinicians to assist in the process of understanding the overall condition of a patient. The Health Timeline extracts information from EHR systems and presents the data in an interactive interface.

The design principles followed to develop Health Timeline were based on a simple and comprehensive visualization. In the “Related work” section, we identified similar visualization tools. These tools supersede the Health Timeline in terms of features and complexity. However, in this study, the participants did not receive an introduction or explanation on how Health Timeline software works, this was part of the intended strategy of the study. That is, to design and develop a simple yet comprehensive visualization that would require no introduction to clinicians. In other words, we aim to provide an intuitive visualization of clinical data that clinicians can understand without requiring training.

The Health Timeline builds on similar time-based visualizations and tries to address the challenge of representing data clearly and in a meaningful way without overwhelming the user with excessive details and complex interfaces. During the first assessment, the participants became acquainted with the Health Timeline visualization. In contrast, more complex visualization software, such as LifeFlow [18] or EventFlow [19] would probably require the participants to get acquainted with the tool before conducting the experiment.

The key requirements of Health Timeline came from Australia’s “My Health Record” initiative and were gathered from interviews with clinicians and Information Technology experts with a background in data visualization. These requirements are summarized as follows:

General Criteria: The domain value of an insight is determined using a five-point Likert scale as suggested in the methodology [7] and applied in previous studies [8, 9, 21‐23]. The value depends on the insight’s complexity and depth. An insight is complex if it can associate more than one element in the data by providing a relationship or association between them, for example; “if “a” increases then “b” decreases”. The depth is the degree to which the underlying reasons behind the data are explained, that is, a plausible explanation was given that could justify the state of the data. For instance, an explanation for values outside of the recommended range for certain physiological measurements could indicate the presence of a disease. In our study, we recognized that additional refinement was needed for the evaluation criteria used to assess the insight value, due to the complexity of the data.

The consulting psychiatrists analyzed a sample of 100 insights (extracted from the assessments made by the participants) and evaluated them according to what they considered informative and helpful in understanding the patient’s condition. The consulting psychiatrists were acquainted with the data and thus were able to understand the underlying condition and overall health of the patients. The insights provided to the consulting psychiatrists included the patient identifier number and the text transcribed manually from the voice recording. This allowed them to have a reference to the original data to objectively determine if an insight was accurate and meaningful in understanding the medical history and clinical condition of the patient.

Insights that describe associations of events regarding their frequency, pattern and irregularities were evaluated with two points. These insights connect multiple occurrences of similar events and show that the clinician can track down a pattern. For example, insights like “the patient has regular appointments with a General Practitioner” or “the patient is taking a high dose of the medication with regular frequency”. These insights have a time component as the participant identifies a regularity or irregularity in their occurrence.

The insights that include theories that could explain some or all the symptoms and thus the reasoning behind a prescription, or a specialist appointment, were evaluated with three, four or five points depending on how much information they could combine to produce a valid clinical statement. These insights could be considered a hypothesis and could explain valid clinical scenarios that result in drug treatment, laboratory tests, specialists’ visits and in-patient treatments. The following insight provides an illustration: “the patient has experienced depression and anxiety, that would explain the prescription and regular use of the drug treatment and also visits to the specialists, this also ties together an emergency episode in January 2013 and another admission in June, overall the patient’s mental health improved towards the summer and it seems that changing the drug treatment improved the outlook”. Table 1 summarized the criteria for each value with examples extracted from the transcription of the assessments.

The insight-based methodology recommends two mechanisms to record the insights, the “thinking aloud process” and the use of a written diary to record the steps taken during the data analysis. We aimed at making the participation in the study as simple as possible since the participants were professional psychiatrists with busy schedules. For this reason, we opted to use the “thinking aloud process” and capture the insights, comments and other statements via phone calls and recording software. We later transcribed the recordings manually to proceed to the analysis stage of the study.

The “thinking aloud process” is one of the most common techniques in usability studies [26]. It consists of asking the participant to verbally express the thought process while using the system under testing. In this study, we asked the participants to follow a link received via email and once they were ready to begin, they called a number and started the “thinking aloud process” of verbalizing their thoughts as they explored the data. This was an effective and convenient way to conduct the study since the participants had the freedom to conduct the assessments at their convenience.

By using the “thinking aloud process” we were able to capture the actions the participants were conducting during the assessments, for example scrolling, panning and zooming to a particular time frame.

The patient selection was based on the complexity of the diagnosis. All five cases had a diagnosed mental disorder and received continuous care and monitoring throughout their treatment. Three cases had in-patient episodes. A fourth patient was compliant with the treatment and the disorder was managed throughout the recorded period. A fifth patient had substantial changes in overall mental health.

The selected data included Pharmaceutical Benefit claims, Medicare Benefits claims and hospitalization dates over 3 years between 2012 and 2014. The data also included visits to General Practitioners, specialists, laboratory tests, emergency hospital admissions, drug treatment, among other categories. The claims records were sourced from Medicare Australia and hospitalization events were sourced from SA Health. Australia’s Pharmaceutical Benefits Scheme (PBS) contains information about the type of medication, amount of medication supplied, and the date of supply, while the Medicare Benefits Scheme (MBS) contains information on tests and services by Medicare eligible practitioners and the associated date of the service. The PBS and MBS measures were relabeled into clinical terminologies by psychiatrists involved in the project. The data used are considered a clinical history of a patient as part of the EHR.

All five patients were presented with both visualizations to account for control. The display order was controlled to first show the Health Timeline and direct any bias towards this visualization upon starting the assessments. There was no introduction or instructions given to the participants on how the Timeline works. The idea was to build a visualization that would be intuitive from the start.

The participants conducted the same assessments independently. The participants were asked to make one assessment at a time. The order was the same for all the participants. Each assessment featured the clinical data of a single patient. The patient clinical data was presented to the participant twice, one time in tabular data and another in the Health Timeline visualization. The participants conducted a total of 10 assessments, 5 with the Health Timeline and another 5 with the tabular data. The visualizations were alternated throughout the study. The first assessment used the Health Timeline, the second tabular data, the third Health Timeline, the fourth tabular data and so on. The order of the patients (clinical data) was also controlled so that no consecutive visualization would show the same patient data.

The experiment was intended to be performed by clinicians. As such, time was a major constraint. Empirically we have observed that taking no more than 30 min of their time would be ideal. Keeping each assessment to no more than three minutes was considered to be short enough for them to be willing to take part in the experiment. The 30-min time frame was determined by the three-minute duration of each assessment, a total of 10 assessments which showed the clinical data of five patients with both visualizations, Health Timeline and the textual data. Additionally, we allowed clinicians to take the assessments at their convenience, so if a clinician was able to spare three minutes, then that would be enough to conduct one of the 10 assessments.

A total of 10 psychiatrists were invited to take part in the study. Five of them accepted the invitation. Each participant conducted a total of 10 assessments, 5 with the Health Timeline and 5 in textual data. The presentation order was the same for all participants.

We recruited five professional psychiatrists to participate in the assessments. The participants had experience in the field and had previously worked with similar patients as those used in the study. The participants also expressed their written consent to participate.

A total of 121 min of audio were transcribed. The transcribed text for each insight was organized using a spreadsheet that included the patient identifier, the participant identifier and the time at which the insight was mentioned. The data set produced was saved as a comma separated value file for further analysis and contained the following attributes (Additional file 1):

As in previous studies [8, 9, 21‐23], we analyzed the data and applied the following metrics to evaluate the data representations:

Mann–Whitney–Wilcoxon (MWW) was used to calculate the statistical significance of the metrics per assessment and the time to first insight since they are not normal distributions. MWW is the recommended method for these distributions [28, 29].

The statistical significance of the total number of insights was calculated using standard Chi squared because the two populations could be treated as categorical data. The insight value corresponds to the categories, the values are discrete from 1 to 5 using a Likert scale. Therefore, the statistical significance can be obtained with Chi squared [29].