Eye-tracking for assessing medical image interpretation: A pilot feasibility study comparing novice vs expert cardiologists

Many experience-based differences in the visual interpretive process have been revealed through eye tracking [2, 4, 11‐13]. Monitoring eye movements is valuable for objectively characterizing the visual search process and in some cases predicting diagnostic outcomes. For the present study, eye movements can be parsed into two meaningful units: fixations and saccades. Fixations describe the momentary pauses of the eyes to foveate a restricted region of space, and saccades describe the rapid, ballistic movement of the eyes between successive fixations [19]. Fixations are characterized by their location in the visual world, and their duration; in general, the more fixations and longer their duration, the more visual attention and interest the viewer has in a region [20]. Saccades are thought to be preceded by an attentional shift to a different location, which results in a saccade to afford foveation of a new goal region [21, 22]. Saccades can reveal how dramatic (usually measured in degrees, or amplitude) the shifts of attention are across a scene. The peak velocity of saccades can help researchers gain insight into varied states of workload and arousal. Specifically, higher peak saccade velocity correlates with greater sympathetic nervous system activation, for instance during states of arousal or uncertainty [23], possibly driven by excitatory inputs to oculomotor neurons from the reticular formation [24, 25].

Tracking the allocation of visual attention over a medical image allows us to dissociate between a failure to view critical scene regions, versus a failure to accurately interpret those regions [26]. Specifically, if a physician fails to fixate on critical regions of a medical image, any diagnostic decision errors that accrue can be sourced to a failure to find critical diagnostic regions. In contrast, if a physician fixates on critical regions but does not arrive at an accurate diagnostic decision, the error can be attributed to the interpretive process. This distinction is critically important for identifying possible sources of error and how they change as a function of expertise development, and informing the development of tailored student assessments and training curricula. For example, eye tracking can exemplify expert eye movements to medical students, allowing them to learn viewing strategies for reviewing complex slides [5]. Eye tracking also holds potential for evaluating student competency progression, objectively assessing skill development and providing formative feedback [6, 27].

Earlier research using eye tracking to understand expertise-related differences in visual search among medical professions is restricted to examining static images due to the complexity of tracking and interpreting eye movements over moving (dynamic) scenes [28, 29]. This is because not only are the head and eyes constantly moving relative to the computer monitor, but also relative to a moving scene with constantly changing stimulus locations. For instance, for a cardiologist reviewing an angiogram, a region of diagnostic importance may move across a scene as tool angulation or position changes, necessitating a tedious frame-by-frame coupling of region location and eye location over time. For this reason, most studies examining eye movements with medical images artificially restrict zooming and panning behaviour, simplifying analysis but also possibly reducing relevance to behaviour elicited during routine clinical practice [26, 30, 31].

To increase the efficiency of interpreting eye movements over dynamic scenes, researchers can temporally couple eye tracking with logged interface behaviour to relate eye position to specific video frames [32‐34]. To prioritize video frames, a behaviour analysis can assist in inferring a viewer’s interest in certain video frames: using the ShotRank technique, the frequency and duration of viewing video frames indicates interest in the information available on those frames [35‐37]. Specifically, repeated interest in certain video frames (e.g., pausing, reviewing) can provide important information regarding the spatiotemporal distribution of critical diagnostic regions throughout a video. The ShotRank technique was applied herein to prioritize analysis of video time frames and associated eye movements. Some automated image processing and dynamic region of interest tracking techniques are also becoming available in eye tracking software packages [38‐40], but remain heavily reliant on researchers manually defining and correcting regions over the course of the video.

Over three million angiograms are performed each year in the United States and European Union [41, 42], with diagnosis and treatment decisions relying on a cardiologist’s interpretation of dynamic angiogram images, yet the underlying interpretive process has not been adequately studied. Thus, this pilot study provided a first examination of the behaviours and eye movements characterizing cardiologists’ interpretation of coronary angiograms, with implications for the development of next-generation educational and training curricula and assessment methods.

Eight physicians with a range of experience in interventional cardiology were recruited, including six residents and two faculty experts.

Overall diagnostic accuracy was high (x̅ = 0.96, s = 0.04). Experts demonstrated higher accuracy (x̅ = 1.0, s = 0) than residents (x̅ = 0.95, s = 0.04); however, this difference did not reach traditional significance levels (U = 11.0, p = 0.086).

Regarding case-level assessment, FFR request, and treatment recommendation, novices and experts showed similar responses. Seven of the eight participants assessed the case as single-vessel CAD, with one novice selecting multivessel CAD. Seven of the eight participants suggested they would not request FFR, with one novice suggesting they would. Finally, seven of the eight participants recommended PCI treatment, with one novice suggesting medical therapy only. Thus, in this small sample, overall accuracy was high, though experts showed significantly higher accuracy than novices when estimating maximum diameter stenosis, and novices showed some variation in overall case assessment and treatment recommendation.

Experts spent significantly less time reviewing the entire case (total review time; x̅ = 132.4s; s = 41.4) relative to novices (x̅ = 309.8s, s = 91.7), as depicted in Fig. 1; this difference reached significance (U < 0.01, p = 0.046). Experts also paused the video to examine individual frames less frequently (x̅ = 3.0, s = 2.8) than novices (x̅ = 45.5, s = 75.9), though this pattern did not reach significance (U = 2.5, p = 0.24).

Frame-by-frame viewing behaviour was assessed for each video to identify frames with consistently high viewing frequency. An example frame-by-frame assessment is included in Fig. 2, for video RAO30 of the left coronary artery. To account for certain frames being increased due to a single participant’s behaviour, a z-score threshold (x̅ + 2SD) was used for identifying critical frames. This process resulted in six critical frames distributed across five videos (RAO30, RAO22/CAU20, RAO10/CRA40, LAO45/CAU23, and LAO38/CRA1; note that RAO10/CRA40 contained two critical frames). These videos and frames were carried forward into the analysis of frame-specific eye movement behaviour.

As expected given the shorter review times, experts had fewer fixations (x̅ = 691, s = 352.1) than novices (x̅ = 1,358.8, s = 512.6); however, when considered relative to review time, experts and novices had a similar number of fixations per second (x̅ = 5.1, s = 1.08; x̅ = 4.3, s = 0.55, respectively; U = 10, p = 0.18). The mean duration of fixations did not differ between experts and novices (U = 5, p = 0.74). For saccades, experts showed fewer saccades (x̅ = 705, s = 370.5) than novices (x̅ = 1,389.8, s = 524.9); however, when considered relative to review time, experts and novices had a similar number of saccades per second (x̅ = 5.1, s = 1.2; x̅ = 4.4, s = 0.55, respectively; U = 10, p = 0.18). Finally, peak saccade velocity was significantly lower in experts (x̅ = 184.5, s = 34.1) than novices (x̅ = 277.9; s = 90.2), U < 0.01, p = 0.046.

For each of the six critical frames identified, an expert interventional cardiologist (author BN) identified a single rectangular diagnostic region; this region of interest (ROI) was determined to be the single most important region of the frame for accurately arriving at the case diagnosis. The expert also annotated the unique diagnostic information inherent to the region.

The video and frame-specific analysis examined differences in fixation and saccade parameters within versus outside the defined ROIs, as a function of cardiologist expertise. Each video and frame are described below (see Tab. 1 for descriptive statistics).

On frame 54, the expert-defined ROI was annotated as follows: This shows an intermediate lesion (30% stenosis) in the mid-LAD. It is important to carefully evaluate this blood vessel. Experts and novices showed a similar number of fixations on frame 54 (U = 4.0, p = 0.51). The number of fixations on the ROI did not differ between the two groups (experts x̅ = 3.0; novices x̅ = 7.3).