Methodologies for evaluating strategies to reduce diagnostic error: report from the research summit at the 7th International Diagnostic Error in Medicine Conference

Logic models

A key aspect that the group identified as lacking from many studies on diagnostic error, as with other evaluations of patient safety practices, was a logic model or logical framework underlying the solution [23, 24]. A logic model should clearly define the problem to be addressed, the intervention to be evaluated, and most importantly, why the proposed intervention would be expected to work. Development of solutions within a logic model or logical framework is essential for determining whether failure of a given solution has occurred due to the underlying idea or the implementation of that idea.

In a logic model, the main idea is the mechanism of action, preferably informed by theory, about why the solution is expected to cause an improvement in diagnostic error. For example, if the intervention includes just-in-time information presented to the clinician in an electronic medical record that highlights examination findings contradictory to the final diagnosis, a logic model would describe the cognitive biases that might be corrected and the clinician behavior change that might be provoked by this additional information provided in this way. Further, the logic model could hypothesize counterproductive contextual pressures (e.g. “alert fatigue”, poorly designed presentation of the information) that could inactivate the expected mechanism of action. Clear logical frameworks can help teams identify appropriate intermediate measures during the design phase that can be used to troubleshoot interventions early during implementation. In addition, because most solutions to diagnostic error have been applied to very specific diagnostic situations, a clear statement of the logical frameworks that underlie specific interventions will likely help to classify solutions into taxonomies. Such classifications could enable generalizing across different diagnoses and medical specialties to identify generic solutions in some instances.

For example, the Society to Improve Diagnosis in Medicine has released a “Patient Toolkit for Diagnosis”, a structured series of questions and prompts that helps patients to record pertinent medical information before, during, and after a clinical encounter [25]. The use of the tool is hypothesized to reduce diagnostic errors, but what is the logical basis for this theory? One way to express the supporting logic model would be a table showing the logical connections between intervention elements (e.g. provision of the toolkit to patients, training nurses to discuss the toolkit with patients to uncover useful diagnostic clues, and physician training about best practices to incorporate patient-generated pre-appointment information into diagnostic workflow) and the intermediary effects hypothesized to lead to a reduction in diagnostic error (e.g. patients adhere to follow-up testing more often, more relevant patient information is recorded by nurses, and doctors generate broader differential diagnoses). For each intermediary effect, a scientific basis is included with ways to measure it. In this example, the evaluation team needs to ask a number of questions: Is there evidence that patients fill in forms before appointments? Do nurses seek out the information that patients have recorded in advance of the appointment? What do we know about the amount of information presented and the length of the differential list? Does a longer differential list increase the likelihood of making a correct diagnosis? Are patients who have engaged in documentation more likely to adhere to physician recommendations? The keys to this process are seeking evidence that supports or contradicts each logical connection and iteratively using previous evaluations to generate additional measurement options. Specifying the logic model and measuring the intermediary effects helps troubleshoot interventions when they do not work as hoped.

Prototyping and pilot testing

The second recommendation of the group was to apply engineering principles such as rapid, iterative prototyping and simulation during the development of solutions for diagnostic error [26]. Use of these approaches can prevent one from carrying an ineffective solution into a real-world production environment where it might be met with dissatisfaction by busy clinicians, inconvenienced patients, and result in questionable efficacy, when more design and testing upfront may have led to a clear need for alternative strategies. While these strategies are important in developing sustainable solutions, the group acknowledges that funding for these approaches is currently limited because the majority of funders tend to favor traditional scientific approaches. The onus is on investigators to justify the advantage of a prototyping approach for diagnostic error solutions.

Study designs less likely to be useful

While post-diagnosis retrospective epidemiological studies can be important for suggesting solutions to diagnostic error, they are seriously limited by the problem of hindsight bias. This bias results from the human body’s limited repertoire of generally non-specific “real” symptoms (e.g. pain, fever, cough, etc.) combined with “red-herring” symptoms (from heightened sensitivity to normal physiology, unrelated chronic disease, simultaneous conditions, or even embellishment in order to impart gravity to the situation that reflects the anxiety they are experiencing related to the lack of a diagnosis or fear of a misdiagnosis). The expression of disease, in turn, leads to: (1) the Bayesian nature of the diagnosis (i.e. although pneumonia is always characterized by fever and cough, knowledge of the probability of pneumonia when a patient presents with fever and cough is what matters for diagnosis) and (2) the low signal-to-noise ratio that is particularly an issue for rare diseases that present with non-specific symptoms. For example, only a tiny proportion of patients with complaints of diarrhea have celiac disease and many have no disease at all; instead they simply fall into a tail of the distribution of bowel movements among normal individuals. Testing everyone with anti-gliadin antibodies to uncover the small proportion of patients with celiac disease has enormous costs and will generate numerous false positives. Yet, if patients and physicians have zero tolerance for missing a case of celiac disease, we may succumb to this faulty strategy [27].

Pre-post intervention designs where a single assessment of the outcome is evaluated before and after the intervention are also problematic because they are subject to secular trends and other confounding factors that can explain the observed difference between the outcome before and after the solution is applied [28].

Recommended study designs

The group unsurprisingly endorsed randomized clinical trials as the ideal design when resources allow and equipoise exists (i.e. genuine uncertainty about the benefit of the intervention). Given the complexity of most approaches to decrease diagnostic error and to avoid crossover effects, cluster-randomization across multiple sites will often be the most appropriate study design.

The next tier of designs includes the stepped wedge and interrupted time series. A stepped wedge design involves the sequential roll-out of an intervention to a group of participants (at the individual or cluster level) in random order over a number of time periods [29]. An interrupted time series design measures the outcome at several intervals before and after the intervention and analysis focuses on whether there is a clear “interruption,” with the trend changing at the time of the intervention from the baseline trend of the data [30]. Unlike a pre-post design, both of these designs permit secular trends to be disentangled from the effects of the intervention, and both designs are especially valuable when a randomized clinical trial is infeasible or unethical because of substantial data supporting the benefit of the intervention.

Future advances in methodology will likely involve the use of advanced modeling techniques and methods, such as g-estimation and dynamic simulation, to deal with time-varying confounding and to evaluate complex diagnostic pathways [31, 32]. These complex analyses will require the assistance of experts in these advanced methods. Indeed, the group recommends leveraging best practices (e.g. rigorous and established criteria for diagnostic test evaluation, cost-effectiveness, etc.) and engaging knowledge experts from other domains (epidemiologists, cognitive scientists, patients, health economists, etc.) where they are appropriate to the research question.

Unique challenges in addressing research biases in diagnostic error solutions research

The group acknowledged the unwanted role that various types of bias may play in studies of diagnostic error reduction strategies. Because there is currently inadequate consensus around definitions of diagnostic error, it is critical that within the context of a study that all parties adjudicating whether an error has occurred are working with the same unambiguous definition. Another particular bias relevant to studies of diagnostic error solutions is the improvement in performance that generally occurs when people are observed, the so-called Hawthorne effect [33], which could lead to problems if subjects are more closely observed during a period of intervention than during a control period (or vice versa).

Although the group favors the experimental and quasi-experimental designs discussed in the section above because they are less prone to confounding than observational designs, many diagnostic error studies still require observational designs for hypothesis generation and for practical reasons. Confounding occurs when a (biased) association is observed between the intervention (or other exposure) and the outcome that arises from a common cause of the exposure and outcome. For example, if patients with a greater severity of illness are more likely to receive the intervention and are also more likely to be diagnosed with the disease by the reference standard, then they may have a worse outcome unrelated to the intervention (i.e. the association between the intervention and worse outcome is confounded by severity of disease). Controlling for confounders is important to avoid bias, but the strength and even existence of most confounders is unknown. Although there is controversy regarding their ability to fully compensate for confounding in observational designs [34], propensity scores are one technique that can be used to adjust for confounding resulting from the likelihood the patient will have the disease, the patient’s severity of disease, or the clinician’s threshold for diagnosis which are likely to be important sources of bias in diagnostic error solution research [35].