Tips for Teachers of Evidence-based Medicine: Making Sense of Decision Analysis Using a Decision Tree

This tip is suitable for all clinicians and clinical trainees who are encountering a decision analysis for the first time. The exercise takes 30 minutes and has the following specific goals:

Prior to the session, prepare handouts including Tables 1 and 2, and Figures 1 and 3. Have a blackboard or flip chart available which you will use to first develop and then complete Figure 1, using the information from the Tables. The use of these tips will generally require a planned session, although the presentation of the basic concept of a decision tree might be attempted ‘on the fly.’

Begin by reading the clinical scenario about the construction worker outlined above. Tell the learners that the starting point in a decision analysis is a decision that must be made on behalf of an individual, groups of persons, or population.1 Ask the learners what the main clinical problem is (fractured ankle), the alternative or choice that has to be made (debridement and antibiotics or below the knee amputation), and the outcomes (infection not controlled, full recovery, partial recovery, death, above the knee amputation and below the knee amputation). This helps learners specify the decision problem and its components, which is the first step of a decision analysis. Some teachers may choose to list the problem, alternatives and outcomes on a whiteboard.

The next step is to develop a model to portray the time course of the different options. Explain to learners that the easiest way to do this is to draw a decision tree (Fig. 1), which is a graphical tool to show all the components of the problem and to relate all choices to consequences.1 You begin at the left-hand side of the screen by creating a decision node (□) with the alternatives ‘antibiotics and debridement’ (“antibiotics”) or ‘below the knee amputation’ (“BKA”). If “antibiotics” is chosen, there are two possible outcomes: the foot is saved or infection spreads. These two possible immediate outcomes (“Foot saved” and “Infection not controlled”) emanate from a chance node (○). Explain to learners that a chance node represents a point in time at which probability determines which of several possible events that may occur and is outside the control of the decision maker.1 Following the chance node at the “Foot saved” branch, there are two possible outcomes: full recovery or partial recovery with a limp. If the infection has spread, the patient may die, need an above knee amputation or below the knee amputation. An outcome at a particular pathway is known as a terminal node (◁) indicating the end point of each sequence of events. At this point, emphasize to the learners that you construct a decision tree from the left to right, starting with the initial decision node on the extreme left and moving to the final outcomes on the extreme right, following the temporal sequence of events. When you finish, the skeleton of the decision tree should look like Figure 1.

Following this, explain that the next steps in the construction of the decision tree are to estimate the probabilities of the various outcomes and the patient values to be assigned to the consequences (also known as utilities). For the purpose of the exercise, tell the learners that there is no published randomized controlled trial that compares these two approaches in patients with infected compound fracture of the ankle. However, there are some published data on what happens to patients if one or the other approach is used. You can explain that, although high quality evidence should be sought, when all else fails, probabilities of chance events can be derived from expert opinion, including your own. Present Table 1 to the learners and ask them to insert the relevant probabilities into the tree you have created. Ask for a volunteer to go to the board to demonstrate the result. At this point, highlight the rule that the probabilities under the successive branches emanating from any chance node must add up to 1.00.

Address the issue of how important the various levels of disability being considered are to the patient. You may choose to incorporate statements regarding patient values and preferences into the scenario, or to use published studies when they exist. You may also choose to engage your learners in a discussion regarding how they would themselves go about discerning and understanding the patient’s values if he were their own patient. One or more of them may likely propose to simply ask their patient “would you rather risk death than not have any chance at full recovery (without amputation).” Another learner may point out that this choice does not conform to the options reflected in the tree you have created and may further point out that the actual choice has to do with BKA now or take a chance that may lead to death, full recovery or something in between. At this point, the learners characteristically appreciate how the analytical approach of creating the tree helps guide both a clinician and their patient to an understanding of the nature of the choices before them.

Now draw your learners’ attention to Table 2 and explain that the “utilities” attached to each of the possible outcomes in Figure 1 constitute proportionate patient valuations of the quality of life attached to these outcomes, with a value of 0 assigned to the outcome of death and a value of 1 assigned to the outcome of full recovery with no amputation. Have a learner write utility values that the group believes are likely to reflect the values of their own patients on the figure on the board. In this process, learners see how a decision tree evolves from an initial identification of therapeutic options, together with a set of estimates of probability of the potential consequences at every point and an independent estimate of relative patient valuations of the different possible outcomes. When you finish, the decision analysis tree should look like Figure 2A.

Now you are ready to help your learners analyze the decision tree. This can be done by explaining to the learners that the main objective of constructing a decision tree is to enable you to estimate the “expected value” of the different therapeutic choices considered at the outset. “Expected value” (EV) is a term used to denote the overall value of each of those choices in a fashion that takes into account patient preferences and priorities. Another term that is used for the same thing is “expected utility”.4 EV is a way of expressing in numbers which alternative would provide the highest value for the patient.1 The process of calculating the expected value is “averaging out and folding back” and starts at the right-hand side of the tree and works back to the root (left-hand side) of the tree.1 Explain to the learners how the EVs at each chance node are calculated in a stepwise fashion along the lines of the legend to Figure 2B and demonstrate the process through one or two steps. Then invite the learners to complete the process. Learners rapidly understand how the final calculation of the expected values of the two alternatives is estimated. Compare the expected value for antibiotics and debridement approach (0.773) with below the knee amputation (0.70).

The second tip works best when presented after Tip 1. The exercise takes 20 minutes and has the following specific goal:

Refer to Figure 2B as completed in the first part of the exercise. What would happen to our decision if we changed some of our values or probabilities where the data were weakest and where we were most unsure about those values? Is the decision altered if different probabilities and value consequences were assigned in the decision tree? Ask each learner to independently consider different probabilities and patient values from those listed in Tables 1 and 2, based on their intuition. Compare the learner responses and identify the 1 or 2 variables reflecting the greatest level of uncertainty. Ask the learners how they would handle the uncertainty. After a moment, a participant offers the suggestion that you could reanalyze the decision tree using a different value for the variable with the greatest level of uncertainty and see if this changes the expected values previously derived for the alternatives. Compliment this participant and then offer that she has just invented the concept of a “sensitivity analysis”. You may choose to interpolate a didactic explanation at this point: “A sensitivity analysis is the process of repeatedly folding back the decision tree using different values for the probabilities of outcomes and relative patient values to determine whether the implied decision would change as a result.”

To help learners understand how to interpret a sensitivity analysis, begin by assigning different probabilities of the foot ultimately being saved (“Food saved”) to each learner by varying the range from 0.20 to 0.80. Ask each learner to reanalyze their decision tree using their assigned probability to come up with the expected values for “Antibiotics” and “BKA” alternatives using the process outlined in the last tip. Then plot the results from each learner. When finished, the results should look like Figure 3, a one-way sensitivity analysis. Now ask the group to interpret the graph. Facilitate the process by asking the group to focus on where the lines cross (i.e. “threshold”). The learners will readily identify that the threshold occurs when the probability of “Foot saved” is at 0.34, at which point the decision changes. Remind learners that the higher expected values of the two alternatives provide the better alternative for the patient. Learners will notice that if the chance of antibiotics saving the foot is less than 34%, the better option is below knee amputation; and that when the chance of antibiotics saving the foot is more than 34%, the better option is to use antibiotics and debridement. Finally, tell the learners that multiple probabilities and relative patient values can be varied simultaneously to identify the most important variable(s) that may change a decision.

How can decision analysis be applied to individual patients in real-time patient care? Proper use of evidence-based medicine involves more than identifying the best available evidence and applying the results to your patient. This is because patients have individual characteristics that make them different from the “average” patient in a clinical trial. Patients may value the effect of treatment and side-effects differently, and this different view of outcomes or value consequences needs to be taken into account. Also, the results of a trial will be reported as an average probability of effect, and this probability may need to be adjusted for your particular patient’s characteristics. The best way of taking account of all these adjustments is through the use of decision analysis. For common clinical problems, shared decision making between clinician and patient is possible when patient preferences are inserted into a generic decision tree with set probabilities during real-time patient care using computer programs (assuming that patients are comfortable and are able to state their preferences on a utility scale ranging from 0 to 100).

We field tested the teaching tips to verify the clarity and practicality of the teaching script. One of the authors (S.K.), an experienced teacher of evidence-based practice who was not involved in the development of the teaching script, field tested the script used in this article on two occasions once in Durham, North Carolina and once in Miami, Florida. Learners were residents in internal medicine from interns to third-year residents. Regardless of the level of training, all learners in both settings reported that they had little to no experience with decision analysis. Despite this, in both sessions, learners were easily engaged in the interactive creation of the decision tree in Tip 1. As described in the script, starting with the case scenario, the learners easily identified the main problem, the choices to be made and then possible outcomes. These were written on a white board and served as the basis for the decision tree that was drawn interactively using the choices and outcomes that the learners identified. After generating the decision tree, the learners easily went through the steps of incorporating the probabilities and values provided in the tables onto a pre-prepared handout of the decision tree without probabilities or utilities (Figure 1). Initially, the learners were broken into groups of three to discuss the case and generate their own utilities for each outcome. These were then compared to other groups as well as to the utilities in Table 2. Remarkably, all groups of learners came up with similar utilities closely matching those in Table 2. The learners were also able to “fold back” and calculate the expected values following a demonstration of the first calculation. The completion and calculation of the decision tree (Tip 1) took 30 minutes to complete. Discussion of sensitivity analysis (Tip 2) flowed directly from Tip 1 as the learners asked questions expressing concern about the “estimated” probabilities and values and took 20 minutes.

Learners uniformly reported that the scenarios, tables and set up were extremely effective in allowing them to complete an exercise in an unfamiliar arena. They valued the clinical relevance of the scenario. Learners reported that the most important points they took away from the session pertained to the importance of incorporating values into the decision-making process, as well as the role of uncertainty in decision-making, a concept made clear through Tip 2 and the sensitivity analysis. The first time the session was taught, a non-medical example was used for Tip 1. Many learners reported that they did not see how they would use these principles in their practice of medicine. For the second field-test, the non medical example was replaced by a clinical scenario, and SK included a greater emphasis on relevance. She brought an example of a cost-effectiveness analysis from the current medical literature which relied heavily on the principles of decision analysis and also discussed how the decision analysis framework could be used to illustrate the general principles in every day decision making. Notably, in the feedback from the second group of learners several reported that the most important take home message for them was how they would apply these principles to every day decision making.

For most educators, this is a session that will require deliberate preparation. SK spent 3 hours preparing for the session the first time it was taught and another 90 minutes preparing for the second teaching. Handouts were pre-prepared to provide the “skeleton” of the decision analysis so that learners could label the probabilities and utilities themselves as they worked through each step. In order to avoid difficulties of navigating the math calculations in real time, SK brought a calculator to each session and assigned one of the learners to serve as the “math checker” to do calculations so that the learners (and teacher) could focus on the principles and flow of the session.

In a randomized controlled study of final year undergraduate medical students at The Chinese University of Hong Kong, the students’ clinical decision analysis skills under simulated conditions improved.5 This was attributed partly to these tips. This trial took place outside of the context of the EBM Teaching Tips project.