Outpatient process quality evaluation and the Hawthorne Effect

Abstract

We examine the evidence that the behavior of clinicians is impacted by the fact that they are being observed by a research team. Data on the quality of care provided by clinicians in Arusha region of Tanzania show a marked fall in quality over time as new patients are consulted. By conducting detailed interviews with patients who consulted both before and after our research team arrived we are able to show strong evidence of the Hawthorne effect. Patient-reported quality is steady before we arrive, rises significantly (by 13 percentage points) at the moment we arrive and then falls steadily thereafter. We show that quality after we arrive begins to look similar to quality before we arrived between the 10th and 15th consultations. Implications for quality measurement and policy are discussed.

Introduction

When researchers record the behavior of individuals there is always a concern that people will alter their behavior if they know they are being observed. Health care research in developing countries is moving from measuring fixed structural characteristics of health facilities to measuring process quality such as the quality of care provided in consultations.¹ However, unlike a leaky roof or missing medical supplies, doctors can alter their behavior when they know they are being observed. Research in developing countries has shown significant variation in consultation quality (Das & Hammer, 2005; Leonard & Masatu, 2005); can researchers have confidence that this measured variation reflects the real variation in quality that would exist in the absence of a research team? Even when doctors are observed in their own environment consulting their regular patients and care is taken to ensure the doctors that they are insulated from any impact of being observed, it is possible that the mere presence of another person can alter their behavior. This paper examines data on the behavior of clinicians in Tanzania that has been used to measure quality and asks whether the presence of a member of the research team changed the behavior of clinicians, showing evidence of the Hawthorne effect.

We examine the use of one instrument for measuring quality (direct consultation observation or DCO), in which the researcher fills a checklist of expected procedures for a clinicians’ regular consultations. Previous work with this instrument has shown a marked fall in quality beginning shortly after the team arrives, consistent with a diminishing Hawthorne effect. We pair this instrument with interviews with patients conducted immediately after the consultation. The interview attempts to fill the same checklist as the DCO instrument based on the patient's recall of the consultation. This second checklist (retrospective consultation review or RCR) is a reasonable measure of the same quality as observed by the research team with DCO. Importantly, we can measure pre-observation quality by interviewing patients who had their doctor consultations before the research team arrived, allowing us to quantify the Hawthorne effect.

The Hawthorne effect refers to a situation in which an individual's behavior is altered by the observation itself. Mayo (1933) documented a temporary increase in employee productivity in response to a change in lighting intensity and other interventions. Because these interventions varied without the intention to increase worker's performance, Mayo argued that the increase in productivity was the result of personal attention and of the newness of a program (Benson, 2000). Both the methodology of the original experiment and the description of the Hawthorne effect have since been called into question (Jones, 1992; Kolata, 1998; Wickstrom & Bendix, 2000), but the original understanding of this effect has survived these debates. The Hawthorne effect has been described in many different settings, primarily education, but including health care (Campbell, Maxey, & Watson, 1995; Ely, Oshero, Chambliss, Ebell, & Rosenbaum, 2005; Gimotty, 2002; Leung, Lam, Lam, To, & Lee, 2003; Verstappen et al., 2004). The Hawthorne effect is characterized by a positive and temporary change in some measurable behavior in a situation where the observer had no intention to truly affect the other individual's behavior. The Hawthorne effect is distinct from an incentive effect in which clinicians alter their behavior because they suspect they may be penalized or rewarded for their behavior. Though the literature suggests that the Hawthorne effect may require a “perceived demand for performance” (Campbell et al., 1995), the pure incentive effect can be differentiated from the Hawthorne effect by the fact that, under the Hawthorne effect, individuals eventually return to their pre-observation level of activity even when they remain under observation.