The Importance of Measuring the Measures

“Preoperative Needle Biopsy as a Potential Quality Measure in Breast Cancer Surgery,” by Pocock et al., is a valuable contribution to our understanding of the role of preoperative needle biopsy as a quality indicator in breast surgery. The report is valuable not only because of the information presented, but also because it demonstrates the process that should be applied to any proposed quality measure before it is implemented in clinical practice: First to develop a hypothesis, then to collect and analyze data, and then return to the hypothesis with corrections or proposed improvements. In this instance, the authors propose that preoperative needle biopsy of any suspicious lesions before breast surgery is a useful quality measure, demonstrate how this can be applied in clinical practice, then use their results to formulate a tentative benchmark. Important practical information about real-world limitations of the measure are documented, and useful proposals are put forth to improve the ability to collect data and measure compliance in the future.

Application of the scientific method to the development and implementation of quality measures is important for several reasons. First, quality measures are not always what they seem. The Patient Safety and Quality Committee of the American Society of Breast Surgeons recently completed a review of more than 40 proposed quality measures in breast surgery, including a measure of time from initial diagnosis of breast cancer to the time of definitive surgical management. The survey included 20 surgeons who reported on more than 60 data elements on up to 20 breast cancer patients each. Because the surgeons included a group of dedicated and highly motivated breast specialists, there was every expectation that the interval from diagnosis to definitive surgery would be quite short. However, this was not the case. In fact, there was a wide range (mean ± SD, 30 ± 15 days) (Lorraine Tafra, personal communication). Surprisingly, most of the “delays” resulted from patient choice. On review, it was clear that measurement of a simple time interval was not reflective of quality; instead, it would have been necessary to measure a combination of appropriate surgical management and patient choice about the timing of care.

Collection of clinical performance data is also critical for development of quality measures, not only to establish benchmarks but also to show that accurate data retrieval is feasible in practice. Although preoperative percutaneous needle biopsy has been recognized as a quality indicator for many years, remarkably little published information is available to document the frequency of needle biopsy compared with open surgical biopsy, or to establish a benchmark for measurement of clinical performance.1,2 The present study, together with the cited publication by Lannin et al., constitute the bulk of the peer-reviewed literature on how frequently needle biopsy can effectively be performed in clinical practice.3

It is also important to collect data about the measure in order to understand how to use the information to ensure quality care. In their mature form, quality measures should not merely serve as a yardstick of performance, but as a mechanism to help improve quality. The authors of the present study propose how other centers can begin to collect information on percutaneous needle biopsy, and they suggest that the measure is important enough to add as a separate data field in national databases. Indeed, the process of measurement alone should begin to improve performance.4 This report, from a comprehensive breast center at a major academic institution, presumably represents the highest level of performance, with expert imagers and surgeons working in collaboration to achieve the best care. However, if a breast center does not meet the proposed benchmark, what are the best corrective measures? A limitation of the present study is that it does not explain how to help other centers achieve such a high level of compliance.

In addition, the current study presents data that are interpreted as reflecting the performance at a treatment center. But isn’t this measure more a reflection of the performance of individual physicians who practice at a center? Whether or not the patient is taken to surgery without first undergoing a needle biopsy is the decision of the individual surgeon working within the context of available local resources. In appreciation of these limitations, the authors themselves recognize that “In order to better assess the effectiveness of this quality measure, it would be necessary to design a study across a range of different institutions and across a broader period of time, extending beyond a year.”

To help clarify some of these issues, the American Society of Breast Surgeons recently announced a pilot program to collect data on three surgeon-controlled quality measures, including percutaneous needle biopsy before open biopsy.5 This program, which is currently open for enrollment, is a voluntary Web-based reporting tool to collect information on the following: (1) percutaneous needle biopsy before open surgery, (2) orientation of the surgical specimen, and (3) verification that an image-guided target has been surgically removed.

Although surgeons will have to register for the program, the data collected are not publicly linked to the individual surgeon, with the goal being simply to obtain information about these measures in a wide range of clinical practice. It consists of simple “yes/no” reporting of the individual quality measures for each open surgical breast procedure, with a pull-down list of reasons if the measure was not performed. With participation of a diverse group of surgeons from different practice environments, the program should obtain information that will eventually allow these measures to be implemented most effectively. The National Consortium of Breast Centers has a similar project that collects data on percutaneous needle biopsy from centers rather than individual surgeons.6 Both programs allow participants to monitor their performance compared to the larger groups, which should itself result in improved performance.

In the present environment of proliferating standards and organizations focusing on quality measurement, the development of quality measures should be based on data that demonstrate not only the efficacy of the measure in assessing quality, but also how the measure can be applied to improve performance. The development of quality measures must be distinguished from the process of defining benchmarks for performance because this requires a more complete understanding of the complex interaction of physician judgment, patient choice, and available resources. In light of demands from insurance plans and the public, it is understandable that there is a perceived urgency to implement any widely recognized quality measures. In an editorial published in 2007, Auerback et al. warned of the dangers of action over evidence in this arena: “The temptation is to circumvent traditional models of evidence when it comes to quality improvement, but this temptation has always existed in medicine for those seeking cures to conditions with high morbidity. Just as in the rest of medicine, we must pursue the solution to quality and safety problems in a way that does not blind us to harms, squander scare resources, or delude us about the effectiveness of our efforts.”7 The present study reminds us that before even the most widely accepted quality measures are used to assess clinical performance, it is important to begin by measuring the measures.