The health disparities cancer collaborative: a case study of practice registry measurement in a quality improvement collaborative

Sixteen community health centers, supported by the Health Resources and Services Administration (HRSA), participated in the HDCC. HRSA directs its resources toward financially, functionally, and culturally vulnerable populations [14]. Basic characteristics of the 16 health centers participating in the HDCC are described in Table 1. The collaborative activities were led and supported by HRSA, the Centers for Disease Control and Prevention, and the National Cancer Institute (NCI).

From 2003 to 2004, the HRSA HDCC administered the BTS, a collaborative model [15] developed by the Institute for Healthcare Improvement (IHI) [16]. The HDCC adapted elements from the 'chronic care model' to improve the quality of cancer screening and follow up. The chronic care model is defined by six elements: healthcare organization, community linkages, self-management support, decision support, delivery system redesign, and clinical information systems [17]. The HDCC's learning model involved three national, in-person sessions and the expectation that local teams would be organized at health centers to pursue PDSA cycles relevant to cancer screening. The 16 centers were selected through an active process that involved telephone interviews with health center leaders to assess their enthusiasm and willingness to commit the resources necessary for success. The local teams consisted of employees with multiple backgrounds and roles, including providers (physicians, physician assistants, and nurse practitioners), nurses, appointment staff, and laboratory and information systems personnel. The effort and staff time allocated averaged four full-time equivalent (FTE) per team with an aggregate of 950 hours per team. Participating health centers reported performance measures to each other and central facilitators, and talked by teleconference monthly.

HDCC measures of screening and follow up for breast, cervical, and colorectal cancer were collected over 15 months in the collaborative (See Additional File 1 for full description of the performance measures). These measures assessed four critical steps in the cancer care process: the proportion of eligible patients screened, the proportion screened receiving notification of results in a timely manner, the proportion of abnormal results evaluated in a timely manner, and the proportion of cancer cases treated in a timely manner [18]. Screening measures were based upon United States Preventive Services Task Force (USPSTF) guidelines and finalized through a process of discussion and group consensus among collaborating health centers. These performance measures were similar to the cancer screening measures developed by the National Committee for Quality Assurance (NCQA) [19] and the Physician Consortium for Performance Improvement, sponsored by the American Medical Association (AMA) [20]. In contrast to other measurement systems, the HDCC did not exclude age-appropriate individuals due to medical reasons or patient refusal (as was done by the Physician Consortium for Performance Improvement). Conversely, other systems did not incorporate timely follow-up (notification, evaluation, or treatment) as part of their indicator sets.

Health centers reported the size of the patient population who were eligible for screening and follow up and received screening and follow up every month from September 2003 through November 2004. Information was reported to HDCC facilitators from HRSA, NCI, and IHI. We obtained Institutional Review Board approval, as well as written consent from each participating health center, to use the self-reported practice registry data.

Community health centers each created a practice registry of individuals eligible for screening or follow up among patients who had been seen in the health center at least one time in the past three years. All health centers participating in the HDCC used the practice registry data software provided by the HDCC; nationwide, HRSA community health centers were encouraged, but not mandated, to use the software. Data entry varied from the wholesale transfer of demographic information from billing data queried for age-appropriate groups to hand entry.

In 2000, HRSA supported the development and deployment of electronic registry software. Over the next five years, HRSA continued to support numerous iterations of the registry software to address both the increasing scope of the collaboratives (such as cancer screening) and the needs of clinicians and other frontline-staff users. Informing this process was an advisory group of health center clinicians and technical experts that provided insight and guidance about critical registry functionalities and the needs of measurement to effectively support practice management. Training in the software was provided by HRSA at a national level, as an adjunct to collaborative learning sessions, and at the regional and local level by the Information System Specialist (ISS). The training typically consisted of four- to eight-hour interactive sessions in which participants would have a 'live' experience on laptops.

The registry software assembled individual patients seen at the health center into an aggregate population to share with other HDCC sites. The data were posted on a secure data repository to be shared with HDCC facilitators and benchmarked against other health centers. A data manager from the medical records department at each center who had training in use of the registry uploaded the data.

The process of entering patients into the practice registry fell into two general categories: a process whereby patients seen at the center in the previous month were entered into the practice registry as they were seen, and a process whereby patients who had been seen at the center before the previous month were entered into the practice registry based on the criterion of being seen at least once in the past three years. The number of patients described as eligible in any given month represented the number of patients that the health center had so far been able to enter into the practice registry. Eligible patients in the practice registry were then searched on the last work day of each month to identify who had received screening or follow up within an appropriate timeframe. The number of patients who were up-to-date with screening or follow up was reported and shared among collaborative participants on a monthly basis; no shared information was identifiable at the patient level.

We anticipated a start-up period of about three months when the practice registry would be in the process of being implemented at the health centers. To test this assumption, we determined the completeness of monthly registry data reported by each health center over the first three months (September 2003 through November 2003) and the last 12 months (December 2003 through November 2004). Within each interval, we determined the proportion of months when data were not reported from each health center (center-months). Preliminary analyses confirmed our initial assumptions: during the first three months of the collaborative, 12.5% of the months over which reporting was possible were absent for screening mammography. For screening Pap test, 10.4% of months were absent; and for colorectal cancer screening, 16.7% were absent. This level of missing data was more than twice as high as was observed during the last 12 months of data reporting (see Results); and consequently, we chose to focus subsequent analyses on the last 12 months of the collaborative. Analyses were performed across 16 health centers over 12 months, thus, data reporting was possible for a total of 192 center-months.

We conducted three primary analyses:

During the 12-month period under evaluation, self-reported practice registry data were available from 16 community health centers for screening mammography in 95%, or 182/192 of the center-months over which reporting was possible. For screening Pap test, data were available for 95% of the center-months, and for colorectal cancer screening, data were available for 94% of the center-months.

All participating health centers reported practice registry data regarding cancer screening (Table 2). The proportion of health centers who reported practice registry data for other care processes were the following across different cancers: documented notification of screening test results (37 to 63%); evaluation of abnormal screening test results (12 to 32%); and delivery of treatment within an adequate time frame after cancer diagnosis (6 to 13%).

The HDCC as a whole had large enough numbers of women and men eligible for screening mammography, screening Pap test, and colorectal cancer screening to detect a change of 2% to 3% in cancer screening (Table 3). Likewise, the numbers of individuals who received breast, cervical, and colorectal cancer screening tests were large enough to detect a 3% to 6% change in the documented notification of each screening test result within 30 days. The numbers eligible were such that only a 15% to 24% change could be detected in the additional evaluation of abnormal screening test results, and only a change of 40% or more could be detected in the delivery of treatment within an adequate time frame after cancer diagnosis.

As the HDCC progressed to healthcare processes more distal to the initial screening event, the number of health centers reporting practice registry data decreased, and the size of the detectable change increased. In the HDCC, reporting practice registry data on the follow up of abnormal results and treatment of cancer was voluntary. Both the small number of events reported, and centers that reported them, commonly made it infeasible to test for statistically significant changes in follow up or treatment, even over the entire collaborative. The small number of abnormal screening results reported--and the even smaller number of cancer diagnoses--have at least three primary explanations: the frequency of these care processes or events was indeed small; the medical information was available in a local medical record but the health centers did not report these events in automated form to the HDCC program, even when they did occur; and health centers did not have routine access to the medical information necessary to report the measures because the care occurs outside their practice.

At any single health center, it is possible that no cancers were detected during the period of time under evaluation (about 3 in 1,000 screening mammograms detect a breast cancer [23]), but it seems very unlikely that any given health center would not have any abnormal results to report (approximately 1 in 10 screening mammograms are abnormal [24]). Because all health centers were not reporting all data describing each cancer care process, selection bias clearly threatens the validity of general inferences drawn from the data collected in the overall collaborative.

As demonstrated by example in the case of the HDCC, a larger number of eligible patients allows more precise measurement of practice performance [6]. A primary care population usually has enough individuals eligible for cancer screening so that multiple health centers--joined together by a collaborative--have sufficient power to detect small changes in screening. Of the screening follow-up steps reviewed, the highest percentage of health centers reported timely notification of Pap test results (62.5%), most likely because these services were performed onsite at the health centers. Yet overall, the same level of precision and power possible for screening was not possible for the measures and comparisons of diagnostic follow-up or treatment events. Therefore, health centers in the HDCC may have felt less accountable for reporting care processes that occurred infrequently knowing the limitations of measuring these clinical processes [25].

Health centers may have had concerns about how misascertainment of only a few cases could potentially make their overall performance appear much worse. Concerns about negative perceptions have allegedly driven reporting behavior in other settings. For example, health maintenance organizations were more likely to withdraw from voluntary Healthcare Effectiveness Data and Information Set (HEDIS) measure disclosure when their quality performance was low [26]. Reinforced by concerns about the potential negative perceptions of their employees or other health centers, participating health centers may have chosen not to invest their limited time and resources into reporting voluntary measures with few events.

The limited ability of the HDCC to detect changes in additional evaluation or treatment also was a function of the clinical setting in which HDCC measurement took place--community health centers delivering primary care. Compared to the number of abnormal tests identified in a primary care practice, more abnormal tests will be found in procedural settings (e.g., mammography centers and gastroenterology practices) where these tests are performed across multiple, referring primary care practices. Similarly, more cancer diagnoses will be found where cancer patients are treated (e.g., oncology and surgery practices) from multiple, referring practices.

On a practical level, primary care health centers may simply not have routine access to the medical record data necessary to report information related to diagnostic follow-up and treatment. There was no uniform workflow for data from institutions outside the HDCC, and we suspect that the lack of access to data outside the primary care practices contributes most to the small number of abnormal screening results and cancer diagnoses reported. Health policy experts have emphasized that single-practice data systems are insufficient for effective care coordination across practices [27]. For example, the extremely low rate of timely treatment after cancer diagnoses among reporting health centers (3 to 24%) very likely represented the lack of a systematic way to collect feedback from oncology practices rather than quality gaps; data across practices is very difficult to locate outside the context of integrated data and delivery systems. Health centers appeared to report what little information was available regarding follow-up and treatment and shift their focus to cancer screening. In the subsequent HDCC regional collaborative, substantial emphasis was placed upon building communities of practice to help address the lack of coordination between primary care and subspecialty practices [28]. Community health centers may perceive it as unfair to hold primary care practices accountable for whether or not their referral was evaluated or treated in a timely fashion given that the clinical delivery (and financial benefit) of these services falls within the scope of other practices in the healthcare system. In the HDCC, this perception may have further contributed to non-reporting of such distal events, even though on a system level, appropriate and timely follow-up is essential for a successful cancer screening program.

In assigning accountability for performance, one general approach is that any individual provider is held accountable for those activities directly under his/her control. This approach is taken in measurement systems supported by the AMA's Physician Consortium for Performance Improvement for physician office practices. An alternative approach to assigning accountability would be the integration of performance measurement across multiple healthcare settings to capture services across the full continuum of cancer care. Here the accountability is placed upon the healthcare system--or a network of physicians--as opposed to individual providers. Building upon our HDCC experience, policymakers may want to consider new methods to identify and reward the team of providers responsible for the care of patients with complex medical conditions [29, 30], including cancer. Policymakers may also want to consider our findings as reinforcing evidence of the need for patient-centered medical homes [31], if they make additional resources available for coordinating care with other providers and using data systems to track referrals and results.