The Medicare Health Outcomes Survey program: Overview, context, and near-term prospects

On 22 July 2004 this article replaced the incorrect version that had originally been published. The differences are minor and of no consequence to the data or conclusions.

Medicare managed care plans are an important source of health care services for beneficiaries. At present, 5.3 million beneficiaries receive care in these settings, of whom 4.6 million are enrolled in Medicare+Choice plans (since renamed Medicare Advantage plans, effective January 1, 2006). The number enrolled in Medicare Advantage plans is projected to increase to 13.6 million by 2010, given the recent passage of the Medicare Prescription Drug, Improvement and Modernization Act [1]. Thus, an estimated 30 percent of Medicare beneficiaries will receive care in managed care settings by 2010. The quality of care provided in managed care settings is a critical and growing issue.

A managed care organization's ability to provide quality health care is particularly important to the Medicare program. Medicare provides services to 34.6 million elderly and six million younger beneficiaries with disabilities [1]. A substantial share of beneficiaries are poor, with 40 percent having incomes of 200 percent of the federal poverty level or lower. Chronic conditions are prevalent, with 57 percent of non-institutionalized beneficiaries self reporting arthritis, 55 percent reporting hypertension and 37 percent reporting heart disease [2]. More than one-third need assistance with at least one activity of daily living [3], while 28 percent self reports their health status as fair or poor [2]. Thus, Medicare enrollees are often those vulnerable to problems with quality in managed care settings [4‐7].

CMS's current vision involves enhancing "quality and efficiency in an evolving health care system" [8]. Some of CMS's specific goals include securing and enhancing health and satisfaction, creating a system of high quality care, and providing guidance for the overall health care market to improve beneficiaries' health [8]. CMS has made considerable progress in becoming a value-based purchaser of health care by pursuing high quality care for beneficiaries at a reasonable cost. Value-based purchasing includes several strategies directed at improving the quality of care, encouraging the efficient use of resources, and improving information to beneficiaries to assist them in making choices. Performance measurement is a critical component for all these purchasing strategies.

In 1996, the Centers for Medicare & Medicaid Services initiated the Medicare HOS program. The Medicare HOS is a survey that measures a health plan's ability to maintain or improve the physical and mental health of its Medicare beneficiaries over time. It is the first national survey to measure the health related quality of life and functional health status of Medicare beneficiaries enrolled in managed care [9]. The HOS consists of the SF-36^®, questions concerning Activities of Daily Living (ADLs)³, chronic health conditions, demographics, and survey administration. The use of the HOS to collect health outcomes data mirrors a national trend in quality improvement activities to utilize outcome-based measures of quality. The goal of the HOS has been to gather valid and reliable health status data in Medicare managed care for use in quality improvement activities, public reporting, plan accountability and improving health outcomes based on competition [10]. The HOS has the potential of becoming a model of government leadership in our nation's quest to improve the quality of health care for all Americans.

CMS has sponsored a program evaluation of the HOS, a CAHPS®-HOS Integration study, and analytical studies of the HOS data. The results of these studies will provide CMS the information it needs to improve the future administration of HOS. The current evaluation of HOS, which will conclude in Fall 2004, includes three components: 1) a description of the context of HOS development, 2) an examination of the HOS instrument and operational protocols, and 3) an assessment of the policy issues concerning the utility of HOS data. The contextual analysis presented here is one component of the total evaluation.

This paper explores the historical context that led to the development and implementation of the HOS program; it also describes the HOS and examines its administration, its role and function within CMS, and the direction that the program may take in the future. The information for this paper was obtained through a review of the published health services literature, national reports, unpublished CMS documents and reports, and interviews with key CMS staff members (current and former), HOS partners, and Technical Expert Panel (TEP) members.

Quality concerns, during the late 1980s and early 1990s, led to heightened interest in establishing effective mechanisms for measuring and, ultimately, ensuring quality in health care. Following Donabedian's widely recognized paradigm (1988), a variety of methods for defining and measuring health care quality have evolved [11]. From the use of process and structural measures of care, to the use of more quality of life and functional health outcomes-based measures, evaluators have sought to develop and implement the most effective methods for assessing health care quality and performance.

During the time of HOS development, there was an increased interest in measuring outcomes of health care. The use of outcome measures in quality improvement efforts stems, in part, from a desire to focus on the impact of care on patients [12]. Berwick (2002) asserts that the experience of patients should be the ultimate source of defining quality. Berwick views the purpose of the U.S. health care system as enhancing the functioning and health status of citizens by decreasing the burden resulting from disease, injury and disability [13]. Rogers (2003) notes that one of the motivations to pursue functional status assessment from the beginning was to try to inject awareness of outcomes into the medical system. There was an interest in moving the focus from "treating diseases" to "treating patients. [14]"

Traditional health outcome measures have included physiologic measures as well as mortality. More recently, a variety of tools to measure additional dimensions of health and well being of importance to individuals have been developed [15][16]. As described by Bierman and colleagues, "health status measures assess the net effect of one or all health problems and treatments on multiple domains of health" [15]. Those health status measures that are generic should consider function across a number of domains as well as be applicable to various health states [15].

CMS like other large purchasers wanted to establish a manner in which to measure the quality that managed care organizations were providing their beneficiaries. According to the 1998 President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry Report, a key element of improving health care quality is the Nation's ability to measure the quality of care and provide easily understood, comparable information on the performance of the industry [17].

Many process and clinical care performance measures existed in the 1990s; however, noticeably lacking were appropriate measures of health outcomes. The Office of the Inspector General (1997), as well as Bailit (1997a, b) released reports encouraging CMS to adopt an outcomes-based performance assessment model that emphasizes the use of outcome-oriented performance data [18‐20]. It was felt that an emphasis on the use of outcomes-oriented performance data would be consistent with the goals of value-based purchasing [21]. Stevic commented, "In the health care industry, everyone was frustrated with measurement. We had been doing measurement for a long time....we were doing a lot of process measures and no one was satisfied that we had the right information....[22]" According to Jencks, "CMS needed to develop an appropriate health outcome measure because one did not already exist and there was not likely to be one developed by someone other than CMS in the near term" [23].

In 1996, CMS decided to partner with NCQA to develop a functional health status assessment tool to be used as a health plan performance measure for Medicare beneficiaries. The Medicare HOS was created to provide health outcomes data as a performance measure of Medicare managed care plans. To justify the emphasis on Medicare enrollees in managed care, NCQA representatives indicated that "the focus has been on managed care because the accountability movement started with managed care. Managed care was promoting itself as an accountable entity that would lead to better care outcomes and the government wanted to show that managed care was valuable..... Also, people thought managed care would grow in Medicare" [9].

According to Kang, functional health status was selected as CMS's outcome measure of choice because it was of greatest interest to Medicare beneficiaries [24]. Paul stated that "the decision to use functional health status was the right one [25]. It reflects the agency's strategic objective of being patient-centered" [25]. Additionally, there was evidence that complex issues such as chronic illness, co-morbidity, and functional impairment are particular problems among the elderly and present distinct challenges to this population. It was believed that a functional status measure was an important outcome of care, due to the complexities involved in the health of older adults, and because of the growing concern about how managed care would affect these conditions and this population's subsequent functional status [15].

Haffer maintains that, "During the Clinton administration, the agency thought of itself as a beneficiary-centered, value-based purchaser of services, and tried to do things to measure the value we [CMS] were getting for the services we [CMS] were purchasing on behalf of the beneficiaries. These performance measures filled a need because we [CMS] could now look at what we [CMS] were spending and compare plans based on standardized, scientifically sound performance measures. [28]" Bierman et al. (2001) said that quality of care is a complex concept and measures such as the HOS examine an important dimension of quality of care for Medicare beneficiaries [15].

CMS established a technical expert panel (TEP) to guide the development, implementation and operations of the HOS [8]. The TEP recommended that the HOS become a component of Medicare HEDIS^® and that the SF-36^® be used as the foundation of the measure. The SF-36^®, a physical and mental health status measure, was used in the Medical Outcomes Study (MOS) [7] and has a history of use in estimating relative disease burden for numerous conditions [27]. The SF-36^® has been used in both specific populations such as the elderly and disabled, as well as in the general population [10]. ]. The tool has been used to determine the effectiveness of various treatments such as hip and knee replacement and heart valve surgery [28][29], as well as the burden of various illnesses such as cancer, diabetes, and HIV/AIDS on populations [27]. Commenting on the reason the SF-36^® was chosen for the HOS, Ware [30] noted that the SF-36^® was the most widely used health status measure at the time [of HOS development]...and was found to be reliable and valid for use with the elderly and other demographic groups. The SF-36^® had good psychometric properties and had been peer-reviewed. Additionally, both a user's manual and normative SF-36^® data were available.

Although the most widely used generic health status measure at the time of developing the HOS instrument, debate existed about its use with elderly individuals [31‐33], As discussed by McHorney (1996), there are "...practical and psychometric issues that can differentially affect the performance of generic health status tools in cross-sectional and longitudinal studies of elderly persons, including mode-of-administration effects, floor and ceiling effects, and score stability versus internal consistency" [16]. Two additional issues are relevant for longitudinal studies – selective mortality and proxy reports [16]. Beirman and colleagues (2001) raise similar concerns related to administration and survey response [15]. For example, while the SF-36^® takes approximately ten minutes to complete, it likely takes longer for participants with less than a high school education [16], older participants [6] and participants with cognitive impairment [34]. However, the previously mentioned issues are not specific to the SF-36^®, and does not seem to rise to the level unacceptability for use with a geriatric population [11][31][33]. These issues will be important to address in the broader HOS evaluation, discussed below.

CMS, responsible for leadership, oversight, coordination, and successful implementation of the national Medicare HOS Program (formerly the Health of Seniors Survey), formed partnerships with organizations that had skills and experience in the areas of health outcomes and performance measurement. The team of HOS partners and technical experts are described in Tables 1 and 2.

The HOS is based on a longitudinal cohort research design, in which a baseline and 2-year follow-up surveys are administered to a sample of beneficiaries in each plan. The survey is primarily a mailed questionnaire that is sent to selected participants, with a series of reminder postcards and telephone interviews among non-respondents and incomplete survey respondents in efforts to increase the survey response rate⁴. The HOS survey instrument consists of three primary components: 1) the SF-36^®, 2) case-mix and risk-adjustment questions, and 3) demographic and other questions required by the 1997 Balanced Budget Act [9]. The SF-36^®, as previously noted, is a multipurpose short-form general health survey, which provides eight scale scores of physical and mental health attributes, as well as two summary measures of physical and mental health status, the Physical Component Summary (PCS) and Mental Component Summary (MCS), both of which are weighted combinations of all eight scale scores [35] (Table 3).

The responses to the SF-36^® are scored as eight Likert scales, summarized using standard weights, and then normed using 1998 US general population data to produce the PCS and MCS scales (see both the SF-36^® Health Survey Manual and Interpretation Guide and SF-36^® Physical and Mental Health Summary Scales: A Manual for Users of Version 1 for details) [40][35]. Each beneficiary who remains in the health plan in which he or she was originally sampled is measured twice, at baseline and again at follow-up; thus, the beneficiary serves as his/her own control. All respondents who die during the two-year follow-up period also are included in the follow-up analysis. Because there are many differences among individual survey respondents based on age and initial health status, for example, the mortality results and PCS and MCS two-year change scores are risk-adjusted⁵. Thus, plan-to-plan comparisons of health outcomes are adjusted for differences in case-mix, using available data on characteristics such as chronic conditions, age, ethnicity, gender, and education, among others. Logistic regression models which incorporate techniques for missing data estimation are used for risk-adjustment; the change in two-year health status is determined as: better than expected, same as expected⁶, or worse than expected [10]. The a priori primary outcomes were "alive and PCS same or better than expected" and "MCS same or better than expected".

The HOS survey is administered to a random sample of 1,000 Medicare beneficiaries from each M+CO at baseline⁷ [10]. Beneficiaries are deemed eligible if they were continuously enrolled for at least 6 months in the same plan and do not have End Stage Renal Disease (ESRD). The survey is administered at baseline, and again two years later; a new cohort is sampled each year [41].

Plans contract with an NCQA-certified vendor to conduct the survey once the sample has been selected and approved by CMS through its contractor, Research Triangle Institute (RTI) International, a subcontractor to NCQA.. Vendors receive HOS survey administration training annually from NCQA. Vendors then administer the surveys in accordance with the applicable NCQA HEDIS^® protocol. Once the survey data have been collected, they are submitted to NCQA for consistency review. The data are then submitted to Health Services Advisory Group (HSAG) for cleaning, aggregation, and analysis. Following data cleaning, aggregation, and analysis, HSAG develops and disseminates data files and reports to CMS, Quality Improvement Organizations (QIOs) and the health plans⁸ [10].

From 1998-2003, six baseline surveys and four follow-up surveys have been conducted, with the seventh HOS round being fielded in 2004. CMS, working with its partners, performs the following tasks as part of the HOS program: 1) Supports the technical/scientific development of the HOS measure, 2) Certifies survey vendors, 3) Collects Health Plan Employer Data and Information Set(HEDIS^®)⁹ HOS data, 4) Cleans, scores, and disseminates annual rounds of HOS data, public use files and reports to CMS, Quality Improvement Organizations (QIOs), Medicare+Choice Organizations (M+COs), and other stakeholders, 5) Trains M+COs and QIOs in the use of functional status measures and best practices for improving care, 6) Provides technical assistance to CMS, QIOs, M+COs and other data users, and 7) Conducts analyses using HOS data to support CMS and HHS priorities [10].

As stated previously, the goal of the HOS is to collect valid and reliable health status data in Medicare managed care for the purposes of quality improvement activities, public reporting, and competition based plan accountability and health outcomes improvement [10]. The overall HOS performance measurement results, by cohort are presented in Table 4.

The data suggest some general performance improvement over time, as the number of plans reporting better than expected scores increased, particularly for physical health. The number of plans with mental health scores worse than expected also declined. However, the effect of plan attrition over time on plan performance of remaining plans is unclear. Additionally, the majority of plans show no change in performance over the two year period, suggesting that the SF-36^® may not sufficiently discriminate differences in plan performance.

Because of the HOS, QIOs and health plans have data that they may use to identify opportunities to improve the quality of care provided to their Medicare populations. In addition, they are regularly provided information and education on intervention strategies that may lead to measurable improvements in health care quality.

HOS data are also used as part of CMS's Health Plan Management System (HPMS), a system of process and outcomes health plan performance measures. These measures are composite scores derived from HEDIS^®, Consumer Assessment of Health Plans Study (CAHPS^®),¹⁰ HOS and disenrollment data, and are used by CMS to rank plans [36]. To develop a plan composite score, the percentile ranks for each reported HEDIS^®, HOS, CAHPS^® and disenrollment indicator are averaged. Those plans deviating substantially from the mean are considered either high or low overall performers based on the national comparison group. By comparing their composite scores against those of others on individual indicators, health care organizations are able to better assess how they compare with a national sample of their peers. The formation of this scale may represent an interim step in the development of a score card that could be used by CMS, managed care plans, and others to assess health plan performance.

HOS data may facilitate identification of beneficiaries at risk of declining physical and mental functioning. For example, Bierman and Clancy (2001) identified chronic conditions that lead to poor physical functioning and mental functioning among elderly women in Medicare managed care plans using HOS data [37]. These conditions include, for example, arthritis, coronary artery disease, diabetes, and reported feelings of depression or sadness [37]. Using HOS data, they examined the extent to which women of lower socioeconomic status and/or racial/ethnic minorities were disproportionately affected by these conditions. They suggest that HOS data can assist in addressing these disparities.

Further, the HOS has provided CMS the ability to use health outcomes data for payment adjustments. CMS is now using the ADL data from the HOS to calculate a frailty adjustor to set payment for Social Health Maintenance Organizations. A variant of the HOS, the Program of All-inclusive Care for the Elderly (PACE) Health Survey, is being used to calculate a frailty adjustor for PACE plans and for demonstrations that target frail Medicare beneficiaries [38].

HOS data have also been used in research. To date, eleven research reports have been made available on the Medicare HOS website, covering topics such as how performance measurement results are calculated, and the health status of younger individuals with disabilities enrolled in M+C plans. These reports are available at http://​www.​cms.​hhs.​gov/​surveys/​hos/​hosresearch.​asp. The HOS website also includes a list of publications appearing in scientific journals.

Researchers have used HOS data to examine the functional status of chronically ill Medicare managed care enrollees to assess the need for disease management programs. A study also reviewed the disease management demonstration projects sponsored by CMS, and discussed how such interventions may help enhance the functional status of chronically ill Medicare beneficiaries over time. Through exploring the change in SF-36^® scores over time, it has been found that the presence of chronic disease has a negative impact on both physical and mental health of enrollees over a two year period [26]. Preventing the onset of disease is the best manner in which to maintain optimal health. However, once a disease has surfaced, there must be interventions in place to manage the disease. Results from the HOS data indicate that opportunities exist to enhance health outcomes in this population [15]. Efforts by CMS, such as the incorporation of guidelines into HOS documents encouraging the use of evidence-based interventions to prevent and manage chronic illness, convey CMS's goal of helping plans use HOS data to improve the health and functional status of enrollees [39].

The HOS has also been used as a vehicle to explore emerging and/or geriatric quality care related health issues and concerns. The HOS instruments have included questions concerning health services utilization, retirement community living, smoking frequency and cessation, healthy days, management of urinary incontinence, among others [10]. The healthy days questions from the Centers for Disease Control and Prevention's Behavioral Risk Factor Surveillance System (BRFSS) were added to allow a link between HOS and BRFSS results, facilitating comparison across federal initiatives. The management of urinary incontinence questions were added in 2003 as a new HEDIS^® measure [10].

Public Law 108–173, The Medicare Prescription Drug, Improvement, and Modernization Act of 2003, provides HOS a clear mandate and focus. Specifically, Section 722 specifies requirements for CMS and Medicare Advantage (MA) organizations that must be undertaken to pursue quality health care for Medicare beneficiaries. Section 722 requires that each MA organization comply with the following:

It also requires CMS to establish appropriate regulatory requirements for the following:

The HOS is now the only approved patient-based health outcomes measure for MA quality improvement and performance monitoring. In addition, HOS provides MA organizations data that they can use as a means of identifying and monitoring enrollees with chronic conditions as part of their chronic care quality improvement programs. Using HOS data in this manner however, may be limited to some extent by the underlying sampling strategy, which was designed as a simple random sample with each MA organization as the unit of analysis. Given the restriction imposed by the new legislation concerning the development of new measures, HOS will likely be CMS's primary health outcomes measure of MA plans for the foreseeable future.

Articles and studies published to date suggest that CMS's leadership and innovation with respect to the development of the HOS represented an important initial step toward the development of one major component of an information infrastructure needed to monitor and improve the quality of health care. Now fully developed, HOS's data and administration are under study. Interviews with HOS partners and stakeholders have revealed that the majority feel that the HOS provides a rich and unique set of valid, reliable, and actionable data. However, interviewees have offered some additional recommendations to improve the HOS, which include: 1) tracking the health status of plan disenrollees, and 2) creating greater incentives for plans and QIOs to use the HOS data.

CMS values programs and measurement tools that facilitate improvements in health care for its beneficiaries. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 mandate the measurement of health outcomes and restrict the development of new measures. Consequently, HOS is CMS's primary vehicle to collect outcomes data, particularly patient-based functional health status measures. The evaluation of the efficacy of the HOS program will provide CMS with essential information to improve the HOS and to determine the best way for HOS to effectively meet this new legislative mandate.