Controlling for race/ethnicity: a comparison of California commercial health plans CAHPS scores to NCBD benchmarks

In 1995 the United States Agency for Healthcare Research and Quality launched what is now the Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) program in an effort to standardize health plan evaluation. The CAHPS Health Plan Survey, a voluntary survey designed to capture data on member experiences with care as a measure of health plan quality, is one of a family of CAHPS surveys designed for specific health care settings [1]. The CAHPS Health Plan Survey is incorporated into the National Committee for Quality Assurance (NCQA) accreditation process for health plans [2]. Consumer assessments are playing an increasing role as quality indicators and in enrollee guides [3]. U.S. News & World reports have ranked California commercial health plans lower than national counterparts in rankings based on performance on NCQA clinical quality measures for prevention and treatment, as well as CAHPS patient experience responses that are adjusted for variables including age, general health rating, and gender [4, 5]. Only one California health plan was ranked in the top 25 percent of commercial health plans in their 2008 report. Even among California's highest ranked health plans, consumer assessment scores are consistently lower than prevention and treatment scores [4].

In 2007, California was home to 12 percent of the American population [6]. However, it accounts for one quarter of the total U.S. health-maintenance organization (HMO) enrollment [7]. HMOs can be broadly defined as health plans that offer prepaid, comprehensive health coverage for both hospital and physician services. Close to half of California's 35 million population is enrolled in an HMO [8]. Since higher HMO penetration in California may forecast higher HMO penetration in the rest of the nation, it is important to understand why member scores are lower in California.

One plausible explanation for lower HMO member ratings of care in California is that the racial and ethnic composition of California differs from the rest of the nation. It may be appropriate to consider race/ethnicity in member surveys because the Institute of Medicine includes "equity" as one of the six domains of health care quality [9]. CAHPS survey results are case-mix adjusted for age, gender, and self-reported health status, but although CAHPS surveys collect data on race/ethnicity, no adjustment is made for race/ethnicity [5]. Other research suggests individual health plans have a greater influence on member experiences of care than race/ethnicity, though this too is not a CAHPS-specific recommended adjustment [10, 11].

Racial and ethnic disparities in survey responses are well documented. A 2003 study looking at Medicaid CAHPS data determined that consumer assessments of managed care varied by race, ethnicity and language, with racial/ethnic minorities reporting lower quality of care [12]. A study using the Medical Expenditure Panel Survey to assess enrollees in Medicaid managed care programs found Hispanics and Asian Americans reported less satisfaction with care than Whites [12]. Findings from the Commonwealth Fund 2001 Health Care Quality Survey showed only 56 percent of Hispanics and 45 percent of Asian Americans reported being "very satisfied" with their health care versus 65 percent of Whites [13]. Additionally, 15 percent of African Americans, 13 percent of Hispanics and 11 percent of Asian Americans felt they would have received better care if they were a different race or ethnicity.

A number of studies have shown that Asians and Pacific Islanders report different experiences with care than other races/ethnicities. Weech-Maldonado et al. reported Asian non-English speakers presented the lowest ratings of care of all the ethnicities and "had more negative reports and ratings of care" than Asian English speakers, whose ratings were similar to Whites [14]. Phillips et al. found Asian Americans the ethnic group most likely to report dissatisfaction with the quality of care provided by their usual source of care [12]. Taira et al. reported Asian Americans rated overall satisfaction, and 10 of 11 scales assessing primary care significantly lower than Whites [15]. They suggested that controlling for Asian-American ethnicity "could significantly improve the profiles of physicians with a large percentage of Asian-American patients." In a managed care setting, Asian patients as a group have been found to be "significantly less satisfied" with various aspects of primary care performance than other ethnicities [16].

California's diverse demographics may significantly impact its consumer assessment scores. Notably, there are more non-Whites than Whites in California with 35.9 percent Hispanic/Latino, 12.4 percent Asian, and 6.7 percent Black [17]. In 2007, California ranked 2^nd in the nation for percent of the total population who are Asian alone [18].

This study will test the hypothesis that California commercial health plan CAHPS survey results are attributable to the racial and ethnic composition of the state's population. California commercial health plan CAHPS survey responses will be compared to the 2005 National CAHPS Benchmark Database (NCBD) for selected measures. CAHPS survey results stratified by race/ethnicity will be presented and the impact of controlling for race/ethnicity as case-mix adjustments for California and nationally, will be discussed.

National CAHPS 3.0 data for commercial adult health plan enrollees in the year 2005 was obtained from the National CAHPS Benchmarking Database (NCBD) after receiving approval from the UCSF Institutional Review Board (#06028447) and from NCBD. We did not analyze CAHPS data for Medicaid, State Children's Health Insurance Plan (SCHIP), or Medicare enrollees. Demographic characteristics measured included: age group, gender, race/ethnicity, insurance plan, type of insurance, time enrolled in insurance plan, educational level, general health rating, state of residence, report having a personal doctor, and report going to doctor's office or clinic in the last 12 months. These measures were selected to reflect the variables adjusted for in other similar surveys, and because in California they were found to be significantly different than the rest of the nation [19]. Nine questions from the CAHPS database were analyzed. Three questions were selected to reflect global ratings of doctor, health plan, and overall care. Six questions that comprise two composites were selected as indicators of interactions with providers that might be impacted by cultural competency and/or racial or ethnic differences: doctor communication and staff courtesy, respect and helpfulness. See Table 1 for a complete listing of the questions used. Two different scales were used to answer these questions: global ratings used a scale from 0 to 10, where 0 is the worst possible and 10 is the best possible; and all composite questions used a four point scale of never, sometimes, usually and always.

We used the provided CAHPS categories for all of our demographic variables except race and insurance plan type. We combined the race and Hispanic variables in the database to incorporate Hispanic ethnicity into our race variable. Anyone who reported they were Hispanic in the Hispanic variable was coded as such regardless of race. Anyone reporting non-Hispanic in the Hispanic variable was coded as the race they specified in the race variable so our race categories were mutually exclusive. The type of insurance variable was collapsed into two categories: HMO and non-HMO. The non-HMO includes the CAHPS categories of POS, PPO, combined and other. Using the state of residence we created a variable to distinguish people in California from those in the rest of the nation (non-California).

In order to facilitate reporting and analysis, we chose to collapse each global rating score into two categories: highly rated or not highly rated. We followed the recommendation of Weech-Maldonado et al. (2008) and anyone responding either 9 or 10 on the worst to best scale was categorized as highly rated on that measure while all lower scores were categorized as not highly rated [20]. We followed CAHPS directions to create our composite variables [21]. We chose to equally weight the questions in each composite and added together all scores to get a possible total score of 16 for doctor communication (composite consists of four items) and 8 for staff courtesy, respect and helpfulness (composite consists of two items). We then collapsed these into highly rated categories based on scoring at least 90% of all possible points. Respondents scoring 14 or more on the doctor communication composite were categorized as highly rated. Those scoring 7 or 8 on the staff composite were categorized as highly rated. Scores lower than these were categorized as not highly rated.

We compared California to the rest of the nation using chi-square tests for the categorical demographic data. Chi-square tests were also used to look for differences in highly rated results for each global rating and both composite reports of care by race categories. We also compared California to the rest of the nation on each of these five measures using chi-square. Stepwise multiple logistic regression was used to see if ratings in California differed from all the other states combined (rest of the nation) on the five measures. Five regressions were done using the dependent measures of doctor rating, plan rating and care rating as well as reports on doctors communication and staff courtesy, respect and helpfulness. We used the following independent variables in each regression: age group (3 categories), education (6 categories), gender (2 categories), general health rating (5 categories), individual health plan (254 categories), insurance type (HMO or non-HMO), location (California or non-California), race/ethnicity (5 categories), and time in plan (4 categories). Fisher's exact tests were used to compare overall rankings of individual health plans for California to the rest of the nation using the logistic regression odds ratio results. Categories created based on the results of comparison to a New York HMO were: statistically higher, not statistically different, and statistically lower.

We analyzed 123,272 adult commercial health plan CAHPS respondents of which 9,952 were from California and 113,320 were from the rest of the nation. California adult commercial CAHPS participants differed from CAHPS participants in the rest of the nation in their demographic characteristics as well as their responses. Respondent characteristics are shown in Table 2. Californians are more likely to be 18-34 years old and less likely to be 65 years old or older. They are more likely to be males, more likely to have some college training, less likely to have more than a four year college degree, and more likely to be in an HMO. Californians are more likely than CAHPS participants in the rest of the nation to self identify as Hispanic or as Asian. They are also less likely to have a personal doctor or to have visited a doctor's office or clinic in the last 12 months.

Ratings in response to the five selected measures for California and the rest of the nation can be seen in Table 3. California CAHPS ratings were lower than the rest of the nation for each of the selected measures except health plan. A similar pattern was noted when we looked at racial and ethnic subgroups. In particular, Hispanics had the same response pattern with the exception that California health plans rated lower than national health plans, while Blacks in California rated their doctor, health plan, care, and staff courtesy, respect, and helpfulness higher, but their doctors communication lower in California than in the rest of the nation. Asians responded with the lowest scores on all measures in California as well as the rest of the nation.

Stepwise logistic regression results agreed that the seven independent variables that contributed to the model in order of significance were: age, general health rating, education, individual health plan, time in plan, race/ethnicity, and gender. Variables that were not significant included type of insurance and California location. Table 4 shows the odds ratios with confidence intervals for all five rating/report characteristics and six of the seven significant independent variables. In the interest of space, the 254 individual health plans were not included in Table 4. In general, individuals with high ratings of their doctor, plan and care tended to be older, in better health, have less than a high school diploma, be in their plan longer, not be of Asian race, and be female. Composite reports of care regarding communication and staff courtesy differed from ratings by gender; males had higher rates than females. Reports of staff courtesy also differed by race/ethnicity. Only Blacks reported higher results, all other races were lower than Whites. In an overview of the individual health plans odds ratios, California plans did not differ from the rest of the nation except for the care rating where more California plans were statistically higher (data not shown).

We hypothesized that racial and ethnic differences in member experience of care scores, coupled with the higher concentration of Hispanics and Asian/Pacific Islanders in California would explain the lower scores California commercial health plans have received on CAHPS surveys. However, our analysis of adult enrollees in commercial health plans found that race/ethnicity was only one of multiple variables that influenced responses to selected CAHPS questions in California and the rest of the nation. When these variables were all accounted for, CAHPS scores for California health plans were no different than the rest of the nation. We found that the variables that contributed first in our stepwise logistic regression were age, general health rating, education, and health plan. Time in plan, race/ethnicity, and gender were also significant, but they were consistently less influential than the four variables noted above. Our findings suggest that national comparisons of patient experiences of care should adjust for a broader range of variables.

California health plan enrollees rate their care similarly to health plan enrollees in the rest of the nation when we controlled for age, general health rating, education, individual plan, race/ethnicity, time in plan, and gender characteristics. Standardizing populations for these variables prior to comparing individual health plans or health plans by state would lessen the likelihood of ascribing differences in member reports of experiences of care to the health plan or state, when they may have more to do with the enrollees themselves.

Stratifying member experiences of care by race/ethnicity highlighted important differences in CAHPS ratings between different racial and ethnic groups in both California and the nation. Reporting on race/ethnicity disparities in member experiences of care could raise awareness and increase accountability for reducing racial and ethnic disparities, especially among individual health plans. Wider dissemination of public reporting which accounts for race/ethnicity and incentive programs that reward reducing disparities in care may help achieve these objectives.