Utilization of the propensity score method: an exploratory comparison of proxy-completed to self-completed responses in the Medicare Health Outcomes Survey

Literature exists documenting the differences between self and proxy-completed responses on health status surveys. For example, some research demonstrates that proxy-completed responses tend to more accurately report conditions that are less private and more observable, but tend to underestimate less observable conditions such as emotional and affective states [4, 5]. Additionally, Yip, Wilber, Myrtle, and Grazman found that mean scores were significantly lower for proxy-completed responses compared to self-completed responses on the Physical Functioning, Vitality, and Mental Health scales of the 36-Item Short-Form Health Survey (SF-36)[6].

Other research also has indicated significant disagreement between proxy-completed responses and observers for instrumental activities of daily living (IADLs). In this research, proxy-completed responses underreported IADLs compared to observers who watched subjects engaged in IADLs [7]. Systematic biases were found in the National Health Interview Survey; results indicated that proxy-completed responses underreported disabilities for those aged 18 to 64 years, but overreported disabilities for those 65 and older [8]. Data from the Canadian SF-36 indicated that proxy-completed responses tended to underestimate health status, with poor to moderate agreement between proxy-completed responses and the disabled elderly [9]. In examining data from Canada's National Population Health Survey, Shields [10] found significant differences between self and proxy-completed responses. In general, proxy-completed respondents underestimated the prevalence of certain health conditions. However, disagreement between self-completed and proxy-completed respondents was less likely for conditions that the proxy respondents were more likely not to mislabel, such as diabetes, heart disease, and cancer. It is evident that there are inconsistencies in the literature regarding proxy-completed and self-completed respondents and continued research is necessary to understand these inconsistencies.

Because proxy-completed responses are often necessary in assessing health outcomes for the elderly, it is important that methods be found for examining selection bias. The propensity score is one such method used to reduce selection bias in observational studies. This paper explores the use of the propensity score method in understanding the differences between proxy and self-completed responses, by applying this method to the Medicare HOS data.

Donald Rubin [3, 11‐13] pioneered the propensity score methodology, which has been used extensively in medical research [e.g. 14–17]. Theoretically, this method is similar to an experimental design, but it is applied to survey or observational data and has the potential to reduce selection bias. Simply put, the propensity score is the probability that an individual belongs to a naturally occurring treatment group, based on the individual's background characteristics (covariates). Since the propensity score summarizes the information on the background characteristics in a single summary score, it has a distinct advantage over standard matching techniques [18‐20]. These latter techniques require the investigator to find subjects who are closely matched on each of many individual covariates, an often difficult task. Once the propensity scores have been calculated, the "treatment" and "control" groups (in this case, proxy-completed and self-completed respondents) can each be stratified into similarly matched comparison groups based upon their propensity scores. For each stratum we can then examine two groups of survey respondents, a group of proxy-respondents and a group of self-respondents that have similar propensity scores and who were "randomly" assigned to the groups in the sense of being equally likely to be a proxy or self-respondent [2].

For example, in a 2002 study on nephrology consultation in acute renal failure, Mehta et al. [17] used the propensity score methodology to assess the timing of nephrology consultation and in-hospital mortality. In this example, the authors created the propensity score using the characteristics that differentiated the delayed and early consultation groups. These authors state, "Inclusion of the propensity score as a covariate in a multivariable regression accounts for likelihood of 'treatment' (in this case, timing of consultation), and may adjust for unobserved, confounding, and selection bias, thereby refining regression estimates".

Similarly, Kilborn et al. [14] utilized the propensity score methodology in a nonrandomized study of amiodarone and mortality among acute myocardial infarction patients with atrial fibrillation. Patient characteristics that were associated with prescriptive use of amiodarone were incorporated into a regression analysis through the use of the propensity score methodology.

Assessing the difference between proxy-completed and self-completed responses in survey research is analogous to nonrandomized treatment studies such as the two discussed above. Proxy-completed responses can be conceptualized partly as the results of selection bias, and partly the result of true differences between proxy and self-respondents. Generally, proxy respondents who answer survey items for the respondent occupy a specific role in the respondent's life such as a family member (spouse, child), friend, or professional caregiver. Essentially, these proxy respondents bring a cognitive role set with them when they complete survey items on behalf of the respondent. This role set can bias responses to survey items. For example, if an adult child completes a survey for an elderly parent, the adult child may understate the physical and mental health status of the elderly parent. Indeed, these results were found in analyses conducted on the Canadian SF-36 [9]. Hence, it becomes very important to test for differences between proxy and self-completed respondents. If differences are found, adjustment procedures such as the propensity score should be used to reduce those differences in further analyses on outcomes such as physical and mental health status.

The propensity score is a methodology that has heuristic potential for quality of life research, which generally involves self and proxy-completed responses. We apply this methodology to three cohorts of data that include self-completed and proxy-completed responses from the Medicare HOS in order to determine if differences in physical and mental health status remain after adjustment using the propensity score.

The Medicare HOS assesses the physical and mental health status of the Medicare elderly enrolled in managed care in the United States. Beginning in 1998 and continuing annually, a baseline cohort is created from a randomly selected sample of 1,000 Medicare members from each applicable Medicare contract market area. In plans with fewer than 1,000 Medicare members, the sample includes the entire enrolled Medicare population that meets the inclusion criteria. Medicare beneficiaries who are continuously enrolled in the health plans for at least six months are eligible for sampling [21].

The data collection protocol includes a combination of mail and telephone surveys. Multiple mailings, standardized telephone interviews, interviewer training, and methods for maximizing response rates are well established in the Health Plan Employer Data and Information Set (HEDIS) specifications [22]. The Medicare HOS instrument includes the SF-36 health survey, which is a widely used multi-purpose, short-form health survey. Psychometric properties, reliability and validity studies of the SF-36 as well as normative data are available in user manuals [23, 24]. The SF-36 yields an eight-scale profile of scores and is a generic measure as opposed to one that targets a specific age, disease, or treatment group. The eight scales form two distinct higher ordered clusters that are the basis for scoring the physical component summary (PCS) measure and mental component summary (MCS) measure. For this analysis, the SF-36 individual scale scores, as well as the PCS and MCS scores, have been normed to the values for the 1990 general U.S. population, so that a score of fifty represents the national average for a given scale or summary score. Higher scores on the SF-36 represent better physical and/or mental health status.

The respondents included in this study were beneficiaries in baseline cohorts I, II, and III; the data sets represented survey results for 1998, 1999, and 2000, respectively. The Medicare HOS cohort I baseline consisted of 279,135 Medicare members who were sampled from 269 Medicare+Choice organizations (M+COs) representing 287 contract market areas. Cohort II baseline consisted of 301,184 Medicare members from 283 M+COs in 312 contract market areas, and cohort III baseline consisted of 298,883 Medicare members from 275 M+COs in 306 market areas. Several criteria were met in selecting the final analytic sample. First, all duplicates were removed; i.e., only the first survey was used for any beneficiary. Second, beneficiaries in plans for Program of All-Inclusive Care for the Elderly (PACE), as well as EVERCARE (a program that provides care and care coordination to vulnerable, chronically ill beneficiaries) in cohorts II and III were removed (0 in cohort I; 4,225 PACE and 5,015 EVERCARE beneficiaries in cohort II; 3,267 beneficiaries in PACE programs in cohort III; PACE and EVERCARE beneficiaries have significantly lower PCS and MCS scores and are much more ill than non-PACE and non-EVERCARE beneficiaries in the Medicare HOS). Third, surveys must have had a response for the question, "Who completed this survey form?" and finally, responses must have had a survey for which the PCS and MCS scores were calculable. Based on these criteria, the total sample size of proxy-completed responses was 65,668 and for self-completed responses the total was 457,837.

In addition to the SF-36, demographic data, activities of daily living (ADLs), chronic conditions (angina pectoris, arthritis, cancer, congestive heart failure, Crohn's disease, diabetes, emphysema/asthma/chronic obstructive pulmonary disease [COPD], hypertension, myocardial infarction, other heart conditions, sciatica, and stroke), and three depression-screening questions were examined for differences between self-completed and proxy-completed responses.

Three steps were necessary in applying the propensity score method to the Medicare HOS data. First, self-completed and proxy-completed responses were examined to establish differences between the groups of respondents (unadjusted comparisons). These two groups were compared on demographic variables, the SF-36 scores, type of chronic condition, type of impaired ADL, and three depression-screening questions.

Second, the propensity score was used to create comparison samples. The propensity score matching process involved developing a stepwise logistic model [12] to determine which demographic, disease, and disability variables affected the likelihood of a proxy response. Based on the values of these predictors, each beneficiary in the data set had an estimated probability of using a proxy, which is the propensity score.

Third, due to the large sample sizes, a stratified random sample was drawn from each decile of the distribution of the propensity score from the proxy-completed and self-completed respondent groups. Once the sample was drawn, resulting in a risk adjusted sub-sample, comparisons were made to determine whether proxy-completed responses differed from self-completed responses. Given the large sample sizes, effect sizes were used to determine significance. Effect size is defined as "...the degree to which the phenomenon is present in the population...or the degree to which the null hypothesis is false." Cohen [25] operationally defines effect sizes as follows: a small effect size is one that accounts for 2% (0.02) of the variance, a medium effect size accounts for 13%, (0.13) and a large effect size accounts for 26% (0.26). Cohen's effect size for proportions (p) was used to calculate the effect sizes for Tables 1 and 2 (h = φ₁ - φ₂, where: φ = 2arcsin √p). Cohen's effect size for means was calculated as: