Measuring Health Spillover Effects in Caregivers of Children with Autism Spectrum Disorder: A Comparison of the EQ-5D-3L and SF-6D

A common metric used to quantify the effectiveness of healthcare interventions in cost-effectiveness evaluations is the quality-adjusted life-year (QALY), which incorporates both the quantity and quality of life gained [1, 2]. The QALY has a number of useful properties that led to it being the standard for conducting cost-effectiveness analysis [2]. One area of concern, however, is the fact that, in practice, QALY measurement for cost-effectiveness analysis typically focuses solely on the health effects accruing to patients, as if these were isolated individuals [3]. By now, it has been shown that health effects in patients are typically associated with substantial spillover effects on the health and well-being of caregivers and family members [4‐6]. Failure to include such spillover effects in economic evaluations can lead to a misrepresentation of the burden of disease and the benefits of health interventions [7]. This, in turn, may lead to suboptimal decisions, both from a healthcare and a societal perspective [8].

Regulatory agencies now recognize the need to incorporate spillover effects in economic evaluations. Both the National Institute for Health and Care Excellence (NICE) and the Second US Panel on Cost-Effectiveness in Health and Medicine recognized the potential for spillover effects to influence estimated cost-effectiveness ratios and recommend including them in a reference case analysis [9‐12]. The Second US Panel also emphasized the importance of increasing research efforts on clarifying how to incorporate family and caregiver spillover effects in economic evaluations [10].

Despite the recognition for the need to include health spillover effects when valuing health interventions, little guidance exists for including spillover effects in cost-effectiveness analysis [8]. For example, there is no guidance for incorporating spillover effects into a cost-effectiveness analysis in the context of clinical trials that could inform regulatory agencies. One option would be to capture these effects by measuring health states across trial arms for patients, caregivers, and family members. This results in a focus on health (rather than well-being), which has the advantage of being the most relevant outcome in most studies and decision-making contexts, making effects comparable across groups and able to be aggregated. In the design of such clinical trials, decisions need to be made about which instruments are able to capture spillover effects in QALY terms. Early research on spillovers in Alzheimer’s disease attempted to estimate effects by comparing caregiver outcomes across clinical characteristics such as stage of disease and setting [13], but likely failed because the instrument was not sensitive or did not discriminate well [7, 14]. Subsequent analysis showed that traditional measures of burden and health changed in the expected direction, but the Health Utilities Index Mark 2 (HUI-2) did not capture these changes [14]. While a large literature has emerged that allows us to understand whether a given instrument is valid for measuring QALYs for different conditions affecting patient populations [15‐17], research identifying instruments that are valid and responsive in measuring spillover effects in caregivers and family members remains understudied. Indeed, we are aware of only two studies that have compared different generic preference-weighted instruments to measure spillover effects. Payakachat et al. [18] compared three preference-weighted health instruments to measure spillover effects among caregivers of children with craniofacial malformations. Bhadhuri et al. [19] compared two preference-weighted instruments to measure spillover effects among family members of meningitis survivors.

Family and caregiver spillover effects, in terms of health and well-being, may be particularly pronounced in child health interventions [20‐22] and for mental health conditions where social support systems may be lacking [23, 24]. Interventions for children with autism spectrum disorder (ASD) have the potential for substantial spillover effects in caregivers and family members due to an increased prevalence of psychiatric and medical co-morbidities such as anxiety, behavioral problems, sleep disturbance, and cognitive issues [25‐27]. Preventing symptoms of ASD in the child is thus likely to improve family and caregiver health and reduce burden [22].

Given the potential for interventions such as medications or applied behavioral therapy to improve the health of children with ASD, a comparison of generic preference-weighted instruments to determine whether they capture spillover effects in QALY terms of health associated with treatment for children with ASD appears warranted. Therefore, the purpose of this study was to assess the ability of two commonly used generic preference-weighted instruments, the three-level EuroQol-5 Dimension instrument (EQ-5D-3L) and the Short Form-6 Dimension (SF-6D) [28, 29], derived from the 12-item Short Form survey version 2.0 (SF-12 v2.0), to value spillover effects in caregivers of children with ASD in order to provide guidance about their use, especially in the context of clinical trials and other approaches where indirect elicitation techniques are warranted.

The study sample contained 224 caregivers of children with ASD. The response rate for the completion of the postal survey components was 115 of 220 (52%) at one ATN site and 109 of 179 (61%) at the second ATN site. In our sample, 90% of caregivers were female, around 60% had at least a college education, and 76% were married (Table 1). The average caregiver age was 39.4 years, and the average child age was 8.4 years. Of the children included in the study sample, 87% were male (Table 1). A complete description of the demographic characteristics of the study sample can be found in Hoefman et al. [31].

The Second US Panel on Cost-Effectiveness Analysis in Health and Medicine and other government agencies around the world have emphasized that research on valuing spillover effects in family and caregivers is warranted [9‐12]. While spillover effects can also be studied in a broader context, increasing knowledge of the health effects among caregivers and family members in QALY terms is highly relevant and consistent with a societal perspective [4] and a healthcare perspective [8]. A number of approaches have been adopted to value spillover effects in QALY terms, including direct and indirect elicitation [62‐65]. In the context of clinical trials and other intervention studies, indirect elicitation techniques are likely to be favored, as is the case with measuring patient QALYs, and require guidance about appropriate instruments consistent with the large literature devoted to identifying the most appropriate instrument for patient conditions. Despite the obvious appeal of indirect elicitation techniques for capturing spillover QALYs, research on comparing different instruments is lacking.

The need for guidance on different approaches for measuring spillover effects in QALY terms is especially important given the recent change in recommendations by the Second US Panel. Traditionally, spillover effects were included in cost-effectiveness analyses using monetary costs, often measured by the additional time devoted by a caregiver to caring for the patient. Inclusion of non-monetary values, or QALYs of caregivers and other family members, along with monetary costs raised concerns about double-counting [65]. Indeed, a recent review of methods for valuing informal care offered guidance for including spillover effects in either monetary or non-monetary terms because of issues with double counting and other concerns [63]. The Second US Panel now recommends the inclusion of both monetary and non-monetary spillover effects in economic evaluations [10].

This study compared the EQ-5D-3L and SF-6D with respect to their ability to capture spillover effects in caregivers of a child with ASD. To compare the instruments, we first assessed whether they would provide similar results for similar caregivers. In particular, if the two instruments were correlated with each other and with other measures of caregiving quality of life or health, it would suggest the measures were valid instruments. Both measures demonstrated convergent validity as they were strongly correlated with each other, the CarerQol, and the CES-D. While the SF-6D exhibited a stronger correlation with the CES-D, the EQ-5D-3L exceeded criteria for a strong correlation.

Second, we assessed whether the instruments would provide similar results in relation to the characteristics of the child with ASD. In particular, scores on measures of child health were compared in relation to the top and bottom of the distributions for the two instruments as well as the difference in instrument scores in response to changes in the child health measures. Significant differences in child health scores in relation to the distributions of the two instruments demonstrates discriminative power while differences in instrument scores in relation to differences in child health measures demonstrates clinical validity. Both instruments demonstrated discriminative ability; however, the SF-6D had a greater percentage of significant findings than the EQ-5D-3L. With respect to clinical validity, the SF-6D similarly performed slightly better on the measures of child health, with significant differences in average health utility scores relative to scores on four of the five child measures (PedsQL™, HUI-3, CBCL, and CSHQ) compared with significant differences for two child measures (HUI-3 and CSHQ) for the EQ-5D-3L. Neither measure was associated with the child’s age, IQ, autism severity, or diagnosis. Significant differences in average scores across the child health measures, but not the child’s age, IQ, or autism diagnosis, indicates that it is differences in child health and behavior that drive spillover effects among caregivers.

Based on the comparison of the two instruments in this study, some guidance can be offered for those interested in developing clinical studies to measure caregiver spillover effects associated with caring for a child with autism. Either the SF-6D or the EQ-5D-3L are likely to capture health effects among caregivers in QALY terms for interventions or changes in the clinical characteristics of children with autism that are associated with measurable health effects for the child. Interventions such as new molecules for the treatment of behavior problems that produce meaningful changes in the CBCL are likely to have spillover effects for the caregiver that can be captured by standard preference-weighted instruments such as the SF-6D or the EQ-5D-3L, and we recommend their inclusion in clinical trials and other research designs that can identify causal effects.

Researchers such as Hoefman et al. [48] suggest that the effects of caregiving on caregivers can be measured with the same preference-weighted instruments used to measure HR-QOL in patients. Surprisingly, few studies have assessed preference-weighted instruments to determine whether they are sensitive or responsive for measuring caregiver or family spillover effects. Given that regulatory agencies recommend indirect elicitation with preference-weighted instruments to measure patient QALYs [9‐12], more research appears warranted to compare preference-weighted instruments for measuring spillover effects in other contexts, such as adult children caring for their parents, and other conditions in children, including somatic and mental health conditions.

Several limitations to the study should be noted. First, we limited the caregiver quality-of-life and health measures to two previously validated instruments: the CarerQoL-7D and CES-D. It can be argued that the CarerQoL-7D captures a different construct (burden of caregiving) than the health utility instruments (HR-QOL) and the CES-D is limited to mental health problems. Still, both of these measures should be correlated with health utility measures as greater caregiver burden translates into worse HR-QOL. This was the case in this study and, more importantly, the comparison of the EQ-5D-3L and the SF-6D demonstrated strong correlations with our measures of caregiver burden. The information produced with caregiver-specific instruments such as the CarerQol-7D may be more appropriate in evaluations of interventions targeted at caregivers specifically given that the CarerQol-7D measures the impact of caregiving beyond health effects [31, 48].

Second, we relied on caregiver self-reports regarding health states for themselves and their children. This approach may lead to problems of endogeneity, especially in study designs where treatment effects cannot be identified. Alternative designs, where the child is rated by a family member other than the primary caregiver, may provide an indication of the extent to which caregivers project their own health states onto the rating of their children [63]. Clinical studies based on exogenous instruments, such as randomization or disease states, are likely to limit problems with endogeneity and can identify spillover effects using indirect elicitation techniques. In addition, there have been considerable methodological advances in using direct elicitation techniques to measure spillover effects [64]. Direct elicitation techniques may be particularly sensitive to a given population and expanding research on direct elicitation techniques to include caregivers of children with ASD could supplement the findings from our study.

Third, the comparisons were based on the EQ-5D-3L, which has only three response levels per construct, rather than the EQ-5D-5L which has five response levels. The EQ-5D-5L may have increased validity and discriminative power [66] and has been suggested to have greater responsiveness than SF-6D in other caregiver contexts [19]. Our finding that both the SF-6D and EQ-5D-3L can be used to capture spillover effects for interventions involving children with autism remains and likely translates to the use of the EQ-5D-5L. Finally, we limited this investigation to health-related spillover effects in primary caregivers. Broader investigations, including observing effects in other family members and effects beyond health in an extra-welfarist context, remain important as well [8].

Capturing spillover effects in cost-effectiveness analyses is necessary to ensure accurate valuations of healthcare interventions and programs. Our comparison of the SF-6D and EQ-5D-3L health utility instruments indicated that both can capture health-related spillover effects in terms of health among caregivers of children with ASD, although the SF-6D had stronger discriminative power and clinical validity in this context. The findings provide useful information for researchers and practitioners interested in developing protocols for measuring caregiver spillover effects in QALY terms. It is feasible to use indirect assessment in clinical studies to measure caregiver spillover effects associated with interventions to improve the health of children with autism. Regulatory agencies recommend indirect assessment to measure patient QALYs, and this study provides evidence for recommending a similar approach to capture caregiver spillover effects associated with child health conditions such as ASD.