Background
In both Canada and the United States efforts are underway to develop systems to assess the quality of health care as a first step to improving services. In the US nursing home sector, the implementation of the Minimum Data Set has been linked to improvements in quality of care [
1‐
4]. Recently, the Centers for Medicare and Medicaid Services (CMS) has developed web pages to give consumers, and the public at large, further information about the quality of nursing homes and home care services across the country. The "Nursing Home Compare" and more recent "Home Health Compare" web sites allow individuals to view several quality indicators (QIs) for individual providers[
5,
6].
When comparing health care providers across a set of QIs there is a concern that they may differentially admit clients with a greater likelihood of triggering on quality issues. Since the indicators are defined as events that are preferable to avoid (e.g., skin ulcers, untreated pain, weight loss), in the absence of risk adjustment, these providers would appear to be delivering poorer quality of care. Risk adjustment attempts to adjust for different populations of clients who may be at greater risk for experiencing quality issues that is a function of their clinical status rather than the quality of care.
In the US nursing home sector, the CMS recently funded a large-scale project to review all potential long-term care QIs and the risk adjustment process. The evaluation of risk adjustment included a review of both client-level and agency-level covariates. The research team revised the original set of client covariates to create a set of new models and assessed the effects of using an agency-level covariate, the Facility Admission Profile (FAP). The FAP represents the prevalence of the quality issue for residents newly admitted to the facility and was intended to adjust for potential selection bias (i.e., facilities preferentially admitting clients with a greater likelihood of triggering on the indicator) and ascertainment bias (i.e., ability of providers to detect a quality issue that might increase the QI rate). However, after extensive analyses, the research team did not recommend the use of the FAP, given that the FAP did not appear to be a particularly useful measure of ascertainment bias [
7].
Nevertheless, in October 2002, the CMS included the FAP in three quality measures reported on the Nursing Home Compare web page[
8]. As part of this same initiative, Kidder et al.[
9] examined the Nursing Case Mix Index from the RUG-III grouping system as well as seven of the RUG scales as potential risk adjusters. In their final list of quality measures, 12 indicators for chronic care residents and four indicators for post-acute care residents, included a covariate related to the RUG-III system.
Home Care Quality Indicators (HCQIs)
The current research was part of a project to develop a set of 22 Home Care Quality Indicators (HCQIs) based on items in the Minimum Data Set for Home Care (MDS-HC). Implementation of the MDS-HC is complete or underway in 15 states and in 7 Canadian provinces and territories, where it is being used mainly as a clinical assessment instrument. In Michigan, the MDS-HC is being used as part of the MI-Choice waiver program to reduce nursing home admissions. In Ontario, Community Care Access Centre (CCAC) case managers use the instrument to determine needs, allocate services and make placement decisions.
The HCQI derivation was accomplished using data from Ontario and Michigan since these were the regions with the largest scale implementation in Canada and the US, respectively. In their paper describing the HCQI derivation, Hirdes et al.[
10] recommended the use of client-level risk adjustment for all but four indicators, and suggested the use of the Agency Intake Profile (AIP) as another method for risk adjustment. These various approaches to risk adjustment are the focus of this study.
The set of HCQIs used in this study were developed by members of interRAI, a non-profit multinational organization dedicated to the development and refinement of assessment instruments for older adults and persons with disabilities, and their related applications. The HCQIs represent outcome measures that document an agency's rate for triggering on a quality issue. For example, one indicator measures the prevalence of weight loss among individuals who are not considered to be palliative clients. The HCQIs include a mix of prevalence measures (i.e., measured on a cross-section of clients at one point in time) and incidence measures (i.e., failure to improve or incidence of an event measured across two points in time). All HCQIs are defined as events to be avoided such that a higher rate on the indicator is indicative of poorer performance.
This paper uses a dataset from two Canadian provinces to further explore these three types of risk adjustment for the interRAI HCQIs. It explores the effects of risk adjustment at both the level of the region and at the level of the individual home care provider.
Methods
Data
The RAI Health Informatics Project was a two and a half year research study begun in 1999 in which fourteen of Ontario's 43 CCACs used the MDS-HC as part of usual practice for all adult (18 years and older) home care clients. CCACs provide several different services: information and referral, case management and placement in long-term care facilities. CCACs purchase in-home services from external contracted agencies through a "request for proposal" process. Services provided include in-home physiotherapy, occupational therapy, personal support and homemaking, medical supplies and equipment, and care by other professional such as social workers, dietitians and speech-language pathologists. Case managers oversee the assessment process, make referrals to service providers and then monitor the care provided in the home. Funding for CCACs is based on an annual budget provided by the Ontario Ministry of Health and Long-term Care. CCACs are governed by independent, non-profit boards of directors, accountable to the Ministry of Health and Long-term Care.
In Ontario, it is now mandatory for all long stay home care clients to be assessed with the MDS-HC. However, at the time of the RAI Health Informatics project, the MDS-HC was not mandated for use. As such, some clients (e.g., those expected to be on service less than two weeks) did not receive an MDS-HC assessment and were not included in the current project.
At this same time, Manitoba conducted a pilot implementation of the MDS-HC in the fourteen offices of the Winnipeg Regional Health Authority (WRHA). The WRHA is one of twelve health authorities in the province, and is responsible for providing health services to the approximately 646,000 residents of Winnipeg and the surrounding suburban area. The WRHA receives annual funding from the government of Manitoba. The home care program of the WRHA was established in 1974 with a mandate to provide effective and responsive health care services in the community to support independent living and facilitate admission into LTC facilities when independent community living is no longer an option. Home care services include personal care, nursing, counselling, occupational therapy and physiotherapy assessment, referral to other agencies and coordination of services.
In the WRHA region, an MDS-HC assessment was completed by care coordinators only if the client was anticipated to be on service for at least 90 days. As a result, the study sample was focused on a long stay population of home care clients.
All participating sites in the WRHA and in Ontario identified a set of case managers who received a two-day training session led by a member of the research team. They then used the MDS-HC instrument as part of their usual in-home assessment.
The data used for this study represent a cross-sectional cohort of home care clients assessed between November, 1999 and December, 2002. Two CCACs in Ontario and four WRHA sites, were removed from the database because they submitted fewer than 20 assessments, resulting in a total of ten WHRA and twelve Ontario sites. Of these, three Ontario CCACs and eight offices of the WRHA also submitted client reassessments at approximately 90 days, which allowed for the calculation of failure to improve/incidence HCQIs.
The protocol for data collection was reviewed and received ethics clearance through the Office of Research Ethics at the University of Waterloo, Canada.
Risk adjustment
Risk adjustment attempts to adjust for differences in client populations that may bias the HCQI rates. Organizations that provide care to more impaired clients will tend to have higher unadjusted rates, regardless of the quality of care they provide. As such, risk adjustment methods are used to maximize the ability to make fair comparisons across providers. With any type of risk adjustment, caution must be exercised to prevent over adjustment (i.e., adjusting away poor practice). The choice of risk adjustment covariates is therefore highly important and should not include variables that would be considered to reflect suboptimal clinical care.
There are two generic types of risk adjustment that have been recommended for the HCQIs. The first adjusts for differences in the population at the client-level[
10]. Potential adjusters can include both individual assessment items and summary scales embedded within the MDS-HC. The HCQI developers evaluated a large range of potential covariates, considering their distributional properties, strengths of association with the outcomes of interest, consistency of findings across jurisdictions and potential for clinically inappropriate adjustment (e.g., benzodiazepine use was not considered a reasonable adjuster for falls). As a result, from zero to five risk adjusters were recommended for the 22 HCQIs and these client covariates were used in the current project[
11].
The second type of risk adjustment was intended to control for two types of bias at the agency level: a) the agency's ability to identify differences in clients' clinical characteristics and b) differences in who an agency selects for admission. These risk adjustments are performed at the agency level after the individual-level risk adjustments are applied.
The Agency Intake Profile (AIP) was used following the methodology outlined by Morris et al., for the MDS 2.0 quality indicators[
7]. The AIP was calculated for each agency based on clients for whom the MDS-HC assessment was their intake assessment or for clients who had been on service for no more than 30 days. This group of clients was considered to be the intake cohort.
More recently, an alternative form of agency-level risk adjustment has been proposed, to control for potential selection and ascertainment bias. This adjustment employs the Case Mix Index (CMI) associated with the RUG-III/HC methodology[
9]. In this instance, the CMI for the intake cohort was calculated, and adjustments to the HCQIs were performed similar to those used for the AIP.
The process followed for adjusting HCQI rates was the same as that used by Morris et al. in adjusting the US nursing home QI rates[
7]. Three sources of information are required: the agency-level observed rate, the agency-level expected rate and the grand mean across all agencies. The first data element was simply the raw observed HCQI rate for a given home care agency. The next data element involved the creation of an expected rate for each client within a given agency, based on output from a logistic regression model. In this model, each HCQI acts as a dichotomous dependent variable (i.e., triggering on the HCQI or not) and the client-level and/or agency-level covariates are entered simultaneously as independent variables. The expected value for each client is then pooled to create an expected value for the agency. Three separate risk adjusted HCQI rates are thus computed, based on which method is used for adjustment: the expected rate based on the client covariates only (CC), on client covariates plus AIP (+AIP) and on client covariates plus CMI (+CMI).
The final adjusted value can be thought of as an estimate of an agency's HCQI rate if the agency had clients with an average level of risk[
7]. This risk adjustment method is similar to the concept of indirect standardization, in which the ratio of the observed to expected events is calculated then multiplied by the crude rate in the standard population[
12]. In the current project, the standard population used was the combined set of agencies from both the Winnipeg Regional Health Authority (WRHA) and from Ontario.
HCQI rates
For each participating agency, the unadjusted HCQI rates were calculated for all 22 indicators. These rates represent the average rate across all eligible clients for a given agency. For prevalence indicators, the rates were calculated only for clients who had been on service for at least 30 days to avoid penalizing an organization for quality issues that were newly recognized.
Several methods were utilized to assess the impact of risk adjustment, both at the regional and at the agency-level. The unadjusted and three adjusted rates were compared between Ontario and the WRHA, to assess the effects at the regional level. At the level of the individual agency, ranks were calculated for each HCQI and a count was created to examine how often a given agency was among those with the four highest rates. Since higher rates on each HCQI were indicative of higher prevalence or incidence of undesirable outcomes, agencies ranked within the four highest rates were considered to be among the "worst performers." The range between agencies with the highest and fourth highest rates was also examined for both unadjusted and adjusted rates to assess the degree of variation among the group of worst performers.
Discussion
Ontario CCACs exhibited several key differences when compared to the WRHA. Ontario sites had significantly higher unadjusted rates across four HCQIs. However, the degree of variation between the sites was small, with a mean difference across indicators of less than 1%. Ontario as a region had significantly higher AIP values for eight HCQIs, indicating a greater propensity in Ontario to admit clients exhibiting quality issues, and they also had a higher mean Case Mix Index.
Overall, the change in the actual rates at the regional level was small following risk adjustment. When risk adjustment was applied, for the client covariates alone or the client covariates with the CMI covariate, there was little effect on the mean difference between the two regions. In each case, the mean difference was positive (i.e., Ontario had a higher rate on average) and less than 1%. The +AIP adjustment, however, resulted in a negative mean difference (i.e., WRHA higher than Ontario on average) across the set of risk-adjusted HCQIs.
At the agency level, there was a greater influence of risk adjustment, as assessed by agency rankings across the set of indicators. In general, sites in Ontario benefited from risk adjustment and were less likely to be ranked among the worst performers. The opposite was true within the WRHA. Again, the +AIP adjustment had the largest influence when compared to the other two types of risk adjustment.
It is possible that the level of variation between agencies on the AIP covariate was greater than the level of variation between clients on the individual-level covariates. Although a detailed analysis is beyond the scope of this paper, use of a single HCQI example may prove beneficial. When examining the prevalence of falls (an HCQI that showed the largest changes in the agency rankings following risk adjustment), the coefficient of variation (CV) for the AIP value was 23.0. The CV is a relative measure of dispersion about the mean and is calculated by dividing the standard deviation by the mean and multiplying by 100. This CV value was much lower than any of the corresponding CV values for the various MDS outcome scales that reflect differences at the level of the individual client. For example, the CV for the Cognitive Performance Scale was 158.5 and for the ADL Self-performance Hierarchy Scale, the CV was 195.6 (data not shown).
Clearly, the +AIP adjustment had the effect of minimizing differences between the individual providers and it also resulted in rates that had less variability, as demonstrated by the drop in the CV value when comparing the unadjusted and +AIP adjusted rates for the prevalence of falls indicator (CV of 27.7 and 19.1, respectively; data not shown). However, this effect cannot be explained by differences in the variances of the adjusters since the AIP had a lower level of variance than the individual covariates.
It is also possible that the AIP covariate serves as a proxy for regional differences in practice patterns. If the Ontario agencies are grouped into four main geographic regions, the AIP value for the falls HCQI ranged from 24.1% to 34.0%. The degree of variation appears modest, but cannot be discounted as one factor in the ability of the AIP adjustment to minimize differences between providers and between regions.
Although there continues to be support for the conceptual notion of risk adjustment, the potential to over adjust remains a concern. In a recent publication, Mor et al[
15] recognize the possibility for over adjustment and conclude that in the absence of a simple solution to this problem, researchers must carefully consider each performance measure on an individual basis in an attempt to minimize this issue.
In the current project, the AIP covariate represents a continuous, numeric value that can range from zero to one. Thus, it does not simply serve as an agency identifier, but represents the rate among an intake cohort. Furthermore, in the US, Morris et al. determined that the Facility Admission Profile, the conceptual cousin to the AIP, did not substantially improve the risk adjustment models and they did not recommend its use. This decision was not based on a fear of over adjustment, but rather a concern that the FAP added increased complexity without significantly improving the risk adjustment process[
7].
It is also important that to develop a clearer understanding of what is meant by "over adjustment". For example, it would seem reasonable to differentiate between instances of truly inappropriate risk adjustment where variance is due to poor practices (e.g., adjusted for benzodiazepine use for a falls indicator) and "over adjustment" due to the excessive use of spurious risk adjusters resulting in suppression of variance. Another possible example of over adjustment is the use of individual level covariates that are too closely related to the quality indicator. For example, one might argue that using dressing of the upper body as a risk adjuster for a QI on dressing the lower body is a form of over adjustment because the dependent variable is represented on both sides of the regression equation.
This area of research continues to present many challenges. Given the current results and those from the long-term care sector, it appears advantageous to undertake some form of risk adjustment, even though the definitive method may not yet be best understood. Given that these HCQIs are new and so little is known about the risk adjustment process, it seems appropriate that all three type of risk adjustment be considered by researchers and policy makers. The +AIP adjustment represents the most conservative approach and may therefore be most appropriate for public reporting of HCQI results.
The assessment of quality of care, and ongoing refinement of risk adjustment methods, is ultimately intended to provide information to different audiences to assist in continuously improving the quality of care. To date, many initiatives in the US have taken place to provide additional information on the quality of LTC and home health services[
6]. Similar efforts have not begun in the Canadian home care sector.
Several important issues should be addressed prior to moving towards more public reporting of these types of quality data. For example, there needs to be a discussion of the relative importance of the HCQIs. The current project made no attempt to prioritize the indicators, although clearly some would have higher clinical priority than others. To some degree, individual agencies must determine their own priorities. However, a simple system based on prevalence, severity and modifiability may be useful in determining relative importance. For example, pain is a prevalent condition in this population, can be severely debilitating for clients and is clearly modifiable. One might argue that it is of higher importance to address than declining cognitive performance, which is less prevalent and in many cases not modifiable. On the other hand, cognitive decline can have extremely serious consequences for the individual
In addition, a decision is needed regarding the calculation of the HCQIs, in particular, which pairs of assessment are to be used in the calculation of failure to improve/incidence HCQIs and what time period in between assessments is acceptable (e.g., a minimum of 90 or 120 days). There will need to be further analysis to explore the relationship between the length of stay and triggering on the HCQIs. It is anticipated that clients who have been on service longer will show important clinical differences from short-stay clients and would therefore be expected to have differential HCQI rates.
Tracking of the HCQI rates over time will be essential to determine the level of stability in the indicators. It would be useful to measure the amount of variation and to assess methods to summarize the rates over time in order to maximize their stability. For example, it may be important to report annual HCQI rates for a given organization or province if in fact the rates are found to be highly variable over shorter periods of observation. A shorter time frame may also lead to unstable rates if the number of observations is small. In this paper, a minimum of 20 observations was required and a decision would also be needed for the minimum sample size for public reporting of HCQIs.
Finally, it will be essential to assess possible methods to create summary measures across the set of HCQIs. This is not a simple task and one for which little research has been conducted to date. Previous research has pointed to the lack of correlation between measures of quality, but has also suggested that summary measures would be useful for consumers who would likely prefer less complicated information[
15].
Knowledge about the home care sector in Canada remains rudimentary, and our knowledge and understanding of quality assessment in this sector is still in its infancy. The results from the current project serve to enhance the overall understanding of the issues involved in quality assessment and our ability, through the risk adjustment process, to create fair comparisons across providers.
Competing interests
The author(s) declare that they have no competing interests.
Authors' contributions
DD participated in the coordination of the study, assisted in the conceptual design of the study, carried out the data analysis and took the lead in developing the manuscript. JH designed the original study, oversaw the development of the indicators, gave input into the data analysis and development of the draft manuscript. BF was the Co-Principal Investigator in the development of the indicators and provided feedback in terms of data analysis and choice of statistical methods. All authors read and approved the final manuscript.