The calculation of quality indicators for long term care facilities in 8 countries (SHELTER project)

Quality of care is a complex, multi-dimensional concept. The US Institute of Medicine defines quality of care as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes which are consistent with current professional knowledge” (http://​www.​iom.​edu). There is interest in the creation of performance indicators that can measure quality by examining the structure, process, and outcomes of care. One method of identifying potentially good and poor professional quality of care is the use of quality indicators, which can be defined as “markers that indicate either the presence or absence of potentially poor care practices or outcomes”. The aim of quality indicators use is therefore to identify the clinical areas that can benefit from improvement of the care process and to define performance of individual care providers [1].

Quality Indicators (QIs) for monitoring quality of care in nursing homes have been developed using assessment data from the widely implemented Resident Assessment Instrument (RAI) for Long Term Care [2‐4]. Routine monitoring of these QIs led to QI reports being used for best practice comparison between nursing homes. A study commissioned by the US Centers for Medicare and Medicaid Services (CMS) demonstrated that the items from routine use of the RAI for Long Term Care in US Nursing Homes are reliable and that they can be used for the stimulation of improvement of care and reporting to the general public [5, 6]. For most of the QIs some risk adjustment is necessary to allow useful comparison of them between facilities [7, 8]. Although the relationship between outcomes and good and bad care practices were not equally strong for all available QIs, 10 QIs had a good enough relationship between identifiable pro-active and responsive care practices. These QIs have been selected by CMS for periodically public reporting at facility level.

A four step approach was used in the CMS commissioned development and validation of QI’s for nursing homes.

In two papers that came out of the European AgeD in HOme Care (ADHOC) study [9], the calculation of QIs for Home Care was explained and discussed [10, 11].

In this paper, similarly, we aim to explain and discuss Quality Indicators for Long Term Care Facilities (LTCFQIs) based on interRAI LTCF assessments [12] and their calculation. To do that we specifically show results of the calculation at country level in the 8 countries participating in the European Services and Health for Elderly in Long TERm care (SHELTER) study [13].

The adjusted LTCFQIs by country are shown in Table 2. For each LTCFQI, the best care (lowest LTCFQI score) country is shown light (green) and the worst care (highest LTCFQI score) country is shown dark (red). There is wide variation in the results of some of the LTCFQIs. For example, the LTCFQI “High Risk Behaviour problem prevalence” shows a range from 22% (France) to 61% (England) and the LTCFQI “Physical restraint use prevalence” a range from 1% (England) to 32% (Israel).

Additional file 1: Table S1 shows the aggregate LTCFQI measure for each country. The Czech Republic and Israel have by far the lowest summary scores of 5 and 6 (‘best’ quality of care), England the highest score of 33 (‘worst’ quality of care). The aggregate scores of the other countries are in between.

The LTCFQI scores of the 59 facilities can be calculated and presented in the same way. The outcomes for facilities, even in one country, show large differences. For example, the four facilities in Finland that participated in the SHELTER study produced the following results:

In this paper we have explained and discussed the methods for calculating Long Term Care Facility Quality Indicators (LTCFQIs). We used the method to calculate the values of 39 such indicators on assessments from 59 LTC facilities in 7 European countries plus Israel that participated in the Services and Health for Elderly in Long TERm Care (SHELTER) project [13].

Even if the calculations are appropriate, see discussion below, the results of the measurements cannot be considered representative for the eight countries. The recruitment in the SHELTER project was for cost reasons not aimed at representative samples for each country, only a limited number of facilities in each of the countries therefore have participated. The results of individual facilities in a country can be quite different, as shown, as an example, for the four facilities from Finland. Our intention with this paper is to show that professional quality of care can be calculated and compared between locations. To obtain valid comparisons between countries much larger samples of facilities per country need to be included. It is possible, however, that some of the outcomes per country will persist when samples are larger, as is predicted by the experts of LTCF that co-authored this paper.

For the moment, the differences between individual facilities, within and across countries, are probably much more real. If so, what does that imply for an individual facility? This study enables a benchmark on quality of care. Feedback of the results can be used effectively to improve the quality of care in a facility, as has been shown by Boorsma et al. [14] in a RCT study. To achieve this, demands considerable efforts: continued collection of good data, training, routine use of the assessment outcomes in care planning, continued interest by the management of the facilities. The work environment of a Long Term Care Facility is complex with its 24 × 7 hours of care, work shifts, various disciplines with their own manners, interaction with family and volunteers, complex and emotional care demands, budget restraints. Besides, the real effort to improve quality of care needs to be made on the level of the wards, each with its own more or less independent management, work relations, manners and specific groups of residents. Most LTCFQIs, however, cannot be reliably calculated for wards, because of limited nominator and denominator numbers. Additional analysis by management of the facility, therefore, is required to identify problems with quality of care on particular wards.

The validity of the measurements of the LTCFQIs depends first of all on the accuracy of the assessment. In our study the assessors were adequately trained at the start, so that we may assume that the baseline assessment was fine and comparable between facilities in all of the countries [13]. At the time of the second assessment, approximately 6 months later, and the third at twelve months, knowledge may have decreased or increased. Facilities where assessments were completed by nurses that worked in these facilities, may have employed new nurses that possibly have not received the same level of assessment training. Facilities where the assessments have been used in routine care planning, as intended, and where computer output was distributed timely and where the results have been thoroughly discussed, nurses and their care teams have become more experienced and better at recognizing issues that are assessed with the interRAI LTCF instrument. In other facilities, where nothing or less than was intended has been done with the results of the assessment, the opposite can be true. It is even possible that in the latter facilities the second and third assessment have largely been copied from printouts of the baseline assessment. We suspect that some of the incidence LTCFQIs outcomes may have suffered from this. Continued good quality of the assessments needs firm external and internal incentives, as has been shown in the US and Canada where interRAI instruments now have been used on a large scale for some decades [19].

The calculation of QIs presumes that the facilities and their resident population are to a large extent comparable, i.e. in level of care, kind of care, kind of residents, focus of care. This has not necessarily been so in the SHELTER study. The person’s living arrangement at the time of referral varied. In France, 45 percent of the residents came from a rehabilitation hospital. In England and the Czech Republic, most of the residents came from an acute hospital. In Italy, Israel and Germany the majority came from home. The number of residents in various Resource Utilization Groups, and length of stay [not presented in this paper], show that in some facilities (e.g., in Israel) most residents received extensive rehab, or stayed comparatively short (e.g., Czech Republic). Some of this is remedied by the risk adjustment, but likely not all.

In the calculation of the aggregate LTCFQI measure each QI has the same weight and the 75-th and 90-th percentile thresholds are equally important for each QI. These assumptions are obviously flawed. For some quality of care issues a number of QIs are available and for others maybe none. It is known to the authors that efforts are being made by interRAI to develop a more balanced aggregate measure on a subset of the QIs, including adding new QIs to the existing set that look at decline and improvement on particular issues simultaneously. Furthermore, the 75-th and 90-th percentiles are often useful, although not equally meaningful for each QI [3], but this has not been verified for all of the QIs, presented in this paper.

A further remark on the calculation of the QIs concerns the frequency of the assessment and the Facility Admission Profile risk adjustment. In the original research for the development of the QIs [5, 6], residents were assessed at least every 3 months. In SHELTER, follow-up assessments were scheduled at 6 and 12 months. This affects the results of the incidence QIs, but probably not a large extent. The FAP values, however, which are used for risk adjustment, could in this study only be calculated when a resident was assessed at entry to the facility at baseline. This was rarely so, since the facilities were selected that had long-stay residents, which by definition have few new entries in a short period of time. And even so, risk adjustment through FAP values is not perfect.

Quality of life as experienced by the resident is as important as the calculation of professional quality of care and should, when possible, be measured and analyzed additionally to evaluate services delivered and prepare for enhanced services.

To conclude, the LTCQIs are useful measures of professional quality of care in Long Term Care facilities. The indicators are based on the worldwide used interRAI LTCF instrument. When the conditions for measurement are met the indicators appropriately measure quality of care. We have shown how the facilities in the SHELTER study, and to a lesser extent the countries, can be compared with each other with the LTCFQIs.

We calculated the LTCFQIs as follows. We wrote SPSS syntax coding for the indicators, listed in Table 1. We executed this SPSS syntax on the SHELTER data. This generated for each QI an individual value for each resident, SPSS logistic regression analysis output tables for each of the LTCFQIs and risk-adjusted predicted QI values for each SHELTER resident. We transferred per facility (/country) the N of valid QI scores for a QI, the mean value of that QI, and the mean predicted value of the QI score (as derived from the logistic regression analysis), when applicable, to a MS EXCEL application. There we calculated the QI scores per facility (/country). When a QI score needed to be adjusted, we used the formula:

where LTCFQI_x,y is the score for LTCFQI x for facility (/country) y, and predicted_x,y is the mean value of the predictions of the values of LTCFQI x for all individual residents of the facility (/country) y as calculated by the SPSS logistic regression analysis.