Evaluation of bottom-up interventions targeting community-dwelling frail older people in Belgium: methodological challenges and lessons for future comparative effectiveness studies

We developed a step-by-step methodological process combining different approaches. First, we divided the intervention group into intragroup-homogeneous strata to define different disability profiles. Second, we carried out a one-by-one matching of the participants from the intervention group with the control group to account for people with similar long-term care needs. Third, we compared healthcare utilization between the groups studied in order to evaluate the baseline difference in how widely those needs are met (low and high value).

The inclusion criteria for selecting participants in the intervention and the control groups included: [17] being at least 60 years old and scoring at least six on the Edmonton Frail Scale [18]; having a dependency status of A, B, or C on the Katz home scale or a B, C, or Cd status on the Katz residential scale [19]; or having been diagnosed with dementia by a geriatrician, neurologist, or psychiatrist. People in both groups were living at home at baseline.

Two databases were available and were merged with national registration numbers for this study.

The first is the BelRAI database which is the Belgian version of an internationally validated geriatric comprehensive assessment, the interRAI Instrument Home Care version (interRAI HC) [20]. This provided prospective data on the main clinical variables. Functional performance in activities of daily life (ADL) was evaluated by four items (personal hygiene, toilet use, locomotion, and eating) with a total score varying from 0 to 6. A total score above 3 (cut-off of ADL scale) indicates major dependency; 6 is the maximum level of dependency [21]. Functional performance in instrumental activities of daily life (IADL) was evaluated by eight items (meal preparation, ordinary housework, managing finances, managing medications, phone use, stairs, shopping, and transportation) with item scores varying from 0 to 6 and total scores varying from 0 to 48 [22]. A total score above 24 (cut-off of IADL scale) indicates major dependency; 48 is the maximum level of dependency. Cognitive status was evaluated by a cognitive performance score (CPS), which consisted of four items (decision-making, short-term memory, procedural memory, and comprehension). The total score ranges from 0 to 6; a total score above 3 (cut-off of CPS scale) indicates major cognitive impairment [23]. The depression rating scale (DRS), with total scores ranging from 0 to 14, was used to evaluate depression symptoms. A total score above 3 (cut-off of DRS scale) indicates significant depressive symptoms; a score of 14 indicates the presence of all mood symptoms in the last 3 days [24]. Behavioral problems (Behav.) were evaluated by the Aggressive Behavior Scale (ABS), which includes five items (wandering, verbal aggression, physical violence, abnormal social behavior, and resistance of care) [25]. The total score varies from 0 to 12 [25]. A total score above 0 indicates behavioral problems (cut-off of Aggressive Behavior Scale); a score of 12 indicates frequent and varied behavioral problems [25]. The level of presence of a family carer, the Zarit Burden interview, and the WHO-QoL-8 were added to interRAI Instrument in the BelRAI database. The level of presence of a family carer was encoded, according to living arrangements, as a categorical variable with three levels (without a family carer, with a non- resident family carer, and with a co-resident family carer). This categorization allows the differentiation of the intensity of informal caregiving provided [26]. The 12-item version of the Zarit Burden interview (ZBI-12) was used to estimate the burden on family carers. This scale contains 12 items, with scores from 0 to 4. The total score ranges from 0 to 48; higher scores are indicative of greater burdens on family carers. A total score above 10 (cut-off of ZBI-12 scale) indicates that a family carer has significant depressive symptoms [27]. The WHO- QoL-8 was used as a concise instrument to measure the client’s perceived generic quality of life [28]. Finally, an ad-hoc economic questionnaire was used to measure social care services utilization and informal aid. The interRAI HC instrument was filled out at enrolment, at the exit of the patient from the project, and 6 months after enrolment [15]. Only the baseline evaluation data were used for this paper.

Secondly, the IMA database from the Inter-mutuality Agency provided routine data on the healthcare services reimbursed by the National Insurance for Health and Disability Institute (NIHDI).

These two merged databases were cleaned to contain only complete evaluations at baseline. Data were available for 10,783 beneficiaries of Protocol 3 intervention (intervention group) and 605 older persons benefitting from “usual care”, who were recruited at home by nursing care organisations (control group). Of the beneficiaries of Protocol 3 intervention, 1811 lived at home and had no family carer, 5735 had a non-resident family carer, and 3842 had a co-resident family carer. In the control group, 78 lived at home and had no family carer, 268 had a non-resident family carer, and 259 had a co-resident family carer.

In addition, the median fiscal income per household by municipality (data from statbel.​fgov.​be) was used as a proxy for socioeconomic level. Belgian municipalities were classified in three categories by using the first and the third quartiles of the median fiscal income per household. Municipalities included in the first quartile, accordingly, have a low median fiscal income per household. The municipalities included between the first and the third quartile, have a medium median fiscal income per household. And the municipalities in the last quartile have a high median fiscal income per household.

In order to identify population groups with specific long-term care needs, we identified “natural” clustering of individuals with a similar disability profile in the intervention group, using statistical analysis to establish classification schemes [29].

The variables used for the establishment of classification schemes were inspired by existing classifications in which clinical scales (ADL, IADL, depression, and cognition) and comorbidity were included [30‐32]. These classifications highlighted the fact that cognitive and functional limitations fit the different disability profiles better than comorbidities [30]. Indeed, the need for long-term care services is associated with specific limitations [10] (i.e. significant IADL limitations are associated with needing household aid services (domestic help, home care worker, or meals-on wheels, etc.); significant ADL limitations are associated with needing personal nursing care; and significant cognitive impairment is associated with needing supervision [33]). Hence, the comorbidities were not included in our classification. A specific scale for behavioral problems was included in our classification because people with behavioral problems represent a population with specific health and social care needs, including the need for supervision by family carers and the need for respite care for those carers [34]; in addition, this population is likely to make less use of support services [34] and imposes a greater burden on family carers [35].

The “natural” clustering was, consequently, established by combining the different scores on functional limitations, cognitive performance, and the presence of behavioral problems in a Principal Component Analysis (PCA), followed by a cluster analysis (based on the PCA correlation matrix between the initial variables and the principal components) [36]. A hierarchical algorithm (Ward algorithm) was used to define the number of groups by the decomposition of the inertia of the cloud of points while minimizing the loss of information at each additional clustering. The number of groups obtained was the result of a trade-off between intra-group homogeneity and inter-group heterogeneity [36]. This method was employed using the FactoMineR package [37] in R. Disability profiles were presented by means of the descriptive statistics of the BelRAI scales. For each scale, the median and interquartile range, both for the total score and for each item, and the proportion of individuals above the total score cut-off were presented.

The control group was constituted by one-by-one matching of P3 beneficiaries with individuals from the control group, based on a similar level of presence of a family carer and maximum similarity in scores on the clinical scales, age, and sex.

The establishment of the control group followed a multi-stage process. The initial step consisted of two successive stratifications. First, intervention and control groups were stratified by the level of presence of a family carer to ensure similarity of the groups studied with regard to this variable. The rationale behind this initial step was twofold. First, older people with co-resident family carers were frailer than both those without family carers and those with non-resident family carers [38]. Second, the level of utilization of health and social services is associated with the level of presence of a family carer [39]. Within the intervention group, each group of participants with a particular level of presence of a family carer was, in turn, stratified by disability profile. Then a model of propensity scores (with binary dependent variable intervention vs control groups) was created for each stratum, following a step-by-step approach. Each step consisted in a forward selection of a variable among those obtained from the BelRAI scales (ADL, IADL, CPS, DRS, presence of behavioral problems) and the predisposing factors. A likelihood ratio test was then performed to determine whether there was a significant difference between the intervention and control groups with regard to the variable added. A new variable that significantly improved the model was retained. The matching function used was the one-to-one matching of the nearest neighbors with replacement to make it easier to find the best match for a P3 beneficiary among all the potential controls. This was mandatory in our case because of the smaller size of the control group relative to the intervention group. The Matching package in R was used for this purpose [40].

Finally, the evaluation of the covariate balance was carried out using the standardized mean difference (smd) and the analogous variance ratio (VR). To evaluate multiple covariates balance, the average standardized mean difference (SMD) and the Geometric Mean Variance Ratio (GMVR) were used [41, 42].

The cluster analysis highlighted five disability profiles (Table 1). The first disability profile (N = 2040) was called “low-level limitations (low limit.)”, and consisted of individuals with no significant functional limitation or cognitive impairment.

The matching balance was respected: all the average standardized mean difference were smaller than 0.25 and all the geometric mean variance ratio were smaller than 2, excepted for strata with a small number of Protocol 3 beneficiaries and a small number of potential controls (e.g., where there were only 29 beneficiaries and one control without informal caregivers and with functional, cognitive, and behavioral problems, or where there were 121 beneficiaries and 9 controls without an informal caregiver and with functional and cognitive impairment) (Table 2). That can be explained by the limited number of people living alone at home who had significant cognitive impairment.

The stratification of the study population is often presented in the literature as a crucial process in order to test the effectiveness of interventions and their adaptation to the specific needs of people within groups [54]. The five disability profiles created in this study identify groups with specific long-term care needs [30].

A control group with similar characteristics in term of disability and predisposing and enabling factors was established. However, the groups studied differ in terms of healthcare utilization. The study sheds light on the difficulties involved in combining disability profiles and healthcare utilization to define the control group, in the actual context of available Belgian data.

Our results show that a database of healthcare reimbursements is not a good proxy for disability profiles (which make possible the identification of needs for long-term care). Not only are these data not specific enough to differentiate between disability profiles, but people with similar disability profiles do not necessarily utilize healthcare in similar ways.

A partial explanation of this difference could be the occurrence of acute health problems versus a progressive deterioration of health status. In our case, hospitalization within the 2 months before inclusion in the cohort has been used as a proxy for deterioration of health. This deterioration could mainly result either from acute problems for which the patient was being hospitalized [55] or from “iatrogenic disability”, defined as “the avoidable dependency which often occurs during the course of care” [56]. The disability profile of older people without recent hospitalization was considered to be the consequence of a progressive deterioration and not the result of a recent acute problem. For people with such a profile, differences between the intervention and control groups could be viewed as underutilization of the healthcare system or as mismanagement of people’s health problems.

Another explanation could be the social characteristics. Anderson’s socio-behavioral model of health service utilization proposes a series of predisposing factors alongside age and gender as social structure components, the health beliefs and a series of other enabling factors alongside the living arrangements of family career as the community ressources, the organization of health and social care, the knowledge of the services, the autonomy in their utilization, the income, the supplementary health insurance,... These factors should be integrated to the analysis to better understand the utilization and the need for health services [47]. However, these variables are not available.

To obtain more accurate information on disability profiles and more generally on health status, Electronic Health Records (EHR) may be of use in the future. However, in Belgium, these are not yet used by every practitioner and therefore they are not routinely available for scientific research [57]. The whole system is actually in development (EHR and pooling of information for population health and scientific studies), in Belgium. It is nevertheless not expected to include information collected by social worker. Indeed, in Belgium, due to a split decision-making power between the federal authorities and the federated entities [46, 58] the organization and the funding of formal social services are separated from health care services. These information are even though essential to better support frail older persons at home.

If the objective is to assess the effectiveness of an intervention, the easiest way to evaluate it is to compare the intervention group to a control group with similar baseline characteristics (in our case, with similar long-term care needs) [59]. As discussed earlier, the baseline characteristics must include disability profiles and healthcare utilization. Indeed, a similar disability profile allows us to evaluate the impact of the intervention on the evolution of clinical outcomes and on long-term care needs. On the other hand, similar baseline healthcare utilization facilitates the evaluation of the consequences of the interventions for how widely long-term care needs are met and for reducing the use of low-value healthcare services. Such a reduction can be interpreted as a decrease of the pressure on the healthcare system.

As shown in this study, some proxies for healthcare utilization appeared to be associated with some disability profiles. However, this association was not clear enough to allow the determination of disability profiles from healthcare utilization. So, in our study, we were not limited by the “technical” capacity to use variables in the control group (indeed, the propensity score method does not limit the number of control variables [60]). The limitations of this study are related to the availability of individually measured variables in the control group.

There were two difficulties related to the scarcity of the variables required to create an “ideal” control group. First, people in the intervention group were recruited through different entry points, including at discharge from hospital, from the waiting lists of the nursing homes, and from other sources. People in the control group were recruited from among those receiving nursing care at home, for organizational reasons. The different recruitment points constituted an important potential for a selection bias. However, they make it possible to appreciate the importance of integrating healthcare utilization into the discussion about unmet long-term care needs. Second, the number of controls was considerably smaller than the number in the intervention group, due to resource constraints. The method of one-by-one matching of the nearest neighbors with replacement that was used in this study accounted for the replication of certain controls. Other statistical approaches were considered, including bootstrapping on the matching or changing the ratio of Protocol 3beneficiaries to controls. However, these alternatives did not entirely resolve the scarcity of some profiles within the control group. The best solution would have been to include more controls from diverse entry points, which would have demanded considerable time and financial resources. Another option would be to exclusively use more diverse routine data (such data is either not yet available in Belgium or else links cannot yet be made between such data) to establish the control group. Indeed, routine data are recognized as the best tool for evaluating large-scale interventions. They are recommended in comparative effectiveness studies [61], as they are available for the whole population. The use of routine data also makes it possible to study real-world effectiveness [62, 63] and helps in policy- and decision-making, both through prediction of future needs and comparison with other public programs [64].

In Belgium, routine data provides information on reimbursed healthcare consumption, including healthcare utilization and proxies for comorbidities, but not disability profiles. These data can therefore help track the utilization of healthcare services over time, but they give no idea of long-term care needs and the coverage of those needs. Nevertheless, Belgium is trying to reorganize its health system and hopes to draw on the expertise gained from Protocol 3 projects to take a step forward in the implementation of projects designed to provide integrated care for people with chronic health problems [58]. These integrated care projects should involve a comprehensive reorganization of healthcare and social services, as well as the pooling of resources and policy adjustments. Such projects can only be successful if, at the same time, appropriate tools for data sharing are developed. These tools may be used for assessing the needs of patients and for evaluating healthcare utilization (by using population data). Clinical scales may help to better characterize disability profiles.