Where’s the evidence? a systematic review of economic analyses of residential aged care infrastructure

This review was conducted in accordance with the Joanna Briggs Institute (JBI) guidance for the systematic review of economic evaluation evidence [30]. A protocol for this systematic review was registered with the PROSPERO International Prospective Register of Systematic Reviews on 30 January 2015 (http://​www.​crd.​york.​ac.​uk/​PROSPERO; registration number CRD42015015977).

Eligible studies included full economic evaluations (e.g. cost-effectiveness analyses, cost-utility analyses, cost-benefit analyses), partial economic evaluations (e.g. cost analyses, cost minimisation analyses, cost consequences analyses), and randomised trials reporting more limited information, such as estimates of resource use or costs of interventions, pertaining to organisational and environmental characteristics aimed at improving the quality of care for older adults in a residential aged care setting. Organisational characteristics related to the overall business structure of the aged care provider, and included attributes such as demographics, proprietary status, size, and affiliation. Environmental characteristics referred to the physical setting and included tangible attributes such as private rooms, access to outdoors, familiar home-like components, and secure units.

Eight electronic bibliographic databases were searched from inception to 8 October 2014, including AgeLine, the Cumulative Index of Nursing and Allied Health Literature (CINAHL), Econlit, Informit (databases in Health; Business and Law; Social Sciences), Medline, ProQuest, Scopus, and Web of Science. An update search was run on 14 December 2015.

The search strategies were developed and reviewed with the assistance of two Health Sciences Librarians with expertise in systematic reviews. The strategy combined terms relating to nursing homes, economics, and older people, limited to English language. No study design or date limits were imposed on the search. The full search strategy is available on PROSPERO (http://​www.​crd.​york.​ac.​uk/​PROSPEROFILES/​15977_​STRATEGY_​20150030.​pdf).

Due to the large number of results retrieved when searching the multidisciplinary database ProQuest, limits to source type (scholarly journals, reports, dissertations and theses, conference papers and proceedings, and working papers) were applied to this database that were not part of the original search strategy. Newspapers, trade journals, wire feeds, magazines, other sources, books, and encyclopaedias and reference works were excluded.

Titles and abstracts of studies retrieved were reviewed in full by the primary author (T.E.). A second reviewer (see Acknowledgements) independently screened 10% of the titles and abstracts (L.P.L). Full text reports were retrieved for all citations that appeared to meet the inclusion criteria. All full text reports retrieved were reviewed independently by the primary author and second reviewer (T.E. and L.P.L.). Disagreement was resolved through discussion and consultation with a third reviewer (R.M.). Reasons for excluding studies were documented. The reference lists of included studies were hand searched for additional studies by the primary author (T.E.).

The JBI Data Extraction Form for Economic Evaluations was used to extract data from the included studies (http://​joannabriggs.​org/​assets/​docs/​jbc/​operations/​dataExtractionFo​rms/​JBC_​Form_​DataE_​EconEval.​pdf) [31]. Standardised data items extracted included descriptive data about the study and analysis including (i) study population/participants, intervention, comparator(s) and outcomes; (ii) study methods including prices and currency used for costing, time period, sensitivity analyses and measures of resource use; (iii) study context (geographical, health care and broader service delivery setting and culture); (iv) analysis methods. Results for the resource use and/or cost and/or cost-effectiveness measures and the author conclusions were also extracted. The primary author (T.E.) extracted all data. Neither the study selection nor the data extraction was blinded.

Critical appraisal of studies was undertaken using the JBI Critical Appraisal Checklist for Economic Evaluations (http://​joannabriggs.​org/​assets/​docs/​critical-appraisal-tools/​JBI_​Critical_​Appraisal-Checklist_​for_​Economic_​Evaluations.​pdf) [31], adapted from the Drummond checklist [32], which addressed: the study question; description of alternatives; identification of costs and outcomes; establishment of clinical effectiveness; accuracy, credibility and timing of costs and outcomes; incremental analysis; sensitivity analyses; and generalizability. Studies were rated as ‘yes’, ‘no’, or ‘unclear’ in terms of their compliance with each quality criterion in light of the objective of the study. For instance, a study which was designed to focus only on costs would by definition not have considered outcomes and so it may still score a ‘yes’ on item 3 which considers whether all relevant costs and outcomes have been identified. A study which was designed as a full economic evaluation on the other hand would need to identify both costs and outcomes to meet this criterion. As the search strategy did not impose date limits, the purpose of this appraisal was not to exclude studies that pre-dated the use of current economic evaluation methods. Rather the purpose of appraisal was to identify methodological issues with the study design that may result in biased measures of cost and/or effect in order to inform the interpretation of study results.

The JBI Critical Appraisal Checklist for Economic Evaluations was chosen for the current study as it covers the same ten items as the Drummond checklist with the addition of an eleventh item which addresses the generalizability of results to the setting of interest for the review [31]. The appraisal was conducted by the primary author (T.E.) and ratified by a second reviewer (R.M.). Disagreements were resolved through discussion between the primary and secondary reviewer.

Data extracted from included studies were analysed and synthesized in a narrative summary to address the stated review objective. Synthesis included (1) key findings pertaining to organisational and environmental characteristics aimed at improving the quality of care for older adults in a residential aged care setting; (2) a review of approaches taken to include health and quality of life effects in the identified economic analyses; (3) a review of approaches taken to include residents with dementia in the identified economic analyses; and (4) consideration of key methodological issues for consideration in the future design and conduct of economic evaluations of residential aged care infrastructure. This review was prepared in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [33].

The study selection process is presented in Fig. 1. The electronic database search yielded a total of 23,059 citations; an additional 4 citations were identified through searches of reference lists of included studies. A total of 14,012 unique citations were identified after duplicate removal. After title and abstract screening 13,809 records did not meet eligibility criteria and 7 studies were excluded as the full texts were not available. Full text reviews were conducted for 196 articles and 14 studies, from 16 publications, met the inclusion criteria. The unit of analysis for the purpose of this review was the study, rather than individual publications. We report the findings of this review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [33]. The completed PRISMA checklist is included in Additional file 1.

Table 1 presents the main characteristics of studies included in the review. All 14 studies contained a partial economic evaluation in the form of a cost analysis. None of the identified studies undertook a full economic evaluation in the form of a cost-benefit, cost-effectiveness, or cost-utility analysis. The majority of studies (n = 13) were evaluated from an institutional perspective, and only costs occurring within the facilities themselves were considered. Two of the studies were specific to residents with dementia, in which all residents participating in the study had a recorded diagnosis [34, 35].

Ten of the studies evaluated specific organisational characteristics, while four focused on environmental characteristics. The most frequent study design was cross-sectional (n = 11). Other study designs included a cluster-randomised controlled trial (n = 1), cross-sectional time series (n = 1), and prospective cohort (n = 1). Twelve studies pertaining to organisational characteristics were undertaken in the United States with cost data from large data sets collected during the 1970s and 1980s. Only two studies were conducted outside the United States: one study conducted in Switzerland using cost data for the period 1993–2001 [36] and one study conducted in the United Kingdom during 1990–1992 [37]. Three of the studies evaluating environmental characteristics were conducted in the United States [35, 38, 39], while the fourth was conducted in Australia [40].

The number of participating facilities per study ranged from 1 to 3,492 (mean: 424; median: 150). Of the three studies that recruited resident participants, sample sizes varied widely (n = 44 [35]; n = 601 [40]; n = 2,405 [37]). The 11 studies that did not recruit resident participants collected facility-level data only, such as operating costs or staff time. Clinical outcome measures - defined as outcomes involving measurable changes in a resident’s health or quality of life - were reported in 3 studies (across 4 articles) [40‐43]. A summary of main clinical outcomes for the 3 studies are reported in Table 2 and include measures of agitation, quality of life, social interactions and behaviour, cognitive status, function, and a composite measure of poor quality based on rates of decubitus ulcers, catheterization, physical restraints, chemical restraints, and drug errors.

Interventions reported in studies pertaining to organisational characteristics fell into four broad categories: proprietary status, affiliation, size, and location.

Interventions reported in studies pertaining to environmental characteristics fell into two broad categories, functional modifications and home-like environments.

Table 3 presents the results of the assessment of methodological quality of the included studies. The methodological quality of included studies varied widely and a high level of uncertainty was found in the generalisability and transferability of findings. The primary methodological issues identified included: a failure to establish clinical effectiveness in any of the studies, the absence of incremental analysis of costs and consequences in all studies, and a lack of sensitivity analyses to assess the robustness of the base case results to variations in key parameters. Although no studies established clinical effectiveness, two studies (in three articles) did provide effectiveness estimates for the interventions conducted [40, 42, 43], while eight studies utilized clinical or process-related outcomes or observable qualities such as staff-to-resident ratios as markers for quality of care [36, 37, 41, 45‐49].

Further methodological issues were identified in relation to the reporting of resource use and costs. Four studies reported mean costs but did not provide a measure of variation or dispersion in the cost results (e.g. standard deviation) [35, 38, 40, 49], two studies did not fully disclose the source of their cost data [35, 47] and two studies did not disclose the date for their cost data collection [35, 38]. Out of ten studies addressing organisational characteristics, only one study reported on resource use, reporting mean staff time per resident per week [37]. Similarly, of the four studies relating to environmental characteristics, only one study reported resource use which was reported in the form of room size measurements [38].

In terms of organisational factors, the available literature suggests that for-profit facilities operate at lower costs than not-for-profit and government-owned facilities, while hospital-based facilities may have lower running costs than free-standing facilities. It is important that these results be interpreted with caution firstly because the cost data presented in these studies are dated, having been collected between 1976 and 1989. Secondly, all but one of the studies addressing proprietary status and affiliation were conducted in the United States and therefore their transferability to other aged care systems around the world is unclear. That being said, the value of investigating the cost-effectiveness of organisational characteristics should not be dismissed. While the evidence pointing to cost differences may be dated, there is current literature which identifies variation in outcomes based on organisational factors. For instance, for-profit facilities have been associated with higher staff turnover [54, 55], lower nursing staff levels [55], and lower quality care overall [56]. Given the available literature indicating differences in both costs and effectiveness, future research which aims to link quality measures with cost data for differing proprietary status may provide insight into questions such as whether additional resources allocated in a not-for-profit organisation are producing better outcomes, or if perhaps these organisations are operating less efficiently.

There is a paucity of evidence regarding the impact of location or size on the running costs and cost-effectiveness of residential care facilities. Our review found only two studies related to locality and one study which investigated facility size and thus it is difficult to draw conclusions. There have been a number of studies, however, which have looked at associations between these organisational factors and clinical outcomes. For instance, in a study investigating the use of feeding tubes among residents with advanced cognitive impairment, residents living in urban facilities and residents living in facilities with more than 100 beds were found to have an increased likelihood of having a feeding tube despite empirical data suggesting that feeding tubes are not beneficial in this population [57]. Facilities with more than 100 beds have also been linked to higher staff turnover which has been found to be detrimental to overall quality of care [55]. In light of evidence which links quality outcomes to size and location, future economic evaluation studies are warranted.

The body of evidence examining the impact of the physical environment on people with dementia has been well documented, and environmental design interventions have been shown to affect behaviour, function, well-being, social abilities, orientation, and care outcomes [14]. SCUs have been linked to lower hospitalisation rates [58] and lower likelihood of using feeding tubes [57]. However, economic evaluations of environmental characteristics and dementia-specific facility designs are scant; our review identified only four studies in this domain. Environmental modifications in the identified studies included homelike environments (e.g. single bedrooms, private bathrooms, decorating, and access to outdoors) and functional modifications (e.g. increasing visual access to bedrooms and activity rooms, extending activity spaces, and building partitions to reduce overstimulation). The economic evidence in this review indicates that environmental modifications come at an additional cost, but are weakly associated with better outcomes in the form of reduced agitation and improved social interactions. It is important for future studies investigating the effectiveness of a particular environmental intervention to conduct economic evaluations alongside these trials in order to build a more robust evidence base surrounding the value of investing in specialised designs.

One very prominent methodological issue that emerged from this review was the heterogeneous range of outcomes that have been used. Some of the direct outcomes measured included agitation, improved social interactions, quality of life, behaviour, function, well-being, depressive symptoms, quality of care, rates of decubitus ulcers, catheterisation, physical restraints, and chemical restraints. Other outcomes, which could be presumed to impact on health, included drug errors, number of regulatory deficiencies, skill level of persons in charge of nursing shifts, range of therapies provided, and number of people waitlisted. The development of guidance towards a more consistent methodology for economic assessment of residential aged care infrastructure is needed, specifically with the inclusion, where possible, of the health and quality of life benefits measured from the perspective of the residents themselves including people with dementia.

There have been numerous instruments developed to measure health benefits such as behaviour, function, well-being, care outcomes, and health-related quality of life, for example. Consequently, it is important for the chosen outcome to be an appropriate measure of achievement for the desired objective. For instance, the desired objective of aged care infrastructure may be to improve the quality of life for the residents who live there. The question then becomes what is the most reliable outcome measure to capture improvements in the lives of residents?

One approach may be to present an array of outcome measures for each alternative, allowing the decision-makers to make their own trade-offs between measures of effectiveness. This is commonly known as a cost-consequences analysis. Another possibility is incorporating a generic measure of incremental benefit, such as the QALY. The main benefit of utilising QALYs in this context would be their applicability to all aged care residents, which would allow decision makers to make comparisons across differing programs. Cost-utility analyses, which use QALYs as the outcome measure, are the recommended economic evaluation in national guidelines developed by government agencies in healthcare such as NICE in the UK [21], and the Canadian Agency for Drugs and Technologies in Health in Canada [59]. While these guidelines were developed for economic evaluations of health technologies, they could potentially be applied to aged care infrastructure, for instance where meaningful differences in health-related quality of life between the intervention and comparator have been demonstrated.

It may also be worthwhile to consider a social context, rather than a health context, as potentially more appropriate in a residential care setting. Current research has acknowledged factors outside of health status such as dignity, independence, and having control over their daily lives as important contributors to residents’ quality of life [60, 61]. A recent systematic review of instruments for measuring outcomes in economic evaluations within aged care recommends the use of a generic preference based measure of health related quality of life such as the EQ-5D to obtain QALYs in combination with an instrument with a broader quality of life focus to capture dimensions of social well-being, such as the Adult Social Care Outcomes Toolkit (ASCOT) designed to evaluate interventions in social care, or the ICEpop CAPability measure for Older people (ICECAP-O) which measures capability in older people [62]. Ultimately, it is important that the chosen method is sensitive enough to measure changes for this population, and broad enough to allow comparisons to be made at a service planning level.

Twelve studies identified by this review did not disclose whether residents with dementia had participated. While it is uncertain whether these studies included participants with dementia, the omission suggests that no consideration was given to this subgroup during study design. One study specified that residents were only approached to participate if judged by staff to be capable of self-completing the study questionnaire [37], which suggests cognitively impaired residents were excluded. When designing economic evaluations, it must be ensured that the study sample is representative for the population being assessed. The quality of an economic evaluation is highly dependent on the source of data used, and its ability to be transferred to other settings. In residential care settings, the exclusion of residents with dementia raises serious concerns regarding the representativeness of data given that over 50% of those residing in aged care facilities have a recorded diagnosis of dementia [7, 9].

In addition to the issues discussed surrounding the measurement of health and/or quality of life effects, and increasing the representativeness of data by ensuring the inclusion of residents with dementia, an important methodological issue to consider is study design. The common methodology used in the health care sector for implementation research is a cluster randomised design, as participant-level randomisation can introduce bias through exposure of the control group to the intervention [63]. Only one of the studies identified used a cluster randomised design [40]. Employing a randomised design to focus upon the impact of organisational characteristics is often not achievable in the aged care sector. It is not feasible to randomize attributes such as the proprietary status or location of an aged care facility. As shown by this review, observational study designs are much more practical in this setting. However, a cross-sectional study design, which was the most frequently used design in included studies, can identify associations but not causality due to the absence of a time dimension. Well-designed observational studies with a temporal dimension (i.e. prospective or retrospective rather than cross-sectional) have been shown to produce comparable results to randomised controlled trials [64, 65].

An alternative option when randomised controlled trials are not feasible or for extrapolating beyond the time frame of a clinical trial is decision modelling [66, 67]. Using a decision modelling approach, costs and outcomes can be predicted using data synthesised from disparate sources and models can be built to extrapolate long term estimates of costs and benefits. While none of the studies identified in this review utilised a decision modelling approach, this may be a viable direction for future research [67].

Transparency in reporting study methods and results is another area that is important when assessing the validity and reliability of economic evaluations. This is not specific to residential care or to infrastructure, but nonetheless an important consideration. A clear example is the cost analysis of special care units published in 1998 by Maas and Buckwalter [35] which failed to disclose the date the cost data was collected or whether costs were adjusted for inflation. The exact date of this study was not stated, though the first preliminary results were published in 1988 [68], 10 years prior to the cost analysis publication. Future economic evaluations in this area should strive to meet the quality standard for reporting economic evaluation as specified in the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement [69] including the quantities of resources used in addition to costs and incorporating the measurement and valuation of service outcomes and quality of life. Disclosures should also be included to indicate the timing of cash flows and the sources of cost data.

This systematic review had a broad scope in order to provide a comprehensive summary of the evidence, and as such we can be confident that we have captured the majority of studies on this subject. The main strength of this review was the systematic and transparent approach which, in combination with the breadth of the objective, allowed for a thorough synthesis of existing economic evaluations of residential aged care infrastructure. The review was conducted to a high methodological standard and met the quality standards set within the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. Critical appraisal of studies was undertaken using the JBI Critical Appraisal Checklist for Economic Evaluations which is a well-recognised and highly regarded Checklist for assessing the quality of economic evaluation studies previously utilised in other high quality systematic reviews published previously [70, 71]. However, the broad scope of this review, and the incorporation of economic evidence meant that it was necessarily time-intensive, requiring more resources for the search process, data extraction, and analysis compared with a narrow scope review. For pragmatic reasons, one author took responsibility for both the initial examination of all citations and for all data extracted from included studies, and as such it is possible that errors occurred. This review had limitations to analysis imposed by the heterogeneity of interventions, methods, and outcomes in the included studies. A meta-analysis was not possible; rather the review relied on a narrative analysis of the included studies. This is a reflection of the research that has been conducted to date, and again highlights the need for future evaluation research to be carefully planned such that the data collected and reported is useful for decision makers.