People with dementia in nursing home research: a methodological review of the definition and identification of the study population

Health care service research aims to evaluate care strategies by obtaining an understanding of the causal factors in improving dementia-related outcomes for individuals living in nursing homes [1]. It is self-evident that a valid method to ascertain a case is a prerequisite for any study [2]. Case ascertainment strategies must rely on an established definition, and diagnostic criteria must be both valid and feasible to conduct.

Dementia may be defined as a clinical syndrome of mental capacity characterized by a substantial global decline in cognitive function that is not attributable to altered consciousness; it consists of a combination of symptoms attributable to various causes or pathological events [3]. Following the Diagnostic and Statistical Manual (DSM) 5th edition [4], dementia is subsumed under the entity of neurocognitive disorders (NCD). The diagnostic criteria for NCD are as follows:

Evidence for the cognitive decline should be based on concerns from the individual patient, a knowledgeable informant, or the clinician regarding the presence of a significant decline in cognitive function and a substantial impairment in cognitive performance; ideally, this evidence will be documented by standardized neuropsychological testing. The DSM-V manual notes that both concern and objective evidence are required to establish a diagnosis because they are complementary. If alterations in cognition are only tested utilizing objective measures, a disorder may be underdiagnosed in individuals who demonstrate high-functioning performance and score in the “normal” range but are still substantially impaired relative to their baseline performance. On the other hand, an exclusive reliance on subjective symptoms may result in underdiagnosing individuals who deny or fail to express their impairments. In either case, it is essential to interpret results in comparison with the individual patients’ prior performance.

The International Classification of Diseases (ICD) defines dementia as a decline in both memory and other cognitive abilities such as deterioration in judgment and thinking or the general processing of information. The decline in memory and cognitive abilities should be objectively verified by obtaining a reliable history from an informant and supplemented by neuropsychological tests or quantified cognitive assessments. The diagnosis of mild impairment is based on the degree of memory loss that interferes with everyday activities. A diagnosis of dementia requires, in addition to the previously mentioned symptoms for mild impairment, a decline in emotional control or motivation, changes in social behavior, and the absence of delirium. Furthermore, the described symptoms must be present for at least 6 months [5].

The diagnostic process follows two steps, starting with the initial recognition of the dementia syndrome and ending with the specification of an etiological subtype. The diagnostic criteria for each of the subtypes are specified in the DSM-V, ICD-10 [4, 5] and publications from different disease-related societies or national institutes (e.g., the National Institute of Neurological and Communicative Disorders and Stroke, NINCDS) [6‐10]. The subtype differentiation may require biomarker diagnostics such as genetic, blood and liquor testing or diverse imaging procedures.

For more than 3 decades, the history of classification systems for mental disorders has reflected both many changes and a relative emphasis on phenomenology, etiology, and course as defining features. When the DSM-III’s (1980) explicit diagnostic criteria were introduced (Feighner-criteria), several revisions followed: the DSM-III-R (revision) in 1987, the DSM-IV in 1994, and the DSM-V in 2013. The ICD introduced diagnostic criteria in its revised version 9, ICD-9-CM (for clinical modification). Every revision aimed both to reduce inconsistencies and to improve clarity and precision. Different versions included changes in the classification, diagnostic criteria sets, and descriptive texts [11].

Dementia diagnoses pose several challenges. First, the lack of valid biomarkers forces healthcare providers to diagnose dementia based on the presence of specific symptoms and the elimination of other conditions with similar symptoms. To arrive at a clinical diagnosis of dementia, a combination of basic assessments,—including physical, psychopathological and basic neuropsychological examinations—are recommended [12, 13]. For the basic neuropsychological diagnosis, different short screening tests such as the Mini Mental Status Examination (MMSE) [14, 15], the DemTect [16] and the Clock Drawing Test [17] are recommended. In clinical practice, these tests should be administered to every patient with dementia or suspected dementia on a regular basis both to quantify cognitive impairments and to enable providers to supervise the progression of the disease. Comprehensive neuropsychological diagnostics are necessary either if diagnostic findings are not congruent in the early stage of dementia or if an etiological classification is required. A dementia diagnosis should not be based solely on one neuropsychological test but instead must be based on both behavioral symptoms and impairment assessments related to daily function. Additionally, blood tests and imaging diagnostics should be conducted to assure both the diagnosis and its etiology [18].

The reliability, validity and utility of the new diagnostic criteria have discussed by clinicians and researchers since the date of their release; furthermore, a consensus about how to best diagnose dementia has still not been found [19, 20]. One result of this controversy is the methodological differences in epidemiological studies that lead to discrepancies in case ascertainment and varying incidence and prevalence rates [3, 21]. Another limitation of the generalizability of study results on dementia in nursing homes originates from the methodological difficulties of assessing the disease in the oldest old population. The normative data necessary to determine “impairment” for this population are lacking and therefore, aging is constitutionally associated with a decline in both cognition and function. Consequently, pathological processes are difficult to distinguish from normal aging; in addition, alterations in function and sensory impairments may both affect subjects’ cognitive performance and limit an accurate assessment [20].

The quality of a dementia diagnosis in German nursing homes is suboptimal: many studies of nursing home residents show both vast inaccuracies and diagnostics that do not conform to either the ICD or the DSM. Studies from Germany show that between 30 and 40 % of people with dementia living in nursing homes are not accurately diagnosed; for these residents, either an etiological differentiation is missing or there is an inappropriate diagnosis [22‐24]. Inaccurate diagnosis has also been found in studies from the United States, Norway, Israel and Ireland [25‐29].

Nursing home research is challenged by inaccurate recorded diagnoses. Ignoring inconsistencies in a recorded dementia diagnosis may provoke misclassifications, bias selection and confound study results [30]. However, the evaluation of a diagnosis within a study is resource-intensive and may be ethically questionable because the diagnosis process is reported as burdensome and stigmatizing [31, 32].

Researchers must develop a sampling strategy that eliminates inconsistencies while remaining both reliable and practicable. We assume that in the past, researchers defined dementia differently and applied different methods to identify residents with dementia in nursing homes. Therefore, we investigated the methods used to define dementia and to identify people with dementia in health services research. Although we assume that the delineated problem is obvious in various countries, we limited the scope of our study to research undertaken in a single country (Germany). In Germany, physicians are not constantly present in nursing homes, and nurses are not allowed to assist in diagnostic procedures; resources for dementia diagnostics in primary care are scarce [33]; hence, diagnostics are often superficial, performed rapidly and lacking recommended measures [34, 35]. Due to the free choice of medical practitioners that is guaranteed by law in Germany, residents within one nursing home may be cared for by different practitioners, making it difficult for researchers to reconstruct the diagnostic procedures that were applied by various medical doctors. These conditions require a carefully considered sampling strategy. The narrowed focus on one country ensures the comparability of the studies because the nursing-home conditions are identical. This allows a synthesis and comparison of the various methods used in the included studies.

Because cognitive impairment (CI) is the leading symptom in dementia and the conditions “dementia” and “CI” are sometimes used interchangeably in scientific publications, the scope of our review encompasses studies that do not exclusively focus on residents with dementia but instead include residents with CI. Provided we found differences, we discuss the implications of different methods on the generalizability and comparability of the study findings.

The following research questions were addressed:

Based on the results, we discuss the implications of the different methods and suggest principles to guide future studies.

Compared to systematic reviews that aim to answer questions about the effectiveness of clinical procedures, there are few well-defined tools and processes for methodological reviews; not all items of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines are applicable (see PRISMA checklist in the Additional file 1: Table S1) [36]. Unlike a traditional Cochrane-style review, Lilford and colleagues [36] recommend methodological reviewers not to obtain a thorough collection of studies, but to ensure a wide search that provides an overview of the topic. They also recommend various safeguards to reduce potential bias, such as multi-disciplinary teams and peer reviewing of the final report. The methods applied in this review consider these recommendations.

The literature search yielded 650 articles. Sixty-four articles were identified to answer our research questions. The results of the literature search can be seen in the flow chart in Fig. 1.

The characteristics of the included studies are presented in Table 1.

This review reveals a sampling challenge for dementia studies in German nursing homes in health services research: uncertainty related to existing dementia diagnoses, constrained resources and ethical dilemmas concerning accurate diagnostic procedures and the achievement of study validity. When designing a sampling plan, the advantages of convenient and efficient access and ethical considerations related to the residents must be weighed against participant-related validity concerns.

In our review, we found 4 sampling methods to identify residents with dementia:

Each method has to be discussed with regard to its practicability and validity.

A guideline- and criteria-based diagnosis of dementia is the most valid criterion to select study participants when the aim of the study is to target people with dementia. The clear benefits of this method are a thorough assessment of dementia-related conditions and a sound differential diagnosis to ensure that the CI is not caused by conditions other than dementia. Challenges related to this approach are the resources required in terms of time, costs and availability of physicians.

The performance of diagnostics by a member of the research team enhances study validity by ensuring conformity to the criteria. As demonstrated in several studies [22, 23, 48‐51, 60‐68], criteria conformity can be achieved in the evaluation of a dementia syndrome. In these studies, both subjective and objective information on the patient’s cognitive decline were thoroughly assessed. However, the validity of the dementia subtype diagnosis, as performed in a few of these studies [22, 23, 60‐65], was questionable. In particular, during the late stage of dementia, a differentiation between dementia subtypes is difficult and requires the use of disease progression measures such as repetitive cognitive tests. It can be assumed that this information was not readily available for a large number of residents because the implementation of diagnostic measures, other than the collection of subjective information, is infrequent in the German primary care sector. A survey of 23 German primary care physicians regarding the use of diagnostic measures showed that only 3 physicians used neuropsychological tests as a component of their diagnosis [34]. There is also evidence showing that in primary care, imaging techniques and other diagnostic features are seldom used; misclassifications that lead to an overestimation of vascular dementia were frequently discovered in several studies [34, 94, 95].

With respect to the reporting of the diagnostic performance, the studies lacked detailed information. Several studies did not report which instrument was used to assess which criterion, whether normative data were used to determine CI, which cutoffs were used and how missing data were addressed. It is recommended that this information be provided when a dementia diagnosis in the oldest-old population is evaluated [20]. Additionally, information on training for the raters who conducted the cognitive assessments was missing in all of the studies. In our opinion, this is a crucial issue because 1 study showed that even physicians who underwent training in cognitive assessments may still fail to use these diagnostic instruments correctly [34].

However, evaluating a study diagnosis also lacks feasibility when resources are constrained. Imaging diagnostics are critical because nursing home residents are often too frail to transport, and nursing homes are often not affiliated with a clinic. Consequently, appropriate technical devices may not be available. As we can see in the studies that evaluated dementia subtypes, only existing findings were used to conduct a subtype diagnosis. Thus, the diagnosis must be made based on neuropsychological and functional tests, clinical examinations and interviews that are conducted by either geriatric psychiatrists or clinical psychologists; this process requires time, experienced assessors and willing participants. The study with the largest sample size involved 589 residents from twenty nursing homes in a German city [69]. Thus, performing dementia diagnostics within a study seems feasible for a sample size up to 600 people who live in a single city. However, we assume that the evaluation of a dementia diagnosis may have a negative impact on the willingness to participate in a study if the assessment is overly burdening or involves invasive procedures [96]. For example, in 1 study, 20 of 113 eligible residents were unwilling to submit to an MMSE [93]. For the majority of the nursing home population, legal guardians decide whether the resident will participate. In our experience from studies in the field, legal guardians are hesitant to permit participation because studies involve measures that will burden the resident.

The use of a recorded medical diagnosis as an inclusion criteria may be appealing, especially in studies with large samples and with a geographically widely dispersed population. However, the well-documented inaccuracy of recorded dementia diagnoses indicates that in a research situation, this approach must be questioned [6, 30]. If researchers cannot be sure which criteria have been used and how they were measured, we must assume that the diagnosis may not correspond to the established ICD- or DSM-criteria. Consequently, misclassification threatens the internal validity of study results. In experimental trials, diagnostic discrepancies may lead to heterogeneity in the intervention and control group that may have an impact on treatment effects. The inclusion of false-positive participants may explain the failure to show treatment effects, and the exclusion of false-negative participants both prolongs the recruitment process and may cause a selection bias.

The third sampling method that combines a recorded diagnosis with a cognitive screening is more feasible than evaluating a diagnosis following defined criteria and recommended procedures. Cognitive tests are comparatively easy and quick to administer and therefore, their administration can be delegated to nursing-home staff members who are more easily accessible than geriatricians or psychologists, such as nurses or medical students. This method may ensure a more homogenous sample with respect to cognition, but it misses other information about functioning and decline in functionality and cognition. Without this information, it is impossible to verify the diagnosis according to established diagnostic criteria [30, 97]. The risk of including false-positives can be reduced, but not eliminated. In this respect, exclusion criteria are relevant. Several studies reported that residents were excluded when diseases other than dementia explained their cognitive decline. However, the combination of a recorded diagnosis, a cognitive screening and exclusion criteria still does not reduce potential selection bias resulting from systematically missing false negatives. Furthermore, it is necessary to decide what to do when combined inclusion criteria contradict each other. In 1 study, these concerns were addressed. Majic and colleagues report that participants were included with a recorded diagnosis of dementia, an MMSE ≤ 24 and a duration of CI of > 6 months, but diagnostics were performed in case of incongruity [48]. They report elsewhere that additional diagnostics were performed in residents if the nursing home staff suspected dementia, but there was no recorded diagnosis [49]. Other studies did not sufficiently report how they addressed incongruity in the combined inclusion criteria and whether they performed any measures to rule out the outlined problems.

The fourth method that was identified in this review is the use of a single test result. The instruments FAST, DSS and the MMSE were used for this purpose.

The FAST staging procedure is used as a diagnostic measure for dementia and relies on the assessment of functional impairments that are attributable to dementia [98]. This instrument was tested in its original language on a sample that was composed of 16 participants who had different levels of cognitive impairment and numerous possible underlying diseases. The testing was conducted by physicians who were in their postdoctoral training phase; their results revealed excellent values for inter- and intra-rater reliability and concurrent validity [99]. Unfortunately, data on internal consistency and discriminant validity are absent [100] as well as the results from the German version of the FAST. To our knowledge, this instrument has not been evaluated for its sensitivity or specificity when being used to detect dementia. In the reviewed studies, it was administered by health personnel, not physicians. Additionally, there are no studies that compare the accuracy of this instrument depending on if the instrument is administered by physicians versus other health personnel.

The DSS is a 7-item scale that assesses both memory and orientation at the time of assessment [69]. The instrument was originally developed in the German language, tested in 589 nursing home residents, and administered by nurses [69]. It was validated against the MMSE, BAS-DEM and the CDR. Utilizing the CDR as the gold standard, the DSS was able to correctly classify more than 80 % of residents; however, it showed small inaccuracies when compared to the MMSE and BAS-DEM. Because the scale is fast and easy to administer, this approach can be considered to be feasible in health services research, even though its validity must be critically discussed. The scientific literature concordantly states that the assessment of single domains of cognition, especially a single cognitive test, cannot accurately diagnose dementia and should not be used to substitute systematic evaluations, examinations and laboratory tests [101]. Because the DSS does not assess changes in cognitive functions, it is not possible to determine if the present cognitive deficits had declined over the past 6 months. Hence, the DSS will misclassify residents who have acute cognitive deficits that are due to infection or dehydration. The same misclassification can occur in residents who have chronic cognitive impairments due to other diseases.

In our review, the MMSE was the instrument that was most often chosen to define the inclusion criteria. The rationale for using the MMSE in the studies was primarily its degree of popularity and widespread utilization. A meta-analysis on the accuracy of the MMSE confirms this choice: in high prevalence settings, the MMSE shows a sensitivity and specificity of 77 and 90 %, respectively [15]. The notable shortcomings of the MMSE in the context of nursing home research were rarely discussed in the studies, although those shortcomings threatened the internal validity of the study results. The MMSE shows a floor effect in very severe score ranges, people with little formal education, and people with severe language problems. [101] Several items are strongly influenced by age and education, which implies a need to adjust MMSE scores when establishing thresholds. We found a variety of MMSE thresholds for staging dementia in the reviewed studies - a result that was also reported in other reviews [12, 20]. This result may be a consequence of missing normative data for the age group of the oldest-old and standardized age-adjusted cutoff values. The authors of a meta-analysis on dementia screening and case-finding tool validation studies suggest alternatives to the MMSE such as the Mini-Cog [102]. However, the suitability of these screening tools for nursing home research needs to be investigated.

In this respect, an associated question must be discussed in health services research in nursing homes: is a valid diagnosis of dementia necessary to define the study population? If the aim of the study is to prove the benefit of an intervention that should be provided exclusively to residents with dementia, a valid diagnosis is essential to prove the benefit for this population. If the aim of the study is to prove a benefit for residents with cognitive and functional impairments independent of their etiology, a dementia diagnosis is dispensable. The same question must be considered by clinicians because in the absence of disease-modifying treatments, the primary advantage of a diagnosis for the oldest-old population has to be determined to justify resource-intensive diagnostics [20]. The need to perform diagnostics in the nursing home population must also be ethically justified because psychological tests can be significantly burdensome to people with CIs.

We also found 4 articles that aimed to investigate residents who had CI but did not explicitly have dementia. To us, it was unclear why some studies focused on the population of people with CI but not on people with dementia because the authors did not elaborate on the question whether the etiology of CI played a role with respect to their research focus. In 2 publications, the terms “dementia” and “CI” were used interchangeably [39, 40]. If the authors did not explain why they defined their population based on a disease or a symptom and later failed to distinguish between the 2 conditions, one may ask whether the question was sufficiently addressed.

In addition to the variability of case ascertainment strategies in dementia studies, this review mirrors the developments in knowledge, concepts, differentiations and diagnostic criteria related to “dementia” that have occurred during the last 30 years. With each decade of dementia studies in German nursing home research, the diagnostic procedures were refined and reflect methodological innovations. Therefore, the definition and assessment of dementia and CI must be considered against the respective state of knowledge at the time a particular study was conducted. For example, in 1 study from 1993 [72], the Feighner criteria were applied; these criteria had been introduced in the DSM-III and became available in the German language with the German translation of the DSM-III in 1984. On the contrary, another study [75] was published the same year; it is obvious that those researchers could not apply today’s diagnostic criteria because they were not available at the time the study was designed and conducted.

An overview of the advantages and disadvantages of the different case ascertainment strategies is summarized in Table 9.

Considering the findings of our review, we suggest the following principles to improve the validity and comparability of study results on dementia in nursing homes.

Investigators must clearly distinguish whether their research addresses residents with CIs or residents with dementia with an appropriate etiology. If residents with dementia are addressed, it should be in accordance with the study aim. Studies that address the population of nursing home residents with dementia should report in detail which method of case ascertainment was used and should discuss the limitations associated with each method. In our opinion, a criterion-based diagnosis is essential when a study addresses explicitly residents with dementia. If residents are included in studies based on diagnoses that do not conform to established criteria, the results from different studies are not comparable. To ensure that a diagnosis conforms to criteria, clinicians who are members of the research team should evaluate a diagnosis during the research project. When financial or human capacities are constrained, one should consider alternative methods to a face-to-face consultation by a geriatric physician, psychologist or psychiatrist. Several methods utilizing an expert panel are described for the diagnosis of dementia [104]. The method by Magaziner et al. [30] appears promising for research purposes: 2 clinicians decide whether dementia is present based on the resident’s history and neuropsychological tests performed by lay evaluators. Compared to direct clinical evaluation, agreement was satisfactory (76 %). Another promising approach is a nurse-administered diagnosis. Studies showed a high agreement between a nurse-administered diagnosis and a multidisciplinary team diagnosis in determining mild CI in primary care [105]; in addition, there is agreement between a Memory Clinic diagnosis [106] and a moderate agreement with the ICD-10 diagnosis (ibd.). To our knowledge, a nurse-administered diagnostic procedure for dementia in nursing home residents has not yet been developed and tested.

Regardless of which method is used to evaluate a diagnosis of dementia within a study, reporting requires details regarding the domains that are tested, the rationale behind instrument use for each domain examined, the cutoff values set for each domain and the management of missing data. Transparency in reporting missing data is needed because approaches for adequately managing these gaps can differ and produce varying effect estimates [103].

If a dementia diagnosis is considered as dispensable or a criteria-based diagnostic procedure cannot be applied due to constrained resources, the study should not target the dementia population, but instead clearly state which population they are addressing.

Not applicable.

Not applicable.