Introduction
Many survivors of acute respiratory failure experience long-term cognitive impairments across a variety of cognitive domains, including attention, memory, mental processing speed and executive function [
1-
6]. To evaluate such impairments, comprehensive neuropsychological test batteries are commonly used [
7-
11]. However, such batteries require trained personnel, expensive licensing fees and several hours per patient for test administration and scoring. Hence, valid cognitive screening tests that are inexpensive, brief and easy to administer would be invaluable in identifying which survivors may develop long-term cognitive impairment [
2]. Furthermore, cognitive screening tests may aid in understanding risk factors and trajectories of cognitive impairments [
5,
12], and may facilitate the development and evaluation of targeted interventions to address cognitive impairments [
13,
14].
The Mini-Mental State Examination (MMSE) is a commonly used screening test for cognitive impairment in both clinical practice [
15,
16], and research [
17-
24]. Among older adults, the MMSE is the most widely used cognitive screening test, with a pooled sensitivity of 88% and specificity of 86% [
15]. Moreover, the MMSE has good performance in identifying mild cognitive impairment in older adults [
14], identifying subtypes of mild cognitive impairment [
25] and predicting cognitive impairment in patients with post-operative delirium [
26].
As a result of these favorable performance characteristics, the MMSE has been commonly used to assess cognitive outcomes in critical care populations [
17-
22,
24]. However, the performance of the MMSE has not been specifically evaluated in acute respiratory failure survivors. Understanding whether the MMSE can accurately screen for cognitive impairment in this patient population would provide important new insights. Hence, the objective of this study is to assess whether the MMSE can detect cognitive impairment, as assessed by a concurrently administered, detailed neuropsychological test battery, in survivors of acute respiratory failure. Additionally, this study explored whether the timing of patient follow-up assessment after hospital discharge or patient characteristics influenced the relationship between the MMSE and the neuropsychological test battery.
Discussion
In cross-sectional analyses of two prospective longitudinal studies of acute respiratory failure survivors, we found that the MMSE had fair to moderate agreement, both for overall cognitive impairment and for specific cognitive domains, compared to more detailed neuropsychological tests. These findings were not influenced by patient characteristics or timing of longitudinal follow-up assessment. Indeed, even when accounting for gender, age, education level, concurrent depression and anxiety status, and pre-existing psychiatric comorbidity prior to ICU admission, MMSE domain scores correlations were weak to moderate with neuropsychological test scores.
The ability of the MMSE to discriminate overall cognitive impairment was only fair based on the receiver operating characteristics curve analysis; expressly, the MMSE had low sensitivity. Even using a conservative cutoff score of <24, the MMSE failed to identify a substantial portion of survivors who had cognitive impairment identified on neuropsychological tests, with poor to moderate negative predictive values. Generally, a screening tool should have an area under the ROC curve of at least >0.8 to demonstrate excellent diagnostic accuracy [
49], and the MMSE failed to meet this threshold at both 6 and 12 months. These results are similar to previous work in cardiac surgery [
50] and bariatric surgery [
51] patients. Understanding the limited extent of the MMSE’s agreement with neuropsychological test scores administered concurrently has important implications. Specifically, screening negative on the MMSE for cognitive impairment does not rule out the presence of cognitive impairment for acute respiratory failure survivors at 3, 6 and 12 months.
There is great interest in screening for cognitive impairment in ICU survivors given the frequent and long-lasting cognitive impairments observed [
52]. Our study provides additional evidence that the MMSE may not be a valid cognitive screening tool for use in this patient population. Evaluation of other cognitive screening instruments (such as the Montreal Cognitive Assessment) in this population is needed to determine if they will better identify cognitive impairments [
53]. Recent work found the Montreal Cognitive Assessment is more sensitive in accurately differentiating mild cognitive impairment from normal cognitive function [
54]. Another approach could include augmenting the MMSE with more targeted instruments. For example, adding a measure of executive function to the MMSE can improve the detection of cognitive deficits [
55].
The MMSE is characterized by highly selective coverage of cognitive domains and does not evaluate learning, delayed memory, processing speed and executive function, among other cognitive domains. Many of these omitted cognitive domains reflect the frontal-subcortical white matter-mediated impairments known to occur in critical illness, and commonly identified in ICU survivors [
4,
14,
27,
31,
56]. Hence, their omission in a short screening test is problematic. As we learn more about the granular and specific aspects of cognitive impairment in acute respiratory failure survivors, better cognitive screening tools tailored to this population could be developed.
Our study is among the first to test the validity of the MMSE in acute respiratory failure survivors. Strengths of our study include the ability to examine overall and domain-specific performance of the MMSE compared to a battery of neuropsychological tests administered concurrently. The ALTOS study provided a relatively large multicenter sample, which employed both a screening and neuropsychological test battery, and data from two time points. Inclusion of the ABC study allowed us to test the generalizability of our findings in an independent cohort using similar neuropsychological tests. Our study also has potential limitations. First, we were not able to fully replicate all of the analyses from the ALTOS data using the ABC study data due to the different follow-up time points and neuropsychological tests used. However, the available comparisons (overall cognitive impairment and attention) demonstrated similar results to the ALTOS study, helping increase confidence in the generalizability of our findings, especially since the studies used different neuropsychological tests. Further, participants in the ABC study had a high baseline psychiatric comorbidity, which is higher than the prevalence observed in other ICU cohorts [
57,
58]. However, while the prevalence of baseline psychiatric comorbidities was different between the two studies (42% for the ALTOS study and 74% for the ABC study), the results of the analyses of the MMSE were similar. Second, despite the relatively large sample size, secondary analyses conducted using binary patient categories had smaller sample sizes and may have reduced power to detect true difference between subgroups. Third, the MMSE and neuropsychological tests were administered using different methods in the ALTOS study (phone versus in-person), potentially contributing to the differences. However, the MMSE is validated for telephone administration, minimizing this concern [
34]. Moreover, in-person administration was used for all testing in the ABC study and demonstrated similar results to the ALTOS study. Finally, the MMSE total score is validated to identify cognitive impairment; however, the domain scores have not been validated as standalone measures. Our assessment of the correlations between the MMSE domains and the corresponding neuropsychological tests should be taken within context of overall study findings, and not as a recommendation to only administer specific domains of the MMSE.
Acknowledgements
ERP takes responsibility for the content of the manuscript including the data and analysis.
This research was supported by the NIH (grant numbers: R24HL111895, R01HL091760, R01HL091760-02S1, R01HL096504 and K23AG034257), the Johns Hopkins Institute for Clinical and Translational Research (ICTR) (grant number: UL1 TR 000424–06) and the ALTA and EDEN/OMEGA trials (contracts for sites participating in this study: HSN268200536170C, HHSN268200536171C, HHSN268200536173C, HHSN268200536174C, HSN268200536175C and HHSN268200536179C). The sponsors had no role in the data acquisition, analysis or preparation of the manuscript.
We thank all patients and their proxies who participated in the study. We acknowledge our dedicated research staff, including the following who assisted with data collection, training and quality assurance and/or data management: Lindsay Anderson, Ellen Caldwell, Nancy Ciesla, William Flickinger, Jacqueline Flynn, Jonathan Gellar, Stephanie Gundel, John Keenan, Christopher Mayhew, Melissa McCullough, Jessica McCurley, Mardee Merrill, Laura Methvin, Kristin Sepulveda, Kelly Swanson, Elizabeth Vayda and Cassie Wicken.
Investigators and research staff from the National Heart, Lung, and Blood Institute Acute Respiratory Distress Syndrome (ARDS) Clinical Trials Network sites that participated in this follow-up study: University of Washington, Harborview (*L Hudson, S Gundel, *C Hough, M Neff, K Sims, A Ungar, T Watkins); Johns Hopkins University (*R Brower, H Fessler, D Hager, P Mendez-Tellez, D Needham, K Oakjones); Johns Hopkins Bayview Medical Center (J Sevransky, A Workneh); University of Maryland (C Shanholtz, D Herr, H Howes, G Netzer, P Rock, A Sampaio, J Titus); Union Memorial Hospital (P Sloane, T Beck, D Highfield, S King); Washington Hospital Center (B Lee, N Bolouri); Vanderbilt University (*AP Wheeler, GR Bernard, M Hays, S Mogan, TW Rice); Wake Forest University (*RD Hite, A Harvey, PE Morris, M Ragusky); Intermountain Medical Center (*A Morris, *C Grissom, A Austin, S Barney, S Brown, J Ferguson, H Gallo, T Graydon, E Hirshberg, A Jephson, N Kumar, M Lanspa, R Miller, D Murphy, J Orme, A Stowe, L Struck, F Thomas, D Ward,); LDS Hospital (P Bailey, W Beninati, L Bezdjian, T Clemmer, S Rimkus, R Tanaka, L Weaver); McKay Dee Hospital (C Lawton, D Hanselman); Utah Valley Regional Medical Center (K Sundar, W Alward, C Bishop, D Eckley, D Harris, T Hill, B Jensen, K Ludwig, D Nielsen, M Pearce). Clinical Coordinating Center: Massachusetts General Hospital and Harvard Medical School (*D Schoenfeld, N Dong, M Guha, E Hammond, P Lazar, R Morse, C Oldmixon, N Ringwood, E Smoot, BT Thompson, R Wilson). National Heart, Lung and Blood Institute: A Harabin, S Bredow, M Waclawiw, G Weinmann. Data and Safety Monitoring Board: RG Spragg (chair), A Slutsky, M Levy, B Markovitz, E Petkova, C Weijer. Protocol Review Committee: J Sznajder (chair), M Begg, L Gilbert-McClain, E Israel, J Lewis, S McClave, P Parsons.
*Principal investigator.
Competing interests
TD Girard has received payment for lectures from Hospira. C Hough received consulting fees from High Point Pharmaceuticals, High Point, NC. EW Ely received consulting fees from Hospira, Abbott Laboratories, and Orion. The authors declare that they have no other competing interests.
Authors’ contributions
All authors contributed to the conception and/or design of this study. DMN, VDD, TDG, PEM, CLH, JCJ, PAM-T, EW and ROH contributed to the acquisition of data. ERP, MH and KSC contributed to the analysis of data and all authors contributed to the interpretation of data. ERP and ROH drafted the manuscript, and all authors critically revised it for important intellectual content and approved the final version to be submitted.