Satisfaction with quality of ICU care for patients and families: the euroQ2 project

Participants came from 21 ICUs (11 from Denmark and 10 from The Netherlands) including both university affiliated and regional ICUs from different parts of the two countries.

Family members of patients admitted to the ICU for 48 hours or more, independent of ICU outcome, were eligible for participation. Up to three family members per patient could participate. Family members were defined as the persons closest to the patient (as identified by the patient), including partners, siblings, children, parents and friends. If more than three family members wanted to participate, the family members themselves chose the participants based on who had spent the most time in the ICU.

Family members were excluded who met the following criteria: (1) under age 18 years; (2) with cognitive impairment; or (3) unable to read or write Danish or Dutch.

Family members who fulfilled the eligibility criteria were approached during the patient’s ICU stay by either ICU nurses or physicians; most family members were approached although sometimes ICU nurses and physicians forgot to do so. Family members received oral and written information about the study and, if they agreed to participate, they provided their name and home address. Three weeks after patient discharge from the ICU, family members received the questionnaire by mail, together with written information and a prepaid envelope. In Denmark, the individual ICUs were responsible for sending out the questionnaires, and the cover letter was signed by the local investigators. In the Netherlands, all questionnaires were sent out by the investigators. In both countries, the completed questionnaires were returned to the investigators. If the questionnaire was not returned, one reminder with a new questionnaire was sent.

For participating families, the following patient data were obtained from the medical record: gender, age, medical or surgical specialty of the admitting physician, diagnosis, length of stay in the ICU, and decisions about withholding or withdrawing life-sustaining treatments. The Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score (SAPS) were also included when available (from 12 and 13 ICUs, respectively). Data on family respondents included age, gender and relationship to the patient.

The euroQ2 questionnaire (see Additional file 1), consists of two sections: the euroFS-ICU, which all participating family members completed, and an extra section containing the euroQODD, which was completed only by family members of patients who died in the ICU. In this paper, we present results for the euroFS-ICU portion of the questionnaire.

Statistical analyses were conducted using Stata 13 [20] and Mplus 7.4 [21]. For comparing background characteristics of Danish and Dutch family members and patients we used the chi-squared (χ²) or Fisher’s exact test and the Mann-Whitney U test as appropriate. To compare family members’ responses between countries, we used clustered regression models with country as predictor and the five-point satisfaction items as outcomes. We tested associations of family and patient characteristics with family members’ responses on the family satisfaction items with clustered single-predictor probit regression models (family respondents nested under patients; outcomes defined as ordered categorical variables) estimated with weighted least squares with mean and variance adjustment (WLSMV). P values were based on Wald’s test. Clustered analyses were used to adjust for participation of more than one family member for some of the patients.

Earlier analysis of the North American version of the FS-ICU had suggested that the questionnaire encompassed two domains (care and decision-making), resulting in a recommendation for computing composite scores for those two domains and for total satisfaction [14]. However, that analysis was based on exploratory factor analysis (EFA), with indicators defined as normally distributed continuous variables, and without the use of strict tests of empirical fit. More recent analyses, based on exploratory factor analysis within a confirmatory factor analysis framework (E/CFA) [22] and using a larger sample, with indicators defined as ordered categorical variables, have suggested that the instrument likely encompasses four domains of family satisfaction: (1) communication with the family; (2) empathy shown to the family; (3) support of the family during decision making and (4) management of patients’ symptoms (work by LD, JRC and RAE) (see Additional file 2). Although the euroFS-ICU is an adapted version of the 24-item FS-ICU, many of the items in the two instruments are identical. Therefore, we hypothesized that the euroFS-ICU would encompass dimensions that are conceptually similar to the four domains identified previously in the North American questionnaire.

Examination of the measurement characteristics of the euroFS-ICU included four aspects: (1) positing a conceptual framework for the domain structure of the euroFS-ICU; (2) using exploratory factor analysis (EFA) to simplify the conceptual structure by removing items that contributed to statistically significant misfit (i.e., the χ² test of fit with p < 0.05) to data from the combined samples; (3) investigating whether the simplified structure was equally appropriate for Denmark and the Netherlands, considered separately and (4) assessing whether a set of “pure” factors (i.e., each indicator contributing to the measurement of only one factor) could be identified, with the resulting factors having equivalent meaning in the two countries. Evidence supporting equivalent meaning between countries required that a model in which the loadings and thresholds for each indicator were constrained to equality between countries produced non-significant misfit to the observed data (i.e., the χ² test of fit with p > 0.05). Equivalent meaning must be established in order to provide legitimacy for between-country comparisons of mean levels on the factors. Detailed descriptions of the analyses are presented in Additional file 2.

A total of 1077 family members participated, 573 from Denmark and 504 from The Netherlands, representing 920 ICU patients. In Denmark, 185 of the 573 participants were second and third family members of the same patient. In The Netherlands, 6 of the 504 participants were second and third family members. The overall response rate was 72% among family members who were approached and reportedly willing to participate, 75% in Denmark and 68% in The Netherlands. The Dutch and Danish participants differed significantly on a number of demographic and clinical characteristics such as age, relationship to patient, reason for admission and level of therapy (Table 1).

Except for inclusion in decision-making processes, the Danish ratings were significantly higher than the Dutch ratings (Table 2). Items with the greatest number of “Excellent” endorsements were concern and caring towards the patient, dyspnea management, atmosphere of the ICU, presence at the bedside and ease of getting information. Items with fewer “Excellent” endorsements and suggesting the need for improvement were management of agitation, emotional support, consistency of information and inclusion in decision-making (Table 2).

In addition to the questions presented in Table 2, the euroFS-ICU contains three items that do not use 5-point Likert scale response options: (1) those who chose “Fair” or “Poor” when asked about inclusion in the decision-making processes were subsequently asked why they gave these responses: 114 family members responded to this question (Denmark, n = 65, The Netherlands, n = 49), with 9% stating that they were involved too much, 63% that they were not involved enough, and 28% that their low satisfaction was due to other reasons; (2) the participants were also asked whether they felt they had adequate time to have their concerns addressed and questions answered when major decisions were made, with 72% answering that they had enough time and 9% that they could have used more time: for these two questions there were no statistical differences between the two countries; (3) finally, the participants were asked to assess overall satisfaction with the care the patient had received from all doctors, nurses and other healthcare professionals: the assessment was made on a scale from 0 to 10, with 0 being worst care possible and 10 best care possible; the median assessment was 9 (inter-quartile range 8–10) with significantly higher scores in Denmark (median 9 (9–10)) than in The Netherlands (median 9 (8–9)) (p < 0.001).

Whereas there was a significant difference between the two countries for almost all ratings, the respondents’ age, gender and relation to the patient had only a small impact on level of satisfaction. Respondent age influenced six of the items, with higher ratings as age increased. These items were agitation management, atmosphere of the ICU, emotional support, opportunity to be present at the bedside, consistency of information and overall satisfaction with care. The respondents’ gender was significantly associated with four items, with female respondents providing higher ratings, on average, than their male counterparts. Two of the items were about symptom management (management of pain and dyspnea) and two concerned staff communication (willingness to answer questions and provision of understandable explanations). The respondent’s relationship to the patient was not associated with any of the satisfaction ratings (see Additional file 2: Table S1a and S1b for details.)

The SAPS was significantly associated with satisfaction, with higher scores associated with higher family satisfaction. The SAPS score was associated with 15 items. The items not associated with the SAPS were symptom management (pain, breathlessness and agitation) and adequate time to have concerns addressed. Death in the ICU was associated with higher ratings on seven items including consideration of family needs, emotional support and overall satisfaction with care. The remaining patient characteristics (i.e., gender, age and hours in the ICU) were associated with few or none of the satisfaction items (see Additional file 2: Table S2a-f for details).

The first step in investigating the structure of the euroFS-ICU items was to assign each of the 20 items a priori to one of the four conceptual domains (communication, empathy, patient care and symptom management, and decision-making) that have been identified in the North American version of the instrument (Additional file 2: Table S3). To achieve acceptable fit to data from the combined Danish and Dutch samples (please see Additional file 2, p. 1 for details), we generated a series of EFA models, using modification indices that eliminated nine items (five from the communication domain, one from empathy, two from patient care and symptom management, and one from decision-making) from the a priori structure. This produced a four-domain model with strong primary loadings, relatively weak cross-loadings, and good fit to the observed data from the combined countries (Table 3; see Additional file 2, p. 11 for details).

However, although analysis of this EFA model within countries showed acceptable fit to the within-country data, the countries were dissimilar in their pattern of loadings, portending difficulties in establishing a factor structure where the factors had equivalent meanings in the two countries (see Additional file 2, p. 11 for details). Moreover, a confirmatory factor analysis (CFA) in which each indicator was allowed to load on only one of the four factors required further elimination of indicators in order to obtain adequate fit to data from the separate countries, and even this model failed when indicator loadings and thresholds were constrained to equality between countries (see Additional file 2, pp. 12–14). As a result of this failure, we could not conclude that the euroFS-ICU contains elements supporting a four-factor structure for which the factors can be legitimately compared between countries.

All of the models tested with these data use a methodology that is widely reported for similar instruments. However, it is based on an important type of model misspecification: the modeling of factor indicators as reflective (or effect) indicators, when they are more appropriately modeled as causal indicators [23‐25]. Reflective indicators are indicators that are caused by (i.e., reflect) a construct, with an individual’s position on all of the indicators tending to rise or fall in concert with that individual’s position on the underlying construct. By contrast, causal indicators are variables that contribute to, rather than reflect, the construct; an individual’s position on some, but not necessarily all, of the causal indicators is expected to rise and fall in concert with the individual’s position on the construct. The difference is in the direction of causation: reflective indicators are caused by the construct; causal indicators contribute to the construct. To achieve statistical identification, modeling a construct with causal indicators requires that there be at least two additional variables that can be used as outcomes of the construct. Ideally, these would be reflective indicators, but they may alternatively be more distal outcomes of the construct. Although the euroFS-ICU includes only one hypothesized domain (the “Communication” domain), for which there are, arguably, reflective indicators, the existence of reflective indicators for this one domain allowed us to test an alternative measurement method.

Figure 1 shows a model in which the quality of ICU communication is measured with a combination of causal and reflective indicators with the additional imposition of between-country measurement invariance. In this model the regression coefficients for the causal indicators and both the factor loadings and thresholds for the reflective indicators were constrained to equality between the two countries. This model provided good fit to the data (p for the χ² test of fit = 0.4147), thereby providing evidence that the combination of causal and reflective indicators measure a latent communication construct that has equivalent meaning in the two countries and on which the two countries can be legitimately compared.

The remaining hypothesized domains were not represented by a sufficient set of variables for use as outcomes (either as reflective indicators or as more distal outcomes) to allow causal-indicator tests of those domains.

Strengths of the study include enrollment of more than 1000 family members from two countries, affiliated with patients who were treated in a large number of ICUs of different types and located in several geographic areas. The response rate among family members approached by ICU staff and willing to consider participation was relatively high (72%) and respondents left few questions unanswered. Despite this high response rate, it was lower than that experienced in an earlier phase of the study (87%), perhaps because the earlier phase included phone contacts to respondents, whereas the current phase used mailed reminders. In addition, the analytic approach in this study was more rigorous than that used for most other measures of family experience. The analyses show the importance of using newer statistical approaches to ensure that multi-item constructs are unidimensional and meet quality standards, as we suspect that other measures may encounter similar challenges of model misspecification in the measurement of latent constructs.

There are also important limitations. SAPS scores were only available for approximately 70% of the sample and from 62% of the ICUs, and the generalizability of these findings may therefore be limited. Additionally, SAPS scores may not discriminate and describe disease severity as well as the APACHE-III scoring and APACHE-IV prediction model, but these scores were not available. If an ultimate objective is to construct multi-item constructs of overall satisfaction and its sub-domains, an important limitation is the absence of reflective indicators of those constructs in the current instrument. Modification of the instrument is already in progress and may allow an exploration of whether such constructs exist and are consistent between countries, or whether contributors to satisfaction vary by country. The validity, reliability, and responsiveness of such measures remain to be determined. Because the current instrument consists primarily of casual indicators, most future analyses with this data set, except for satisfaction with communication, are best limited to the use of single-item measures. A second limitation is the omission of some eligible family members during the study period, owing to ICU staff forgetting to mention the study to them. However, there is nothing to indicate that these omissions were other than random. Likewise, exact numbers of families who refused to participate when approached are missing, but are estimated at less than 10%. A third limitation is that the effect of ethnicity is not examined. As the vast majority of patients in both Denmark and The Netherlands are Caucasians, groups of non-Caucasian family members would be too small for analysis. The lack of ethnic subsamples reduces the generalizability of the study. A fourth limitation is the fact that both of the countries represented in the study are from Northern Europe. Although we identified a model of satisfaction with communication that was invariant for these two countries, it may not fit data provided by ICU families from other parts of the Europe or the world. Addition of data from other European countries and other regions of the world will be important for future studies.