Development and psychometric properties of a questionnaire to assess barriers to feeding critically ill patients

Cabana et al. reviewed 76 studies that assessed the potential barriers to physician adherence to CPGs and assimilated the results into a framework [26]. We selected this framework to guide our research because it aligned with our specific objective of identifying barriers to guideline adherence. We adopted a case study approach [27] to revise and extend this framework and make it more applied to barriers to adherence to nutrition guidelines in the ICU and to include both individual provider level and system level barriers [28]. Several other authors have also revised and expanded this framework to make it more applicable for specific guidelines or innovations [15, 29‐31].

The multiple case study was conducted in four ICUs in Canada between February and April 2006 [32], it included semi-structured interviews with 28 critical care providers (i.e., physicians, nurses, and dietitians) to ascertain attitudes and perceptions about nutrition guidelines. The qualitative analysis of the interview transcripts, related ICU documents, and field notes was guided by Cabana et al.’s framework [26]. Textual coding was conducted independently by two researchers, and relationships between these codes were identified to create key thematic domains, which resulted in a framework for barriers to adherence to critical care nutrition guidelines. The schema and explanatory tables that describe this framework have been published elsewhere [28], but briefly the five thematic domains and associated sub-categories included in the framework were: guideline characteristics; implementation process; institutional factors (i.e., hospital and ICU structure, hospital processes, resources, ICU culture); provider intended behavior (i.e., provider characteristics (i.e., professional role, critical care expertise, educational background, personality), knowledge (i.e., familiarity, awareness), attitudes (i.e., agreement, outcome expectancy, motivation, self-efficacy); and patient characteristics. Table 1 illustrates the framework domains, potential barriers to adherence of critical care nutrition guidelines, and example of potential questionnaire items. Patient preferences, lack of reimbursement, and malpractice liability were barriers present in Cabana et al.’s framework that were not observed to be relevant to critical care nutrition guidelines. The strategy or process for implementing the guideline, the prevailing culture of the ICU [33], and the characteristics of the ICU provider were new themes included in the revised framework. In addition, the term provider intent was chosen to replace physician behavior to better reflect the interdisciplinary nature of critical care, and since an individual’s intention to follow a guideline may not be synonymous with actual behavior [34, 35].

Individual interview summaries and the framework were sent to each key informant who participated in the case study for review and feedback. In addition, we held two face-to-face meetings with experts in the content area (i.e., ICU physicians and providers specializing in nutrition—e.g., dietitians, nurses, and physicians—respectively), and presented our findings to them to assess the comprehensiveness, clarity, and face validity of the developed framework.

The purpose of the questionnaire was to assess barriers to be targeted for change through a tailored guideline implementation strategy [19]. To this end, we intended the questionnaire to be administered to individual providers to determine their perception of the barriers to enterally feeding patients in the ICU in which they primarily work. To maximize the usefulness of the questionnaire, a priori it was decided to focus only on barriers that are amenable to change and can be targeted by intervention strategies to improve practices, rather than non-actionable barriers (e.g., patient case-mix). Acknowledging that national or society guidelines are frequently adapted locally, the questionnaire did not refer to any specific set of published ICU nutrition guidelines but asked respondents to refer to the guidelines currently being used to inform decisions about feeding in their respective ICUs. In addition, we focused on recommendations related to enteral nutrition (EN) only, rather than parenteral nutrition, nutrient supplementation, or nutritional assessment, because these recommendations are uniformly endorsed across published guidelines [1‐5], are supported by the highest level of evidence, and ICU providers generally agree with the recommendations [37].

In addition to our conceptual framework [28], potential items were identified through a literature review of studies of barriers to guideline adherence and by examining the content of existing barrier questionnaires developed in other settings [21‐24]. This initial list of potential items was circulated to experts to obtain input on item comprehensiveness and wording. Redundant or irrelevant items (i.e., represented non-modifiable barriers, or were not applicable to the ICU) were eliminated. Following item generation and reduction, in December 2009, a draft paper-based version of the questionnaire composed of 62 items, including 53 potential barriers, divided into four sections was pre-tested with a convenience sample of 32 critical care practitioners (11 physicians, 11 nurses, and 10 dietitians) from across Canada. Based on this pre-test, the questionnaire was revised and reduced further to 49 items, including 39 potential barriers, divided into four parts (Additional file 1).

Part A consisted of general questions about the ICU environment and the implementation of guidelines (nine items). Part B asked respondents about their level of agreement with the recommendations of critical care nutrition CPGs pertaining to enteral feeding (eight items). Part C focused on barriers to delivering adequate amounts of EN (22 items). Each item in Part A, B, and C used a seven-point Likert scale, to maximize the potential to discriminate among barriers and to allow a neutral response [35]. The items in Part A and B were formulated positively and end-anchored by the adjectives ‘1 = fully disagree’ and ‘7 = fully agree’ and included a ‘don’t know’ option. Parts A and B were intended to assess attitudes towards nutrition in general and the guideline recommendations specifically, because attitudes may influence an individual’s intention to feed and subsequent behavior, such that lack of agreement with these items indicates a barrier to feeding critically ill patients. The items in Part C were formulated negatively and end-anchored with the adjectives ‘1 = not at all important’ and ‘7 = very important’, with ‘very important’ indicating that the item is a major barrier and ‘not important’ indicating that it is not a barrier in their ICU. Each item in Part A, B, and C maps on to one of the five domains of the framework. In addition, Part C included four open-ended questions asking respondents to list additional important barriers to delivering adequate EN in their ICU, to list the most important barriers in their ICU, and to highlight strategies to overcome these barriers. In Part D (six items), characteristics of the respondent are captured.

The sampling pool for field testing was provided by seven ICUs from five hospitals in North America who were participating in a pre-test post-test study evaluating the feasibility of a tailored guideline implementation strategy (The PERFECTIS study [ClinicalTrials.gov identifier: NCT01168128]). At each ICU, in March 2010, all full and part-time physicians, nurses, dietitian(s), the Nurse Manager, and the ICU Manager were invited to complete the questionnaire (n = 409). If the nursing pool exceeded 85, a random sample of 60 nurses was used. To maximize the response rate, the questionnaire was distributed according to a modified Dillman’s tailored design method [38], and respondents were provided with the option of completing a web-based (survey monkey [39]), electronic (fillable pdf), or paper-based version of the questionnaire.

To confirm the results of the field testing and further refine the questionnaire, in May and June 2010 we conducted on-site observational visits at all five hospitals in the field test. Half-day focus groups were completed with ICU physician and nursing leaders, bedside nurses, and dietitian(s). Participants were first asked to reflect on EN provision in their ICU and identify areas where they perform well and areas for improvement. During these discussions, we explored the reasons (i.e., barriers and enablers) for high or poor performance. Attendees were asked to rank the identified barriers in order of their negative impact on the provision of nutrition (i.e., considering the degree of delay it caused and the frequency of its occurrence). Results of the barriers questionnaire were then presented to the group and the top ten ranked barriers from the questionnaire were compared with the rankings provided by attendees in the earlier discussion. A detailed report of these on-site visits is published elsewhere [40].

First, we conducted a descriptive analysis (e.g., missing data, variance, mean, histograms etc.). The frequency of non-response was examined, and items with a non-response of greater than 10% were reviewed and considered for re-wording or eliminated. The standard frequency distributions of responses to each item in the questionnaire were then examined for floor and ceiling effects. Items with a very high (>0.8) or low (<0.2) endorsement frequency (i.e., proportion of respondents responded ‘fully agree’, ‘agree’, or ‘somewhat agree’ in Part A or B and ‘’very important’, ‘important’, or ‘somewhat important’ in Part C) were considered for elimination, because responses to these items can be predicted and including them does not improve the scales psychometric properties [41].

To refine the content of the barriers questionnaire, reduce the number of items and ensure the most parsimonious representation of the underlying constructs, we conducted an exploratory factor analysis.

Missing values were treated as truly missing without imputation. A principal components analysis with varimax (orthogonal) rotation and kaiser normalization was used [42]. Eigenvalues of >1 (Kaiser criteria), the cumulative percentage of variance explained by successive factors, a scree plot, and at least three items with factor loadings greater than 0.5 were used, together with the underlying conceptual framework, to identify the number of factors [42]. Factor loadings of >0.5 were considered acceptable for item retention on a single factor [42]. Items that cross-loaded at >0.5 or loaded 0.4 – 0.5 on a single factor were evaluated by the research team on a case-by-case basis, retained or eliminated based on the item’s conceptual importance, its unique contribution to the factor, and whether it was strongly related conceptually to another factor. Following the descriptive and exploratory factor analysis, we revised the questionnaire.

We examined the reliability of the overall questionnaire and evaluated factor analysis derived subscales using Cronbach’s alpha. As individual questionnaire responses were intended to be aggregated to the ICU level to identify barriers pertinent to the ICU and inform a tailored intervention, a minimum overall and subscale co-efficient alpha of 0.8 was considered desirable, and any item for which alpha significantly increased if the item was deleted from the scale was considered for removal [41].

As the purpose of the questionnaire was to identify barriers to target through a tailored intervention, when scoring the questionnaire we focused on the upper end of the seven-point Likert scale. Item scores were calculated by awarding 1, 2, or 3 points if the respondent identified an item as a ‘somewhat important’, ‘important’ or ‘very important’ barrier respectively. If an item was rated 1 – 4 (i.e., ‘not at all important’ to ‘neither important or unimportant’ it was awarded 0 points. The barrier score was calculated by dividing the awarded points for each item by the maximum number of potential points (i.e., 3), and multiplied by 100. Each factor identified by the exploratory factor analysis was considered as a subscale, so that subscale and overall barriers scores were calculated as the mean score of all items within that subscale and mean of all items respectively. Subsequent analyses were conducted using the barriers score.

To be a useful tool for tailoring interventions it was also important to assess the extent to which individual responses approximate the barriers within their ICU. To this end, respondents were instructed to complete the questionnaire so that responses reflected the average situation in their ICU. To assess whether responses might be aggregated to the ICU level to obtain a single estimate of site-level barriers, we used three indices of within-group agreement and group mean reliability to examine each questionnaire item, subscale, and overall score [43]: intraclass correlation coefficient (1) (ICC [1]) (or Shrout and Fleiss model 1,1 [44]); ICC(2) (or Shrout and Fleiss model 1,k where k = 35 respondents [44]); and the F-test p-value. The variance components to compute the ICCs were calculated using mixed linear regression models with Restricted Maximum Likelihood (REML) estimation, and the F-test p-values were derived from a one-way analysis of variance (ANOVA). The aggregated data were considered reliable if the F tests’ p-values were <0.05 indicating that responses differ in different ICUs and/or ICC(2) (an estimate of the reliability of group means) was >0.60 [45]. ICC(1) is the ratio of between-group variance to total variance and is an estimate of the degree of reliability associated with a single providers assessment of the unit mean. Values of ICC(1) between 0.05 and 0.20 are typical in organizations [43].

Responses to the open-ended questions were reviewed to determine whether respondents identified barriers that were not already included in Part C.

Following completion of the analysis of the field test data, the research team met to review the results and revise the questionnaire. A revised version of the barriers questionnaire was circulated to ICU providers who had provided feedback on earlier drafts during the pre-test and field test. In March 2011, the final version of the barriers questionnaire was pilot-tested in a simple random sample of 60 providers working in a 20 bed closed ICU at a 404 bed teaching hospital in Canada. Using an open-ended format, respondents were asked for feedback and to report the time to completion; we made further revisions where required.

Finally, in May 2011 we administered the barriers questionnaire to a simple random sample of 60 full- and part-time nurses working in a 16 bed closed ICU in a 472 bed Canadian teaching hospital to assess test-retest reliability. The questionnaire was distributed on two occasions, two weeks apart, using the same methods of distribution as in the field test. ICC (Shrout and Fleiss model 2,1 [44]) was calculated between the original item, mean subscale and mean overall responses at the two time points. An ICC >0.7 was considered acceptable [46]. For each item, we also dichotomized nurses responses based on their rating of importance (i.e., ≤4 = not a barrier and >4 = barrier) and calculated kappa co-efficients. A kappa of 0.0 – 0.2 was considered poor agreement, 0.2 – 0.4 as fair agreement, 0.4 – 0.6 as moderate agreement, 0.6 – 0.8 as substantial agreement, and 0.8 – 1.0 as perfect agreement [47]. To further assess the degree of agreement and to identify potential bias Bland and Altman plots were also produced [48].

A priori, we estimated that each ICU participating in the field test would have approximately 80 staff members to whom the questionnaire would be distributed, and that the response rate would be approximately 50%, giving a sample size of 280 and a sample size to item ratio of 7 to 1. This sample size surpasses the recommended minimum of 150 cases and a sample size to number of items ratio of no lower than 4 to 1 for exploratory factor analysis [49]. For assessment of test retest reliability, we aimed to distribute 60 – 85 questionnaires, giving a sample of 30 – 43 when accounting for the anticipated response rate of 50%. This exceeds to 15 – 20 subjects recommended for estimating reliability [50].

The Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board, Kingston, Ontario and the seven hospitals participating in the field, pilot, and reliability testing approved this study (REB# EPID-292-09 and DMED-994-06). Return of the completed questionnaire and/or attendance at the focus groups implied informed consent on the part of participating critical care providers.

Observations with missing values on any of the items in Part C were omitted from the factor analysis (27 of 186 observations) resulting in 159 respondents for this analysis. The principal components analysis indicated a five-factor solution accounting for 72% of the variance. The eigenvalues for each factor and the factor loadings for each item after orthogonal rotation are shown in Table 5. The identified factors closely reflected the conceptual framework that guided the development of the barriers questionnaire.

The Cronbach alpha coefficient for the barriers scale was 0.94. The alpha coefficients for the factor subscales all exceeded the acceptable cut-off of >0.8, ranging from 0.84 to 0.89.The alpha when an item was deleted remained stable for each item, with the exception of one item in factor five (Not enough nursing staff to deliver adequate nutrition) (Refer to Table 5).

The variance components and indices to assess the reliability of aggregating individual provider responses to the unit level are shown in Table 6. A total of 11 of the 21 questionnaire items included in the analysis and subscales two, three, four, and 5, demonstrated statistically significant F statistics and ICC(1,35) values >0.6 in the acceptable range. The ICC (1,1) was greater than 0.05 typically observed in the organizational literature for 10 of the 21 items. However, the values for all three indices were not acceptable for the overall and subscale one scores.

A total of 46 providers participated in the five focus groups, ranging from three to 14 attendees per group. Overall, at each site the important barriers reported by the attendees during the discussions on nutrition performance were conceptually the same as the top ranked barriers derived from the responses of ICU providers at their site.

Of the 39 potential barriers in the field test version of the questionnaire, seven items in Part A and the eight items in Part B were omitted from the revised version, leaving 24 of the original items. Of these items, three (A.8, A.9, and C.22) were reworded. In addition, two new items were added for a total of 26 potential barriers. The revised version of the barriers questionnaire presented (Additional file 2 and available at http://​www.​criticalcarenutr​ition.​com) in this paper consists of two sections. The first section lists the 26 potential barriers to delivery of EN and asks the respondent to rate their importance as barriers in their ICU (See Table 7). These items are followed by two open-ended questions. The first open-ended question asked respondents if there are any other barriers that hinder their ability to deliver adequate EN, and the second asked respondents to rank the three most important of 26 potential barriers to the provision of adequate EN in their ICU. Part B includes six questions about the personal demographics of the respondent. The revised questionnaire has 34 questions.

The pilot test of the revised questionnaire in a separate sample of 43 nurses (response rate 72%) demonstrated completion time was less than five minutes. No further changes were made based on the pilot test feedback.

Of the 60 distributed questionnaires, a total of 17 nurses completed the questionnaire on two occasions, two weeks apart for a response rate of 28%. The ICC (2,1) for total barriers score was 0.64, with subscale scores ranging from 0.39 – 0.62. Only one of the individual items demonstrated acceptable correlation of >0.70. Item ICCs ranged from −0.13 to 0.70. The kappa coefficients were similar to the ICC; three items demonstrating substantial or almost perfect agreement (Table 7). The Bland Altman plots did not indicate any bias between the two observations (Figure 1a and b).