Psychometric test of the Team Climate Inventory-short version investigated in Dutch quality improvement teams

This study included team members of quality improvement teams participating in the quality collaborative of the Care for Better program for home care, care for the handicapped and the elderly in the Netherlands between 2006 and 2008 (for a more detailed description see [14]). These quality improvement teams were participating in the following projects: pressure ulcers, eating and drinking, prevention of sexual abuse, medication safety, fall prevention, aggression and behavioural problems and autonomy. The collaborative used the Breakthrough method as major instrument to quickly spread evidence-based practices across care organisations and enable mutual learning across sites. Although the topics of improvement were different for these projects, the set up of the projects, working with the Plan-Do-Study-Act cycle and starting off with small scale changes first, is the same.

Each improvement team consists on average of one project leader and four other team members. Two months after the start and at the end of each Breakthrough project each team member received a postal questionnaire as part of a larger overall evaluation study. For this study data from three samples were used. The first sample consisted of data from baseline questionnaires (T0) for still running projects and for which no end-measurement data are available yet. In total 125 teams received a questionnaire. Project leaders of 79 teams filled in the baseline questionnaire (T0), resulting in a response rate of 63.2% for project leaders. In total 219 other team members filled in the questionnaire. Response rate of the other team members is difficult to estimate, since no accurate data on the number of team members of teams whose project leader did not fill in the questionnaire were available. The response rate of members of teams whose project leaders did fill in the questionnaire was 62%. After excluding respondents with missing values on the TCI items a total sample of 270 respondents was left for analysis with the T0-sample.

The second sample consisted of end measurement data (T1) only, since several projects already started before the evaluation started and baseline measurement therefore was impossible. In total 83 teams received an end measurement questionnaire. Project leaders of 38 teams filled in the questionnaire, resulting in a response rate of 45.8%. This lower response rate is partly due to the fact that improvement teams participating in projects on pressure ulcers, eating and drinking, and prevention of sexual abuse were not informed beforehand about the evaluation study. In total 101 other team members filled in the questionnaire. A total sample of 139 respondents was left for analysis with the T1-sample.

The third sample consisted of respondents who filled in the baseline questionnaire, as well as the end measurement questionnaire. At this point of the evaluation study only two projects could be studied from baseline until the end of the project. Therefore, this third sample consisted of 38 respondents for whom baseline and end measurement data were available.

The 14-item Team Climate Inventory (TCI) was assessed at baseline and end measurement (see Appendix 1). Two researchers independently had translated this instrument into Dutch. Comparison of the two translations revealed no salient differences and the two researchers agreed upon the final Dutch translation. The TCI has a 5-point response scale from 'strongly disagree' to 'strongly agree', in which higher scores indicate a better or more desirable team climate. Scores for each item in a scale are summed to determine the scale score.

In the end measurement questionnaire, team members' perceived team effectiveness was assessed by the 'perceived effectiveness' measurement instrument of Lemieux-Charles [15]. Four questions, each using a 5-point response scale, assessed the extent to which team members: (1) believed that their team's overall performance met their expectations; (2) were satisfied with their experience as a team member; (3) felt positive about their experience; and (4) would be willing to work on a similar team in the future. The reliability of the total scale was 0.76.

The psychometric analyses comprised three parts. First, the psychometric properties of the Dutch version of the 14-item TCI were investigated by evaluating construct validity. Construct validity was analysed by conducting a confirmatory factor analysis using the LISREL program [16] to test whether or not the items loaded on the intended dimension. Each subset of measured items was allowed to load only on its corresponding latent variable derived from the four-factor theory (i.e. vision, participatory safety, task orientation and support for innovation). No correlation errors either within or across sets of items were allowed in the model, which approach is in line with previous studies of Anderson & West [7] and Kivimäki and Elovainio [9]. Furthermore, in line with the theoretical assumptions of the measurement instrument non-zero correlations between the four factors were allowed.

To test the measurement models in LISREL four indices of model fit were used. The cut-off criteria for these four indices were those proposed by Hu and Bentler [17]. First, the overall test of goodness-of-fit assesses the discrepancy between the model implied and the sample covariance matrix by means of a normal-theory weighted least squares test. A plausible model has low, preferably non-significant χ² values. However, Chi-square is overly sensitive when the sample size is large (anything over 200 [18]), leading to difficulty in obtaining desired non-significant levels [19]. Secondly, the Root Means Square Error of Approximation (RMSEA) reflects the estimation error divided by the degrees of freedom as a penalty function. Values on RMSEA below 0.06 indicate small differences between the estimated and observed model. Thirdly, we used the Standardized Root Means square Residual (SRMR), which is a scale invariant index for global fit that ranges between 0 and 1. Values on SRMR lower than 0.08 indicate a good fit. As a fourth index of model fit the Incremental Fit Index (IFI) was calculated. This index compares the independence model (i.e. observed variables are unrelated) to the estimated model. Preferably, values on IFI should be larger than 0.95.

The second part consisted of reliability analyses of the subscales. Internal consistency of the subscales emerging from the factor analysis was assessed by calculating Cronbach's alpha coefficients. After having defined the structure of the questionnaire and reliability of the subscales, Pearson's moment correlations between the factors identified by the factor analysis were computed and descriptive statistics were analysed.

The third part consisted of investigation of predictive validity and temporal stability. In order to further investigate predictive validity of the TCI, correlations of each subscale with perceived effectiveness were calculated based on the T1-sample. Furthermore, univariate regression analyses were performed. Temporal stability was analysed by conducting paired samples t-tests for each subscale based on the sample of 38 respondents for whom both baseline and end measurement data were available.

The majority of the team members that filled in the T0 or T1 questionnaire was female, which is in accordance with the gender distribution of health care professionals. The mean age was 42 years (sd 9.9) for the T0 sample and 43 years (sd 9.2) for the T1 sample. More than two thirds of the respondents had been working for more than 3 years within the organisation. Furthermore, 232 (85.9%) team members at baseline and 94 (69.1%) team members at T1 worked more than 29 hours per week. Improvement teams mainly consisted of nurses and management staff. Table 1 lists descriptive characteristics of the two samples of team members.

The results of the confirmatory factor analysis with LISREL based on the T0-sample showed a good fit, as indicated by RMSEA of 0.03 – well below the boundary of 0.06 – and IFI of 0.98. The Normal Theory Weighted Least Square χ² was 263.66 and significant (p = 0.0) and the SRMR was 0.05, and thus below the cut-off point of 0.08. In table 2 the standardized solution of the four-factor structure is given. Standardized loadings of the 14 items varied between .68 and .87. Confirmatory factor analysis based on the T1-sample with 139 respondents showed similar results, which are also reported in Table 2.

Reliability analyses based on the T0-sample showed alpha coefficients of the four subscales between 0.73 and 0.80, indicative of satisfactory reliability for all scales of the TCI (see Table 3). Significant positive correlations between subscales were found, ranging from 0.43 to 0.63. Furthermore, the inter-item correlations within each subscale were high. The inter-item correlation between item 2 and item 4 of the vision subscale was the lowest at 0.36. Analyses based on the T1-sample with 139 respondents showed similar results, which are reported in Table 4.

As a final step in the validation of the TCI, its predictive validity was analysed by investigating univariate regression analyses. Participatory safety emerged as the best predictor of perceived effectiveness, accounting for 29% of the variance; vision and support for innovation explained respectively 23% and 24% of the variance. Task orientation, which explained 18% of the variance, appeared to be the poorest predictor.

Temporal stability was analysed by conducting paired samples t-test for the group of 38 respondents for whom baseline and end measurement data were available. Between T0 and T1 no significant changes were found for each of the four subscales (see Table 5).