The German version of the high-performance work systems questionnaire (HPWS-G) in the context of patient safety: a validation study in a Swiss university hospital

Before the current study, one previous conceptual model linked HPWSs to patient safety: Garman et al. (2011) indicated that, as ‘organisational factors’, HPWSs can influence employee and organisation-level outcomes – including patient safety – via multiple pathways [3] (Fig. 1). Two additional critical safety factors affected by HPWS and influencing the same outcomes were staffing and care processes [3, 7]. Chuang et al. (2012) identified three HPWS practices; supervisor support, team-based work practices and flexible work arrangements positively associated with job satisfaction and, when complemented with performance-based incentives, positively associated with frontline health care worker’s perceived quality of care [7]. Other health care related studies revealed that HPWS is positively associated with job satisfaction [12, 13], whereas job satisfaction positively affects perceived quality of care [14, 15]. In systematizing organisational behaviour in terms of three interrelated aspects – culture, structure and processes – Guldenmund’s organisational triangle [16] depicts HPWSs and safety culture as dynamically interrelated. Springing from the ‘underlying values, beliefs and behaviours’ (e.g., at the patient care team and institutional levels), safety culture influences [17, 18] ‘how safety is viewed and treated in an organisation’. And while it is not possible to measure safety culture directly, it can be evaluated in relation to safety climate [9, 10, 17, 19], the surface features of which indicate the characteristics of the underlying safety culture [9].

In contrast to safety culture, safety climate [18] is an aggregation of the ‘shared perceptions of employees about safety relevant aspects’ of their clinical workplaces. By measuring HPWSs, findings supported the association between HPWSs most strongly associated with safety climate [5], e.g. the relationship between HPWSs and higher patient safety scores, lower rates of patient mortality and medication errors [2, 4, 20]. Qualitative study data also indicate that HPWS practices facilitate employees` speak up, an important factor in the context of patient safety culture [21]. Further important factors influencing clinical practice and patient safety include organisational learning and the teamwork climate.

Despite challenges such as construct underrepresentation or construct-irrelevant variance [2], measurement of HPWSs in health care is gaining attention. One promising barometer of HPWS success is Etchegaray et al.’s [5] US-developed and tested 10-item HPWS questionnaire. Based on a literature review and hospital senior executive ratings, this instrument assesses HPWS practice elements such as rewards, employee surveys or job security. The scale showed good psychometric properties (Cronbach’s α = .92); confirmatory factor analysis yielded good construct validity (GFI = .92; CFI = .94; RMSEA = .06) [5]. The HPWS questionnaire allows measurements in health care organisations, while developing or selecting HPWS target practices that promote patient safety outcomes via safety culture.

HPWS assessment provides a basis for intra- and inter-hospital benchmarking. Studying their relationships with patient safety outcomes can illuminate factors that potentially provide leverage points for interventions. As each health care context has a unique set of policy and organisational parameters, it is necessary to assess whether observations in the US also hold for other parts of the world. This should be done via tools validated in the target context. As no instrument is yet available to assess and explore HPWS in German speaking health care settings, this study’s two purposes were (1) to translate the original English-language HPWS questionnaire into German (HPWS-G) and rate its content validity; and (2) to examine its psychometric properties, including internal consistency, construct validity and concurrent validity.

Along with 10 HPWS-G questionnaire items (Table 1), the questionnaire included demographic variables such as profession, clinical area, work experience, gender, and age. The experts rated each item’s validity on a 4-point Likert scale (‘not relevant’, ‘somewhat relevant’, ‘quite relevant’, to ‘highly relevant’). Comprehensibility was rated as a ‘yes’/‘no’ response. For both validity and comprehensibility, space was given for additional comments.

After being personally invited to participate, each eligible expert was provided written information about the study’s purpose and a paper copy of the 10-item HPWS-G questionnaire via internal post.

All 10 item responses were dichotomised as ‘relevant’ (quite or highly relevant) or ‘not relevant’ (somewhat or not relevant) [25]. The item-level content validity index (I-CVI) was calculated by dividing the number of ‘relevant’ ratings for each item by the total number of surveyed experts. To determine the content validity of the overall scale (S-CVI/Ave), all I-CVIs were summed and divided by the number of items [25]. An I-CVI of .78 and S-CVI/Ave values greater than .90 indicated good content validity [26].

Survey data collection took place between 04.10. - 17.11.2017. All eligible RNs and physicians received the 37-item paper-pencil questionnaire via internal mail. A reminder was sent to all participants after two and four weeks, as the period of data collection was extended for two weeks.

Data were analysed using IBM® SPSS® 24.0.0. For the confirmatory factor analysis (CFA), we used IBM® SPSS® AMOS™ 24.0. Descriptive analysis included each item’s frequencies, percentage, mean, median, interquartile range (IQR) and standard deviation (SD). Data were screened for out-of-range values, homogeneity of variances and univariate outliers by preparing boxplots, histograms and scatterplots [31]. To detect multivariate outliers, Mahalanobis distances were calculated [31]. The level of significance was set at p < .05. Before analysis, data were checked for plausibility and comprehensiveness.

The scale-level content validity index (S-CVI) of the 10-item HPWS-G questionnaire was .86, indicating overall good content validity. For the individual items, the content validity (I-CVI) varied between .83 and 1, also indicating good content validity per item. According to the study experts’ ratings and comments, the wordings of five items (‘skills’, ‘information’, ‘teamwork’, ‘appraisal’, and ‘quality’) were slightly revised, or examples added to ensure comprehensibility. After these adaptations, content validity was not re-examined by the experts.

Conducting a missing values analysis, we found more than 5% missing values in demographic items including age (15.7%) and work experience (5.3%) as well as in the HPWS ‘candidate’ item (6.0%). As Little’s MCAR test yielded a statistically significant result (p = .001), we inferred that these data had been missed at random [31]. To conduct a CFA, we replaced missing values with the person-specific mean or, where appropriate, group specific means (items 29¹ and 30²) of the available data.

Analyses of boxplots, histograms and scatterplots identified two univariate outliers in ‘patient safety grade’. Via the Mahalanobis distance, we detected six (2.5%) multivariate outliers. We calculated regressions for all to highlight what distinguished them from the other cases [31]. In comparison to the main sample, their scores were skewed more extremely, as they often involved the answer option ‘disagree strongly’ or ‘agree strongly’. As we assumed that these cases accurately represented a segment of the population, they were kept in the data set. However, as such cases tend to distort results, robust statistics and methods, such as the median or Bollen-Stine bootstrap, were used [33]. Descriptive analyses of the questionnaire items showed a number of significant response behaviour differences between nurses and physicians, between clinicians with and without leadership functions, and between clinical areas.

H2 was supported by a Cronbach’s α of .846. Mean inter-item correlation was .448, and item-scale correlation ranged from 2.77 to 4.62 (Table 4). While ‘teamwork’ and ‘job security’ items showed low discriminating power (.224 and .366), all items were initially retained, as the greatest α increase – by deleting the ‘job security’ item – would be .008. Based on the low factor loading (.053), we decided to delete the ‘teamwork’ item. This produced a corrected Cronbach’s α of .853.

A CFA based on the remaining nine items initially confirmed the proposed 1-factor model with HPWS as latent factor (p (chi²) = .215). Fit indices indicated acceptable to good model fit³ with chi² = 40.880 (df = 27, p = .205⁴), GFI = .968, RMR = .025, NFI = .770, CFI = .902, and RMSEA = .043 (90% CI = .008–.068). All factor loadings except for the ‘information’ and ‘job security’ item were significant (p < .05) and their size was excellent (> 0.70) [40] (Table 5). However, examination of the modification indices showed significant covariations of certain error variables, suggesting that the model was not unidimensional. To explain these results and improve our factor structure, we therefore applied additional statistics as follows. To examine whether local dependency and unidimensionality were violated, leading to biased parameter estimations, we applied item response theory [41]. After the contribution of the latent trait had been removed, this showed some significant correlations among the items. As the highest correlation between the items ‘information’ and ‘workplace’ was negative (r = −.354, p < .000) and other significant correlations were below r < .03, the items’ locally dependency was not clearly confirmed [42]. However, conducted MRFA indicated an ECV index of .888 (95% CI⁵ = .873–.916) and I-ECV values above 0.70, except for the item ‘skills’, suggesting a unidimensional solution. This was supported by a MIREAL of .214 (95% CI^e = .178–.251) as well as seven of nine items having I-REAL values lower than 0.30. Furthermore, the generalized H (G-H) indices of both factors, determined for a multidimensional solution, were below .80, with .670 (95% CI = .611–.692) for factor 1 and .764 (95% CI = .730–.778) for factor 2, indicating poorly defined latent variables [39]. Considering these results, for HPWS the one-factor model seems to be most appropriate.

‘HPWS’ correlated significantly with all five dimensions, i.e., ‘safety climate’ (r_s = .657, p < .01), ‘teamwork climate’ (r_s = .615, p < .01), ‘organisational learning’ (r_s = .660, p < .01), ‘critical incident reporting’ (r_s = .438, p < .01), and ‘patient safety grade’ (r_s = .575, p < .01). Following Cohen’s [43] directions, we found that the ‘critical incident reporting’ dimension’s effect was medium (r > .30); all others’ were large (r > .50). These results support our hypothesis. Additional bivariate correlation tests indicated that HPWS was the strongest predictor for safety climate (r = .673, p < .01), teamwork climate (r = .641, p < .01), and patient safety grade (r = .567, p < .01).

Analysing response patterns revealed differences in response behaviour across professions and clinical areas, as well as across management levels. Regarding safety climate and teamwork climate, such differences have been reported elsewhere [28, 44], highlighting the danger of inferring generalizability across professions or clinical areas.

One of the three items with more than 5% missing values was the HPWS ‘candidate’ item. In that case, we assume that not all respondents could judge whether the best candidate was hired for each job given their position in the team. Considering the above-mentioned inter-group response differences, it might be advisable to use that item only in HPWS questionnaires for respondents with leadership functions. For demographic items, respondents expressed concerns about their anonymity, although we merged the clinical areas, the medical and surgical ICUs, as well as the OR and anaesthesiology.

The HPWS-G’s Cronbach’s α was lower than that of the original version, but was improved somewhat by deleting the ‘teamwork’ item. Initially, we retained both the ‘teamwork’ and ‘job security’ items despite their low discriminant power, as they play important roles in the HPWS conceptual model [3]. Regarding the ‘teamwork’ item, in combination with its low factor loading in the first factor analysis and the fact that it made a statement⁶ without raising a condition, we decided to exclude it. In fact, the same item had been excluded from the original HPWS questionnaire [5]. The ‘job security’ item also showed low discriminant power – possibly resulting from its strongly right-skewed distribution (which can be improved by reformulating the item [45]); however, its standard deviation was good. Differing national-level employment policies may explain why job security appears less important to the Swiss sample than to those in the US [46].

The suggested 1-factor HPWS-G model showed an overall acceptable to good fit. The low NFI value might have resulted from underestimation of fit due to the relatively small samples and violations of multivariate normality [47, 48]. Other fit indices such as the CFI overcome these problems [47]. The 1-factor model fits the results of Etchegaray et al. [5], also evidence that the HPWS–G questionnaire distinguishes between safety climate and teamwork climate. Our data did not confirm this, as safety climate and teamwork climate were not included in the CFA. However, HPWS as unidimensional construct is not equivalent to the four HPWS subsystems mentioned in Garman et al.’s [3] model (‘engaging staff’, ‘aligning leaders’, ‘acquiring/developing talent’, ‘empowering the frontline’). One possible reason for this is a lack of a standard HPWS-related terminology, as well as uncertainty about which HPWS patient safety-related practices, and in which combinations, are most promising.

The HPWS-G questionnaire correlates well with safety climate and teamwork climate scales, subscales for patient safety grade, and items on organisational learning and CIRS. In Garman et al.’s [3] conceptual model, HPWS’s influence on patient safety is mediated by staffing and care processes, which could not be shown in our study. Unlike Etchegaray et al.’s findings [5], our analyses indicated that HPWS was a stronger predictor of safety climate than of patient safety grade. Two other studies support this result. First, Guldenmund’s organisational triangle theoretically maintains [16] that safety climate interrelates dynamically with HPWSs and can influence patient safety. Second, Zacharatos et al. [49] showed a mediating effect of safety climate between HPWSs and both safety incidents and personal-safety orientation. Additional qualitative study findings from health care organizations in the U.S. highlight HPWS as crucial element, facilitating speak up, an important factor in the context of safety climate [21].

Transferred to the health care setting, these findings suggest that clinicians should regard the presence of HPWSs as a predictor of safety climate, which in fact impacts outcomes including patient safety (as HPWSs bolster efforts to prevent adverse events [27]). According to the World Health Organization [50], successful adverse event reduction strategies would lead to ‘over 3.2 million fewer days of hospitalisation, 260´000 fewer incidents of permanent disability, and 95´000 fewer deaths per year’ in the European Union alone. Measures reducing adverse events by increasing patient safety and quality focus mainly on the improvement of safety climate [51]. Facilitating development of a robust safety climate – which leads to associated clinical outcomes including, for example, reductions in central line-associated bloodstream infections [52] or patient mortality [20] – is one clear way HPWSs improve patient safety. However, further longitudinal studies will be necessary both to investigate causality between these work systems and safety related outcomes and to support the very encouraging findings reported here and elsewhere.

Considering this study’s known limitations, its results should be interpreted with caution. The convenience sampling procedure and the fairly low response rate may weaken generalizability. However, compared to the validation study of the original English HPWS questionnaire, response rates differ only slightly (35.9% vs. 37.4% [5]). Also, the validation process took place in three high performing clinical areas of the same hospital. Due to language and cultural differences influencing relevance, comprehensibility, and wording, HPWS items may be rated differently by respondents from other units in the same hospital or other hospital settings in Switzerland. As participation was anonymous, evidence of response processes could not be assessed; instead we analysed response patterns. To replace missing values the advantage of chosen procedure (person-specific mean imputation) is that ‘the mean for the distribution as a whole does not change’ [31]. However, variance of the variables could be reduced [31].