Background
The Institute of Medicine has identified that gaps in organisational culture may contribute to suboptimal patient safety [
1]. Consequently, there has been a growing trend for healthcare organisations to measure patient safety culture. Safety culture refers to
“the product of individual and group values, norms, attitudes, beliefs, perceptions, competencies and the patterns of behaviour that determine the commitment to…an organisation’s health and safety management” (pg ii18) [
2]. There are numerous challenges associated with defining the measurable components of safety culture [
3]. Safety climate, which has been described as the shared perceptions, attitudes and beliefs of employees about the way in which a hospital manages and achieves patient safety [
3,
4], has been used to provide a snapshot of the safety culture of an organisation.
Numerous questionnaires have been developed to quantify safety climate [
4‐
6]. One of the most frequently evaluated and widely used is the Safety Attitudes Questionnaire (SAQ) that assesses six safety-related climate domains including teamwork climate; job satisfaction; perceptions of management; safety climate; working conditions; and stress recognition [
7]. It has been translated into a variety of languages and used in different countries including Taiwan, Norway, Brazil, Germany and Sweden [
8‐
13]. To date no study has validated the SAQ for use in Australian hospitals.
The SAQ has previously been shown to have good internal consistency, test re-test reliability and predictive validity [
7‐
9,
11,
12,
14]. The factor structure of the SAQ has also been tested with confirmatory factor analysis (CFA) [
9,
11,
13]. Whilst the construct validity of the questionnaire can be considered to be acceptable, some studies showed a degree of misfit with the CFA model [
9,
12]. This suggests that some items in the SAQ may not be measuring the same underlying safety climate construct. For instance, the stress recognition domain does not correlate strongly with other SAQ domains [
15,
16]. This may have implications on the overall validity of the SAQ as it is unclear what construct the stress recognition scores are measuring [
15‐
17]. Further psychometric evaluation is therefore required to provide greater detail on the measurement properties of the SAQ. This will inform its use in quantifying the perceived climate of patient safety in a specific clinical environment.
Rasch analysis is a modern psychometric approach based on latent-trait modelling that allows examination of key measurement and scaling properties of an outcome tool [
18]. Rasch modelling enables the conversion of equal units of measurement from raw (ordinal data) scores on items of a questionnaire to interval-level scores [
19]. It also provides an opportunity to examine the unimensionality of domains (i.e. measurement of one underlying construct), ceiling and floor effects and whether or not items are ‘biased’ for specific groups for example based on clinical speciality (differential item functioning [DIF]) [
18]. As such it is argued that the Rasch measurement model is the standard for evaluating the psychometric properties of scales. This is despite the limitation of Rasch analysis which requires a large number of observations to estimate the parameters of the model [
20].
This study aimed to extend the psychometric evaluations of the SAQ Short Form [
7] by examining the internal construct validity of the questionnaire using Rasch analysis in the Australian context. This would allow us to: (1) examine the unidimensionality of the six SAQ domains; (2) investigate the response formats of the questionnaire; (3) assess whether the six SAQ domains are appropriately targeted for the clinical population (floor and ceiling effects); (4) examine the extent to which items distinguish between different levels of safety climate; and (5) assess whether different groups within the sample (e.g. medical versus surgical wards), despite equal levels of the underlying characteristics being measured, respond in a different manner to an individual item.
Discussion
Given the emerging evidence linking positive safety climate with improvements in patient safety outcomes [
28,
29], there has been a reliance on safety climate questionnaires to identify specific wards or units with low levels of safety climate to guide the implementation of strategies to improve safety culture. However, little is known about whether a safety climate questionnaire such as the SAQ adequately measures the safety culture of an organisation or clinical unit [
30]. This study provides new information about the internal construct validity of the SAQ using Rasch analysis in the Australian context. It is one step towards increasing the understanding of safety climate in the measurement of patient safety culture. The Rasch measurement model is recognised as the gold standard for psychometric evaluations of outcome scales [
20]. It has been recommended that Rasch analysis is used during the development phase or when reviewing the psychometric properties of existing questionnaires [
20,
31]. Findings from this study can be used to inform the refinement of the SAQ to improve its psychometric properties in order to accurately measure safety climate in clinical environments.
All six SAQ domains demonstrated unidimensionality and the responses to items in each domain were not dependent on another item. This provides support for summing the items in each domain [
20]. Nevertheless, there were some items in the teamwork climate and perceptions of management domains that may not measure the same underlying construct, as indicated by positive item fit residual values greater than 2.5. In addition, there were potentially some redundant items in the job satisfaction and perceptions of management domains. The presence of disordered thresholds may have affected the fit of these individual items because participants had difficulty distinguishing between the strongly disagree and disagree response options [
18]. The option of a midpoint ‘neutral’ category in the SAQ may have also contributed to the disordered thresholds. It may be worthwhile for future studies to use Rasch analysis to examine whether changing the response options (for example, removing the neutral option) or removing the redundant items (for example, items 18, 24 and 26) may improve the overall model fit of the SAQ. This may improve the ability of the SAQ to distinguish between different levels of safety climate in a clinical setting.
All six domains of the SAQ appeared to have suboptimal targeting. This is particularly evident in the Rasch analyses of the person and item distributions, where all domains demonstrated substantial ceiling effects (Fig.
2). The lack of measurement precision observed may be due to sampling effects because targeting relates to the characteristics of the investigated sample [
18]. The inclusion of only nurses in this study may have also contributed to the floor and ceiling effects observed. Further investigations using other health professionals including hospital executives would be beneficial in order to determine whether the level of safety climate assessed by the SAQ is consistent with staff working in the clinical environment.
Floor effects or low levels of safety climate were also not represented in the job satisfaction, perceptions of management and stress recognition domains. This finding may have implications on how the SAQ can be used as a tool to quantify the levels of safety climate in an organisation as it may not be able to detect small but clinically important changes in safety climate. Given the need for accurate measurement tools to drive improvement in patient safety and optimise resource allocation [
32], further refinement of the SAQ is warranted. This may involve rewording existing items in order to improve the measurement of safety climate at either ends of the scale. It may also be beneficial to include items from other safety climate questionnaires such as the Hospital Survey on Patient Safety Culture (HSOPS) [
33]. This may improve the overall targeting of the SAQ as the item pool is expanded through the use of an item bank [
32,
34], which allows a set of items that measures a single construct to be selected without substantial loss of measurement precision [
34].
Strengths and limitations
One of the strengths of this study was that it was a multi-centre design and included 420 nurses working across 24 acute hospital wards. Additionally, the sample for analysis had worked on the participating wards for a substantial period of time. Most nurses had worked within the organisation for at least one year and had more than two shifts a week. This means that they were aware and conscious of the level of safety climate on the ward. However, the limitations of this study must also be considered. Firstly, participants were nurses working in acute hospitals. We did not include other health professionals such as doctors and allied health staff, which markedly limits the generalisability of the safety climate findings. Secondly, questions regarding falls prevention strategies in the acute setting were combined with the SAQ items. These additional questions were closely related to patient safety and may have affected how nurses responded to the SAQ questions. They may have been more aware of how the ward may or may not be managing patient safety, which may explain the lower levels of safety climate observed in this sample. There is also a potential the sample may be biased towards wards with lower levels of safety climate as it included wards where ‘falls commonly occurred’ and had ‘low levels of use of falls prevention strategies’ [
21]. There may be less nurses in this sample with a positive safety climate attitude compared to the general population. This may have implications on the precision of the estimates from the Rasch model, particularly with respect to targeting and item difficulty. Finally, caution is required when interpreting the results for item bias because differences in responses to the SAQ were not examined based on the age or gender of nurses working in the acute wards.
Recommendations
The SAQ has demonstrated adequate internal consistency reliability as a measure of safety climate in acute Australian hospital. It is also appropriate to sum items in each domain without weighting or standardisation. The results of the Rasch analysis, however, suggest that further refinement of some items and response options may be warranted in order to minimise the floor and ceiling effects and improve overall model fit. This may involve rewording existing items and including new items to accurately measure small but clinical meaningful changes in safety climate. We also recommend that further validation work of the SAQ be undertaken in different settings and amongst different health professionals in order to improve our understanding of safety climate in the measurement of safety culture in Australian hospitals. This includes examining the variability in safety climate across hospitals and whether these differences may be associated with the incidence of patient safety outcomes such as falls, pressure injuries and medication errors.
Conclusion
This is the first validation study of the SAQ using the Rasch measurement model and has provided important insights into the internal construct validity of the SAQ. We found some limitations associated with some items not measuring the same underlying construct as well as substantial floor and ceiling effects. This may limit the ability of the questionnaire to precisely measure the underlying levels of safety climate in a clinical environment. Additional research is needed to refine the SAQ. Further studies linking levels of safety climate with patient safety outcomes including falls and fall-related injuries also warrants further investigation.
Acknowledgements
The authors thank the contribution of Sheral Rifat and Elysia Greenhill for their ongoing assistance with data management. The study could not have been completed without the collaboration and support from the participating hospitals. The authors would also like to acknowledge the contribution of the 6-PACK investigator team: Rory Wolfe, Terry P Haines, Keith D Hill, Sandra G Brauer, Mari Botti, Robert G Cumming, Patricia Livingston, Catherine Sherrington, Silva Zavarsek, Richard I Lindley and Jeanette Kamar.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (
http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (
http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.