Sound psychometric properties of a short new screening tool for patient safety climate: applying a Rasch model analysis

All around the globe unsafe care is a major contributor to death and disability [1]. In higher income countries one out of ten patients are expected to come to harm in inpatient care [2], up to half of the caused harm is deemed to potentially be preventable [3]. To achieve safer health care delivery the culture of the health care organizations has been increasingly emphasised [4]. Patient safety culture is defined as “a pattern of individual and organisational behaviour, based upon shared beliefs and values that continuously seeks to minimise patient harm, which may result from the process of care delivery” [5]. Culture which are the norms and values of an organization is often studied with methods developed in the ethnographic field of research [6].

Alongside the existing body of literature on patient safety culture, the field of patient safety research also employs the term "patient safety climate" (PSC). Although related, patient safety culture and patient safety climate are distinct concepts in the healthcare domain [7]. Patient safety climate concerns the frontline staff's attitudes towards patient safety in their work environment. It is a narrower aspect of patient safety culture that concentrates on how individuals perceive and understand the patient safety culture within their organization [8].

There is growing evidence to support the correlation between PSC and health care outcomes [9]. Review studies have shown that more than 70% of studies report positive associations between PSC and outcomes in the form of reduced readmissions, length of stay and medication errors [10, 11]. WHO encourages governments around the world to “adopt global approaches for establishment of safety culture across the health system” [1].

In the WHO safety action plan (2021–2030) hospitals are recommended to perform regular surveys of the organization’s PSC [1]. In the Organisation for Economic Co-operation and Development (OECD) working paper on patient safety De Bienassis and Klazinga states that “Without measurement and analysis of the status of PSC in health care settings, it becomes virtually impossible to detect and reinforce beneficial trends that enhance patient safety” [9].

Most studies on PSC have been performed in hospital settings with focus on hospital staff [3]. Fewer studies on PSC have been performed in long term care settings and in primary care [9]. However, the authors of this paper have not identified any studies on PSC in privately owned specialist care provider. Earlier studies have indicated differences in reported patient safety in public and private health care [12] and it is therefore important that instruments are validated in both contexts. Instruments also needs to be validated in both emergency hospitals and planned outpatient care as these contexts may differ in ways of working and experience of PSC.

In 2004 the United States Agency for Healthcare Research and Quality (AHRQ) published a survey on PSC (SOPS®) with an update in 2019. The SOPS® 2.0 reduced the included items from 42 to 32 and the measured dimensions from 12 to 10 [13]. Another survey on PSC is the Safety Attitudes Questionnaire (SAQ) that includes 6 dimensions and 30 items [14]. According to De Bienassis and Klazinga these are the two most widely used surveys for international benchmarking of PSC [9]. However, both surveys are relatively extensive, encompassing ≥ 30 items each.

Survey fatigue is a well described phenomenon describing how respondents tire of answering questionnaires which may cause low response rates and potentially affect validity of the survey [15, 16]. Response rates and the quality of the responses may be affected by the length of the survey, the topic and the complexity of the questions [17].

In addition, surveys with multiple dimensions may come with an inherent risk of “diluting the domain” of PSC [18], consequently lowering the validity and strength of conclusions drawn from the results. The above risks have called for more parsimonious models and shorter surveys in PSC measurements [18].

The psychometric properties of the HSE questionnaire were evaluated using Rasch analysis. The analysis assessed item performance, fit to the Rasch model, and questionnaire reliability and validity. A summary of the findings is presented in Table 3.

Step 1: Evidence based on rating scale response processes. The average measures for the response categories advanced monotonically with an outfit MnSq < 2.0 for all scale steps. However, as the category probabilities curve (Fig. 1) show the category 2 was almost completely covered by category 1 and category 3, the added value of step 2 could therefore be considered limited. As the scale steps met our set criteria, we did not collapse scale steps, but proceeded with the analysis.

Step 2a: Evidence based on internal structure (local independence of items). No item residual correlations exceeded our set criterion. The highest residual correlation was found between items #HSE5 and #HSE6, with a coefficient of r = 0.29. Hence, we concluded that the 11 HSE items met the Rasch model assertion of local independence.

Step 2b: Is there satisfactory evidence of internal scale validity? (Item goodness-of-fit). In the first analysis, item #HSE11 demonstrated an infit MnSq value of 1,42 and was therefore considered to fit less well than the other items in the HSE scale. When item #HSE11 was removed from the analysis, the remaining 10 HSE items demonstrated infit MnSq values between 0.7–1.3. Item #HSE10 demonstrated an infit MnSq value of 1.25 when analyzing all 11 items, and 1.29 when item #HSE11 was excluded.

Step 2c: Evidence based on internal structure (unidimensionality). The explained variance was 52.2% which was above the set criterion of 50%. 7.6% of the unexplained variance was attributed to a single contrasting dimension, with an eigenvalue of 1.75. We therefore concluded that there was empirical evidence of unidimensionality in the HSE.

Step 3: Evidence based on response processes (person goodness-of-fit). 36 respondents in our sample (4.7%) were considered giving more variations in responses than expected according to the Rasch model, which was below the set criterion.

Step 4: Evidence based on precision/reliability (separation index). The person separation index of the 10 item HSE scale (excluding item #HSE11) was 2.19, supporting the assumption that the HSE questionnaire could differentiate between at least three different levels of the latent trait (PSC). 43 /651 scored a maximum on all HSE items, giving a ceiling effect of 5,7% of the respondents. One respondent provided minimum scores on all items resulting in a floor effect 0,1%. The person reliability score was 0.83 which met the criterion set to > 0.7.

Step 5: Evidence based on response processes (Differential Item functioning). No items demonstrating significant DIF in relation to Gender, Time of employment, Role within organization, or eNPS. Item #HSE7 did demonstrate DIF in relation to Age, where this item “I am always well received at my workplace when I need help” was relatively harder to agree with for age group 45–55, in comparison to both age group < 45 as well as age group 55 < .

Finally, the correlation coefficient between the HSE mean value index and the Rasch-generated unidimensional measures of the HSE 10-item scale was r = 0.95 (p < 0.01). See Fig. 2.

The present study aimed to evaluate the psychometric properties of the HSE questionnaire in a privately owned specialist care provider. The HSE was originally developed by SALAR in 2018 as a screening tool to assess healthcare staff perceptions of patient safety in acute care settings [21]. However, to our knowledge, no published evaluations of the HSE instrument in settings outside of acute care hospitals have been conducted. Therefore, this study contributes to filling this gap by examining the applicability of the HSE questionnaire in a different healthcare context.

Regarding the rating scale used in the survey, our results suggest the number of scale steps can be reduced from five to four steps without losing any information, as shown in Fig. 1. Specifically, we suggest removing the "I neither agree nor disagree" option (scale step three) from the questionnaire or collapsing it with one of the surrounding rating scale categories. The analysis revealed that this scale step overlaps with the others and does not contribute distinct information.

The results confirmed local independence of all HSE items. However, the study found that the inclusion of item HSE11, which assesses whether patients are offered the opportunity to be involved in patient safety work, was not viable due to high infit statistics [36].

This exclusion may be attributed to a lack of consensus among staff regarding this item, as involving patients in safety work may not be standard practice across organizations [37]. The different interpretations of this item across and between organizations may affect its relationship with the other items in the HSE questionnaire. However, it is important to note the significance of involving patients in safety work, as emphasized by the World Health Organization [1].

With HSE11 excluded the remaining ten items formed a common dimension accounting for more than 50% of the variability in the responses. It is worth mentioning that multiple dimensions are often present in surveys related to PSC. In this context, it is interesting to compare the HSE questionnaire with other widely used international patient safety questionnaires, such as the SAQ and SOPS®, which measure six and ten dimensions, respectively. However, previous studies have shown that further exploration of subdimensions may dilute the intended assessment of the common dimension [13, 18].

The study showed that HSE questions were well targeted to the respondents with less than 5% showing outlier response characteristics. Additionally, there were no flooring effects, and the observed ceiling effect was 5.7%, well below the 15% threshold suggested by Terwe et al. [38]. Further, the HSE questionnaire could separate respondents into different levels of patient safety climate, as indicated by a person-separation index exceeding 2.0 [34].

Interestingly, we found that respondents aged 45–55 encountered greater difficulty in agreeing with statement HSE7, "I am always well received at my workplace when I need help," compared to other age groups. This suggested that this item may pose challenges for this age group. Further investigation is warranted to determine whether this issue stems from the item itself or if it reflects a general difficulty among this age group in seeking help. It is worth considering the real-world findings of communication difficulties between older employees and younger managers, as shown by Kunze et al. [39, 40], as a potential factor influencing these responses.

The analysis revealed a strong linear relationship between the index scores derived from the raw scores of the HSE questionnaire's first ten items and the Rasch-generated unidimensional measure. This indicates that the index provides a reliable measure of PSC, particularly within the range of indices 20 to 70, where the linear relationship is closest. However, caution should be exercised when interpreting index scores outside this range, as they may lead to over- or underestimation of PSC.

These findings contribute to the understanding of the HSE questionnaire's validity and its potential usefulness as a screening and benchmarking tool for assessing PSC in healthcare settings. However, caution should be exercised when interpreting the scores in relation to other patient safety outcomes at a unit [41]. Low scores on the HSE should consider the presence of confounding factors such as poor work environment, job dissatisfaction, and high turnover rates [42]. Further studies are required to explore the identified three different levels of patient safety culture and their potential correlation with other measures of patient safety.

The use of a short questionnaire like the HSE offers several advantages over longer surveys. This study achieved an overall response rate of 66%, which exceeds the average response rate of 45% reported by Zha et al. [43], although it falls short of the 80% response rate often aimed for in federal US studies [44]. The relative brevity of the HSE questionnaire may contribute to higher response rates, as it is easier to complete and lowers the risk of survey fatigue [16, 17]. Additionally, the findings suggest that the HSE can be effectively applied in various healthcare settings beyond acute care hospitals, which is significant considering the predominance of studies on PSC conducted in hospital settings [18].

In conclusion, this study provides valuable insights into the psychometric properties of the HSE questionnaire in a privately owned specialist care provider. The first ten items of the HSE demonstrate good measurement properties, and an index based on these items can reliably assess PSC. However, further research is needed to explore the subdimensions and potential correlations with other measures of patient safety. The HSE questionnaire's brevity and its applicability in diverse healthcare settings make it a useful tool for assessing PSC and identifying areas for improvement.

A strength of this study is that it is set in a privately owned specialist care provider and that it thereby strengthens evidence on validity of the HSE questionnaire. The explored context can be expected to differ from the public emergency hospital environment where the questionnaire was piloted by SALAR [Unpublished data by SALAR], consequently this study adds to the transferability of the instrument. Another strength is that the study included multiple respondent background factors thereby exploring differential item functioning based on these factors. Further, the sample size of 761 respondents is a strength and will provide results within 0.5 logits for item calibrations and person ability measures [45]. A strength of applying the Rasch model in the study is in examining different aspects of validity evidence and that it can predict an individual's performance on a specific criterion or outcome. Another strength of the Rasch model is that it evaluates the internal structure/construct validity by examining whether the items in the questionnaire are measuring a single underlying construct.

The study was performed in a Swedish context with an instrument in Swedish. This may impact generalizability of the results in other contexts. However, the instrument was tested in an acute care hospital context during SALAR’s development process, while this study extends applicability by using a sample from a specialist outpatient care setting. This study does not focus on the content of the questions nor to what extent they are related to the participants’ view of PSC. The study does not rule out that there are dimensions of PSC that are not covered by the instrument. However, the questions were developed by SALAR’s team of patient safety experts and according to their unpublished documentation face and content validity tests showed satisfactory results.

The study's reliance on HR system data to determine eligibility for inclusion in the analysis resulted in a relatively large exclusion of units, which raises concerns about potential selection bias. However, it is worth noting that included as well as excluded units, all provide specialist care, which suggests there may be similarities between the two groups that reduce potential bias. Further research is needed to fully assess the potential impact of the exclusion criteria on the study's findings.

This study confirms that the ten first items of the SALAR HSE short questionnaire measures PSC in one common dimension. This study also confirms that the raw score of the first ten questionnaire items can indeed be used to calculate a patient safety index. In the study population it was possible to distinguish at least three different levels of PSC thereby also enabling benchmarking. In the original work by SALAR only nine out of eleven items were considered to fit the index. Our study contrasts that statement giving evidence that ten items fit the Rasch model and can be used to measure a common concept or dimension. The high response rate of 66% indicates that the questionnaire is accepted in a mixed specialist caregiver context. In conclusion, the HSE questionnaire shows sound psychometric results in its current state, and we recommend that the first ten items should be used to calculate a PSC raw score. Further studies can expand knowledge on the ability to assess the sustainability of the PSC over time.