https://orcid.org/0000-0003-3710-8931
Figures
Abstract
The aim of the study was to assess the reliability and construct validity of the Croatian translation of the Safety Attitudes Questionnaire—Ambulatory version (SAQ-AV) in the out-of-hours (OOH) primary care setting. A cross-sectional observational study using anonymous web-survey was carried out targeting a convenience sample of 358 health professionals working in the Croatian OOH primary care service. The final sample consisted of 185 questionnaires (response rate 51.7%). Psychometric properties were assessed using exploratory hierarchical factor analysis with Schmid-Leiman rotation to bifactor solution, McDonald’s ω, and Cronbach’s α. Five group factors were identified: Organization climate, Teamwork climate, Stress recognition, Ambulatory process of care, and Perceptions of workload. Items loading on the Stress recognition and Perceptions of workload factor had low loadings on the general factor. Cronbach’s α ranged between 0.79 and 0.93. All items had corrected item-total correlation above 0.5. McDonalds’ ω total for group factors ranged between 0.76 and 0.91. Values of ω general for factors Organization climate, Teamwork climate, and Ambulatory process of care ranged between 0.41 and 0.56. McDonalds’ ω general for Stress recognition and Perceptions of workload were 0.13 and 0.16, respectively. Even though SAQ-AV may not be a reliable tool for international comparisons, subsets of items may be reliable tools in several national settings, including Croatia. Results confirmed that Stress recognition is not a dimension of patient safety culture, while Ambulatory process of care might be. Future studies should investigate the relationship of patient safety culture to treatment outcome.
Citation: Mesarić J, Šimić D, Katić M, Tveter Deilkås EC, Hofoss D, Bondevik GT (2020) The safety attitudes questionnaire for out-of-hours service in primary healthcare—Psychometric properties of the Croatian version. PLoS ONE 15(11): e0242065. https://doi.org/10.1371/journal.pone.0242065
Editor: Sergio A. Useche, Universitat de Valencia, SPAIN
Received: February 5, 2020; Accepted: October 27, 2020; Published: November 13, 2020
Copyright: © 2020 Mesarić et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available in Harvard Dataverse repository https://dataverse.harvard.edu/ (DOI https://doi.org/10.7910/DVN/71JL1U).
Funding: The National Centre for Emergency Primary Health Care, Uni Research Health, Bergen, Norway, covered the expenses of data collection. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: EurOOHnet, European research network for out-of-hours primary health care; MSA, Measure of sampling adequacy; OOH, Out-of-hours (service); SAFE-EUR-OOH, Patient Safety Culture in European Out-of-hours services; SAQ, Safety attitudes questionnaire; SAQ-A, Safety attitudes questionnaire—ambulatory version (Modak [14]); SAQ-AV, Safety attitudes questionnaire—ambulatory version (Bondevik et al [11, 15])
Introduction
Patient safety is a major concern of health care services. The World Health Organisation defines patient safety as "the absence of preventable harm to a patient during the process of health care" [1]. Responsible health care organizations strive to avoid adverse events through improvement of the patient safety culture. The concept of “culture of safety” is multidimensional and comprises individual and group attitudes, beliefs, values, and competencies of health care professionals, staff interactions, clinical and administrative procedures, and practices aiming to protect patients from adverse events [2]. Lack of job satisfaction and work related stress might also increase the risk of human error in health care. Assessment of safety culture has been based on a variety of tools and instruments that have focused mainly on hospital care [3]. One of the most widely used instruments to measure safety culture of front line workers is the Safety Attitudes Questionnaire (SAQ) [2]. It has been validated in different languages and has shown good psychometric properties in different settings [4–13]. SAQ consists of 30 items arranged in six major patient safety subscales: Teamwork climate, Safety climate, Working condition, Job satisfaction, Perceptions of management, and Stress recognition [2].
The bulk of health care services is provided within the primary health care setting. However, concerns about patient safety in primary health care have been raised only recently. In 2007, Modak et al [14] extended SAQ with additional 32 items in order to adapt it to the primary care setting as Safety Attitude Questionnaire—Ambulatory Version (SAQ-A). In that study six SAQ subscales were confirmed for the original SAQ items. Additional subscale Ambulatory process of care was hypothesized, but not tested. In 2014, Bondevik et al [15] translated the SAQ-A into Norwegian and modified it by adding three other items from the SAQ-A to the Teamwork climate scale, and moving one item from the factor Perceptions of management to Working conditions. Bondevik et al [15] created two versions of the questionnaire—one for regular general practice, and one for out-of-hours (OOH) practice (used here as SAQ-AV) based on feedback from a group of primary health care providers. They used confirmatory factor analysis and confirmed five out of six SAQ subscales (all except Stress recognition) in the Norwegian setting. To distinguish it from Modak’s SAQ-A, we refer to it as SAQ-AV. SAQ-AV was subsequently translated into Dutch [16], and Slovenian [17], and applied to the out-of-hours primary health care services. In the Dutch and Slovenian settings these five subscales were not confirmed by confirmatory factor analysis. Authors of these studies performed exploratory factor analyses in order to explore dimensionality of SAQ-AV in their national settings [16, 17]
In Croatia, out-of-hours (OOH) primary health care services are provided by health care centres and emergency medical services. The 49 health care centres are primary health care organizations that provide general practice, dental, gynaecological, paediatric, and occupational healthcare, nursing and palliative care, laboratory and radiological diagnostics, and pharmacy services. Emergency medical services comprise first aid and emergency transport. OOH services in the health care centres are provided by general practitioners who usually rotate between their practice in normal office hours and OOH services. These services are available on weekends and holidays, but do not cover night shift. To our best knowledge patient safety culture in primary health care in Croatia has previously not been studied. A Croatian instrument for assessment of patient safety climate in primary care has not been developed until the Croatian translation of the SAQ-AV—the psychometric properties of which are reported here. The study was a part of the international study Patient Safety Culture in European Out-of-hours Services (SAFE-EUR-OOH) led by the Norwegian coordinating group of the European research network for out-of-hours primary health care (EurOOHnet) [18].
The aim of the study was to assess the reliability and construct validity of the Croatian version of the SAQ-AV.
Materials and methods
Ethics statement
The study was approved by the Ethical Committee of the School of Medicine, University of Zagreb (Case number 380-59-10106-15-168/120 of May 20, 2015). Data collection was compliant with Helsinki Declaration ethical guidelines. All participants received information about the purpose of the study, anonymity and confidentiality by e-mail. All participants gave an oral informed consent to participate in the study. Their participation was voluntary and data were analysed anonymously.
SAQ translation
We have translated the English version of SAQ-AV [11] into Croatian. The translation procedure followed the principles adapted from Beaton et al. [19]. The English version of the SAQ-AV was translated into Croatian using a professional translation service. The translation was discussed in an expert committee of researchers and general practitioners and adapted to the Croatian OOH primary care setting. This adapted version was back-translated into English by an independent translation service blinded to the original version. The back-translation was compared with the original questionnaire in order to clarify possible misunderstandings. The final version was piloted on ten OOH general practitioners and was found to require no further modifications.
Scoring
The questionnaire comprised 62 items [11] where study participants rated their agreement on a 5-point ordinal scale: 1 = disagree strongly, 2 = disagree slightly, 3 = neutral, 4 = agree slightly, 5 = agree strongly. All items also included the response option “Not applicable”–which was treated as missing data (non-response) in the data analyses. Scores of negatively worded items were reversed prior to analyses. Thus, higher scores indicate more positive evaluation of patient safety culture.
Study design
A cross-sectional observational study using web-survey was carried out between June 29th and September 1st, 2015. We targeted the 29 largest (out of the total of 49) health care centres in Croatia providing general practice OOH services. According to publicly available statistics (http://www.hzzo.hr) these centres employ approximately 85% of all staff providing general practice OOH services. Recommendations for minimal sample size in factor analysis range from 3 to 10 participants per item [20]. Bujang et al [21] support the ratio 1:3 when other criteria of sampling adequacy are met, and the response scale has at least four levels. For our questionnaire with 62 five level items recommendations for a suitable sample size range between 186 and 620. We contacted all 358 health professionals with an e-mail address working in these health centres, and 185 returned the questionnaire (response rate 51.7%). Respondents who returned the questionnaire could have selected response NA to some of the items. Unanswered items and NAs were treated as missing data in the analysis.
Statistical analysis
Statistical analysis included descriptive statistics (S1 Script), exploratory factor analysis (weighted least squares hierarchical factor analysis with oblimin rotation and Schmid-Leiman transformation), item-to-factor correlations, and intercorrelations of factors, measures of scale reliability (Cronbach’s α and McDonald’s ω hierarchical and total), and corrected item total correlation (CITC) (S3 Script) [22–26]. All analyses were done using the statistical software R version 3.2.2 (2015-08-14) [27], the RStudio (Version 0.99.486) [28] and the R package psych [29].
Item selection
We used Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (MSA) to ensure that our correlation matrix was appropriate for factor analysis [30, 31]. In the first step we have excluded from the analysis items with proportion of response NA exceeding 9% (8 items). According to Kaiser [30] item level MSAi “measures to what extent a given variable belongs to the family, psychometrically.” Thus, item with MSAi below 0.7 were also excluded [20, 30–32]. Following factor analysis items with communality below 0.35 were excluded. Finally, CITC was used to check for homogeneity of items within the scale [33]. All retained items had CITC above 0.3. The number of items included in the final analysis was 37. (S2 Script).
Exploratory factor analysis
If the Croatian version of SAQ-AV is a valid instrument for assessing overall patient safety attitudes in OOH primary care services, correlations between subscales could be attributed to a general factor representing this overall level of patient safety attitude. Therefore, we have used exploratory hierarchical factor analysis to identify the factor structure of the questionnaire and account for subscale intercorrelations. The goal of the factor analysis is to identify a smaller number of quantitative variables (factors) that explain a large proportion of the total variation of questionnaire items. In order to increase interpretability, obtained factors are rotated to achieve a simpler structure. Items loading on a single factor usually belong to the same subscale. When all items reflect a common underlying concept (like patient safety attitude) we may expect that a simple structure can only be achieved at the expense of factor independence (i.e. by non-orthogonal rotation). When rotated factors are correlated, a hierarchical (second order) factor analysis can be applied to these factors to obtain a common higher-order factor representing common variation in all items.
Exploratory hierarchical factor analysis (weighted least squares estimation) was used for exploring the dimensionality of the Croatian SAQ-AV. Factor analysis was done on the correlation matrix (Pearson r’s) with pairwise deletion of missing data. Correlation smoothing was used to achieve positive definiteness [34]. Horn’s [35] parallel analysis was used for selection of the number of factors. Oblimin rotation was used in order to obtain better separation of item loadings on factors and to estimate the inter-factor correlation matrix, which is the input for the second order factor analysis. We assumed that inter-factor correlations could be attributed to a general factor reflecting overall patient safety attitude. Therefore, the inter-factor correlation matrix was submitted to a second-order factor analysis extracting only one factor. Subsequently, Schmid-Leiman transformation of obtained factors was used to create an orthogonal solution with the general factor reflecting overall patient safety attitude, and group factors reflecting specific dimensions of patient safety culture and explaining the residual inter-item correlations after extraction of the general factor.
Reise et al. [26] compare three approaches to factor analysis of multidimensional constructs. The first one, most frequent in the literature, is correlated factors model, where factors are rotated using oblique rotation to achieve item separation and interpretability, allowing for correlation between subscales. However, as Reise et al. [26] highlight, this approach does not enable identification of a single common factor (the target construct) nor its direct association with items. The second one is hierarchical model, where correlations among factors are further submitted to factor analysis to identify a common second order factor. In this approach the single common factor (the target construct) is identified, however the relationship between an item and the target construct is “mediated through the primary factor” The approach we use is the third one, the bifactor model, where each item loads on the general factor and one or more group factors. According to Reise [36] in comparison to the first two approaches, the bifactor modelling enables “(a) partitioning item response variance into general versus group factor sources, (b) determining the degree to which item response data are unidimensional versus multidimensional, (c) estimating the degree to which raw scale scores reflect a single common source, and (d) evaluating the viability of subscale scores after variance due to the general factor has been controlled for.” We report results of Schmid-Leiman transformation without further selection of items to substantiate our conclusions regarding dimensionality of the concept of patient safety attitudes, and the viability of subscales.
In addition to factor loadings we report item communalities, (the proportion of item variance explained by the factor model), uniqueness (the proportion of item variance not accounted for), and the proportion of communality accounted for by the general factor [25, 26].
Scale reliability
We use two conceptualizations of internal reliability. The first one is based on classical measurement theory where reliability is defined as a ratio of true score variance to total variance. The most popular representative of this conceptualization is Cronbach’s α that is based on average inter item covariance and represents “internal consistency” reliability [23]. The other conceptualization is based on the common factor model, where reliability is defined as a proportion of total score variance accounted for by common factors. We have used McDonald’s ω to estimate reliability of general factor, group factors and total reliability [23]. While Cronbach’s α assumes unidimensional concept, McDonald’s ω can be used as an estimate of general scale saturation for a multidimensional construct [22, 23, 37]. When an instrument is composed of two or more subscales, we would like to know whether it measures a unidimensional or a multidimensional construct. McDonald’s ω total is calculated from results of a hierarchical factor analysis as a ratio between variance accounted for by extracted general and group factors and the total variance. ω general is a ratio of the variance accounted for by the general factor and the total variance. ω group is a ratio of the variance accounted for by all group factors and the total variance. However, in calculating ω group variance accounted for by a group factor is taken only over items that load on that factor; thus, for the general factor, ω general and ω group do not have to add up to ω total. For each group factor ω total, ω general and ω group are calculated using only items that load on that factor. For a group factor ω group and ω general add up to ω total. The ratio of the ω general in relation to ω total provides a measure of general factor saturation. Higher values of ω general support unidimensionality, while lower values support multidimensionality of the measured construct. We also use CITC to identify any items that may lower reliability of a subscale.
Results
The 185 respondents who completed the questionnaire came from 28 out of 49 health care centres in Croatia. Respondents included 150 (81.1%) medical doctors and 35 (18.9%) support medical staff (including registered nurses and administrative staff). The majority of respondents were female (84.3%). Thirty-seven (20%) of the respondents were under 30 years old, and six (3.2%) above 60 years. Respondents’ job experience in OOH service was below five years for 47 (25.4%) of the respondents. Job experience was uniformly distributed up to 40 years. Two respondents (1.1%) had more than 40 years of job experience. Only 86 (46.5%) respondents provided data on length of work in the OOH service where they were currently employed. Among those who responded, 32 (17.3%) had worked up to 10 years, 35 (18.9%) between 11 and 20 years, and 19 (10.3%) longer than 20 years in the current OOH service. The younger staff constituted 25% of our respondents (with less than 5 years of job experience). Most respondents (160, 86.5%) worked in shifts, while 18 (9.7%) of the respondents worked morning shift only (e.g. on Saturday and Sunday) and 7 (3.8%) worked afternoon shifts only.
Table 1 provides basic descriptive statistics for the scale items. For 15 out of the 62 items there were no missing data. The number of missing responses for other items ranged from 1 to 49. Items with the highest numbers of missing responses were SAQ42 (“Trainees in my discipline are adequately supervised”– 49 missing responses), SAQ50 (“Important issues are well communicated at shift changes”: 37 missing responses), and SAQ39 (“I am frequently unable to express disagreement with staff physicians/intensivists in this office”: 36 missing responses).
Cronbach’s α for the entire set of 62 items was 0.95 (CI95 = 0.94–0.96). Corrected item-total correlations (CITC) ranged from 0.11 to 0.72 with the lowest three for items SAQ36 (“Disruptions in the continuity of care can be detrimental to patient safety” CITC = 0.11), SAQ27 (“I have seen others make errors that had the potential to harm patients” CITC = 0.16), and SAQ1 (“High levels of workload are common in this office” CITC = 0.17). Five items had CITCs between 0.2 and 0.3, all other items had CITCs above 0.3.
The overall Kaiser-Meier-Olkin measure of sampling adequacy (MSA) was 0.82, and MSA per item was above 0.5 for all items. Out of 62 items 6 had MSA values between 0.50 and 0.59, 8 had MSA between 0.60 and 0.69, 12 had MSA values between 0.70 and 0.79 and 36 had MSA of 0.8 and above.
The items with the lowest MSA values were SAQ1 (MSA = 0.53), SAQ36 (MSA = 0.55), and SAQ50 (MSA = 0.56).
Exploratory hierarchical factor analysis was performed on the Pearson’s correlation matrix between 37 retained items, using pairwise deletion for items that included missing responses. Table 2 and Fig 1 present the structure of the final model. The five extracted factors accounted for 51.4% of the total variance. The proportion of explained common variance of the general factor was 42%, and the ratio between the eigenvalue corresponding to the general factor and the largest eigenvalue of group factors was 2.67. Items loading on the third and the fifth factor had low loadings on the general factor indicating that these factors were possibly not related to the patient safety culture attitude. The proportion of communality accounted for by the general factor for items loading on the third and the fifth factor was below 23%, while for items loading on other factors it ranged between 35% and 66%, with a median proportion of 53%. Three items had loading above 0.3 on the general factor, and the highest group factor loading below 0.3. Since their communalities were above 0.3, they were retained in the analysis.
Value of ω total for the general factor was 0.96, ω general was 0.66, and ω group was only 0.18. Values of ω total for group factors ranged between 0.76 and 0.91. Values of ω general for group factors Organization climate, Teamwork climate, and Ambulatory process of care ranged between 0.41 and 0.56, and for factors Stress recognition and Perceptions of workload they were 0.13 and 0.16, respectively. Values of ω group for factors Organization climate, Teamwork climate, and Ambulatory process ranged between 0.35 and 0.36, for factors Stress recognition and Perceptions of workload they were 0.70 and 0.61, respectively.
Table 3 shows the distribution of items in respect to the original SAQ-A subscales [14] and our group factors. Factors 2 to 4 can be easily identified with subscales Teamwork climate (F2*), Stress recognition (F3*), and Ambulatory process of care (F4*). The first factor (F1*) combines items from Safety climate, Job satisfaction, Perceptions of management, and Working condition subscales. We have named this factor Organization climate [38]. Factor 5 comprises items 47, 49, 52, and 53, added to the scale by Bondevik et al [15] (see Table 1 for item content). We have named this factor Perceptions of workload, as these items relate to “fatigue”, being “burned out”, and “work(ing) too hard”. All items correlated more strongly with their own factor than with other factors.
Table 4 presents values of Cronbach’s α and corrected item total correlation for group factors. All five subscales had Cronbach’s α’s greater than 0.79. Corrected item total correlations ranged between 0.53 and 0.81 for the first factor, between 0.56 and 0.69 for the second factor, between 0.62 and 0.77 for the third factor, between 0.56 and 0.88 for the fourth factor, and between 0.52 and 0.74 for the fifth factor. Croatian translation of items is provided in (S1 Text).
Discussion
Exploratory bifactor analysis of the Croatian version of the SAQ-AV revealed five factors. These factors partially corresponded to previously identified SAQ factors in other countries [2, 14–17]. Separation of items is very good, i.e. items do not have high loadings on more than one factor, confirming discriminant validity of identified subscales. Based on values of ω general, items loading on factors Organization climate, Teamwork climate, and Ambulatory process reflect a common construct, while items loading on factors Stress recognition and Perceptions of workload do not. Relatively lower values of ω group for the first three factors do not support conclusion that these factors reflect viable subscales after accounting for item variability due to the general factor.
The first extracted factor cannot be clearly mapped to any single SAQ-A subscale [11]. It comprises items belonging to the subscales Safety climate (4 items), Job satisfaction (3 items), Perceptions of management (3 items), Working condition (2 items), and Teamwork climate (1 item). We believe that this can be attributed to characteristics of the OOH services in Croatia, where these items reflect aspects of organization climate, which combines these four original dimensions [38]. This factor is highly reliable with Cronbach α of 0.93 and ω total of 0.91. It’s ω general is 0.56, reflecting high common variation with the general factor.
The second factor corresponds to the SAQ-A Teamwork climate subscale. It has reasonably high values of Cronbach’s α (0.84) ω total (0.77), and ω general (0.41), demonstrating its reliability.
The third factor is clearly defined by Stress recognition items. In comparison to SAQ-A subscales, this factor has the most consistent structure. It has relatively high values of Cronbach’s α (0.86) and ω total (0.84). However, it only marginally contributes to the general factor, with ω general equal only 0.13. According to Saxton et al [2] development of the SAQ questionnaire relied on “Vincent’s framework for analysing risk and safety, and Donabedian’s conceptual model for assessing quality”. Pool of items was generated through brainstorming with healthcare providers, and resulted in a six factor model including stress recognition as one of dimensions. However, subsequent studies showed low test-retest reliability, low correlation between stress recognition and other dimensions, and lack of variability in stress recognition where other dimensions showed variation among sites [4, 6, 9, 15, 17, 39, 40] Thus, our results confirm that Stress recognition is not a sub-dimension of patient safety culture attitude. We agree with Taylor and Pandian [41] that Stress recognition is an important standalone construct, however it does not reflect the safety climate.
Modak et al. [14] added “5 items pertaining to the process of ambulatory care based on literature review of outpatient medical errors, quality improvement, and patient safety” to the original SAQ. Their confirmatory factor analysis aimed only to confirm the six factor structure of the original SAQ; thus, “Ambulatory Process of Care Items” were not included in the factor analysis. Our factor four undoubtedly represents SAQ-A Ambulatory Process of Care [14], with four out of its five items belonging to this group of items. This factor exhibits reasonable reliability with Cronbach’s α of 0.82, and ω total of 0.77. Its ω general is 0.41, confirming further that this factor contributes to the general safety attitude scale.
Factor five comprises four items that relate to fatigue, frustration, burn-out, and perception of working too hard. These items differ from items on the Stress recognition subscale in that they relate to perception of personal workload, and not recognition of relationship between stress and performance. We named this factor Perceptions of workload. Even though this factor shows reasonable reliability (Cronbach’s α 0.79, ω total 0.76), it only marginally contributes to the general factor (ω general 0.16), which suggests that it may not be a reliable indicator of safety culture in Croatia. This may be due to a prevalent lack of staff in Croatian health care, reflected also in the fact that 55.7% of respondents agreed slightly or strongly with the item SAQ53 (“working too hard”).
Survey and item level non-response could have an impact on the results of an exploratory factor analysis. We had relatively low response rate of 51.7%. This could influence results if non-response was associated with patient safety climate. In such a case, we would expect to see skewed item distributions, due to lack of extreme responses in either positive or negative direction. However, none of the items in our analysis had skewness larger than 2 in absolute value, thus we believe that self-selection did not have undue influence on our results [42].
Regarding the item-level non-response, we decided to exclude from the analysis eight items with item non-response above 9%. These items can be classified in four groups. The first group (items SAQ6 and SAQ42) relates to training of new personnel and trainees. The second group were items regarding briefings and shift hand-overs (SAQ13, SAQ14, and SAQ50). The third group was related to incidents and incident reporting (SAQ54, SAQ62), and the last item was SAQ39 (inability to express disagreement). Non-response for items in groups one to three can be attributed to the organization of Croatian GP OOH (small number of trainees / new staff—thus rare organization of trainings, one-shift daily—thus no briefings and hand-overs, and underdeveloped incident reporting system). Regarding non-response for item SAQ39, while we have targeted the largest GP OOH, in some of them OOH team may only comprise one physician / intensivist, and for them there would be nobody to express disagreement with. We believe that these items do not reflect well safety climate attitudes in the Croatian setting, and that is why we decided to exclude the items, instead of imputing the missing data. Still, it would be interesting to analyse association between the non-response to these items and other responses, however that analysis is out of the scope of this paper.
It is interesting to compare our results with similar studies in other countries. In comparison to SAQ-A [14], there is a good correspondence between two of our factors with respective SAQ-A subscales (Stress recognition, and Teamwork climate). The original SAQ-A subdimensions Safety climate, Perceptions of management, Working condition, and Job satisfaction combined in a single factor we named Organization Climate.
Organization of the Croatian health care system is more similar to the Slovenian health care system than to other countries in which research on psychometric qualities of SAQ in primary care has been published so far. Due to similarities in organization of OOH services we would expect to find similarities with factor structure in Slovenia [17], however we did not.
Overall, only the Teamwork climate factor showed consistent presence in this and the three studies of SAQ in OOH primary care services in other European countries (Norway, Netherlands, and Slovenia) [15–17]. We have identified the Ambulatory process of Care factor as proposed in the original SAQ-A study [14] (USA). Stress recognition and Perceptions of workload were identified as reliable scales. They had Cronbach’s α equal 0.86 and 0.79, respectively, and their items did not cross-load on other (sub)scales. However, these scales are not subscales of the perception of safety culture, because their respective ω general were only 0.13 and 0.16. Perceptions of workload is a factor recognized in this study, that was not previously identified.
Conclusions
Based on relatively high reliability of our factors, we believe that it would be possible to use the items loading on our factors Organization climate, Teamwork climate, and Ambulatory process as a reliable tool for measuring patient safety culture in Croatian OOH primary healthcare. Our results do not support use of raw subscale scores. A larger study is needed to confirm this. However, this tool may not be appropriate for international comparison. We believe that due to national peculiarities of the organization of health care the factor structure of patient safety culture measurements also varies across countries. For international comparisons item level comparison may be the best option. Our analysis did not find that Stress Recognition and Perception of workload were a sub-dimension of safety attitude in the Croatian primary care OOH service.
Future research is needed to understand better how differences in national health systems affect the structure of patient safety culture and elucidate the relationship between patient safety culture and treatment outcomes.
Acknowledgments
We thank health care providers from GP practices in 28 health centres for participation in the study.
References
- 1.
World Health Organisation. Patient Safety Research: Better Knowledge for Safer Care. Geneva: WHO; 2009.
- 2. Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6: 44–54. pmid:16584553
- 3. Brasaite I, Kaunonen M, Suominen T. Healthcare professionals’ knowledge, attitudes and skills regarding patient safety: a systematic literature review. Scand J Caring Sci. 2015;29: 30–50. pmid:24708205
- 4. Deilkås ET, Hofoss D. Psychometric properties of the Norwegian version of the Safety Attitudes Questionnaire (SAQ), Generic version (Short Form 2006). BMC Health Serv Res. 2008;8: 191–200. pmid:18808693
- 5. Kaya S, Barsbay S, Karabulut E. The Turkish version of the safety attitudes questionnaire: psychometric properties and baseline data. Qual Saf Health Care. 2010;19: 572–577. pmid:20671082
- 6. Lee W-C, Wung H-Y, Liao H-H, Lo C-M, Chang F-L, Wang P-C, et al. Hospital safety culture in Taiwan: a nationwide survey using Chinese version Safety Attitude Questionnaire. BMC Health Serv Res. 2010;10: 234–234. pmid:20698965
- 7. Raftopoulos V, Savva N, Papadopoulou M. Safety culture in the maternity units: a census survey using the Safety Attitudes Questionnaire. BMC Health Serv Res. 2011;11: 238–238. pmid:21951720
- 8. de Carvalho REFL, Cassiani SHDB. Cross-cultural adaptation of the Safety Attitudes Questionnaire—Short Form 2006 for Brazil. Rev Lat Am Enfermagem. 2012;20: 575–582. pmid:22991121
- 9. Göras C, Wallentin FY, Nilsson U, Ehrenberg A. Swedish translation and psychometric testing of the safety attitudes questionnaire (operating room version). BMC Health Serv Res. 2013;13: 104–104. pmid:23506044
- 10. Zimmermann N, Küng K, Sereika SM, Engberg S, Sexton B, Schwendimann R. Assessing the Safety Attitudes Questionnaire (SAQ), German language version in Swiss university hospitals—a validation study. BMC Health Serv Res. 2013;13: 347–347. pmid:24016183
- 11. Bondevik GT, Hofoss D, Hansen EH, Deilkås ECT. Patient safety culture in Norwegian primary care: a study in out-of-hours casualty clinics and GP practices. Scand J Prim Health Care. 2014;32: 132–138. pmid:25263763
- 12. Gabrani A, Hoxha A, Simaku A, Gabrani JC. Application of the Safety Attitudes Questionnaire (SAQ) in Albanian hospitals: a cross-sectional study. BMJ Open. 2015;5: e006528–e006528. pmid:25877270
- 13. Nguyen G, Gambashidze N, Ilyas SA, Pascu D. Validation of the safety attitudes questionnaire (short form 2006) in Italian in hospitals in the northeast of Italy. BMC Health Serv Res. 2015;15: 284–284. pmid:26204957
- 14. Modak I, Sexton JB, Lux TR, Helmreich RL, Thomas EJ. Measuring safety culture in the ambulatory setting: the safety attitudes questionnaire—ambulatory version. J Gen Intern Med. 2007;22: 1–5. pmid:17351834
- 15. Bondevik GT, Hofoss D, Hansen EH, Deilkås ECT. The safety attitudes questionnaire—ambulatory version: psychometric properties of the Norwegian translated version for the primary care setting. BMC Health Serv Res. 2014;14: 139–139. pmid:24678764
- 16. Smits M, Keizer E, Giesen P, Deilkås ECT, Hofoss D, Bondevik GT. The psychometric properties of the ‘safety attitudes questionnaire’ in out-of-hours primary care services in the Netherlands. PLoS ONE. 2017;12: 1–12. pmid:28207909
- 17. Klemenc-Ketis Z, Maletic M, Stropnik V, Deilkås ET, Hofoss D, Bondevik GT. The safety attitudes questionnaire—ambulatory version: psychometric properties of the Slovenian version for the out-of-hours primary care setting. BMC Health Serv Res. 2017;17: 36–36. pmid:28086919
- 18. Huibers L, Phillips H, Giesen P, Remmen R, Christensen M, Bondevik GT. EurOOHnet-the European research network for out-of-hours primary health care. Eur J Gen Pract. 2014;20: 229–232. pmid:24219341
- 19. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25: 3186–3191. pmid:11124735
- 20. Chan LL, Idris N. Validity and Reliability of The Instrument Using Exploratory Factor Analysis and Cronbach’s alpha. Int J Acad Res Bus Soc Sci. 2017;7.
- 21.
Bujang MA, Ghani PA, Soelar SA, Zulkifli NA. Sample size guideline for exploratory factor analysis when using small sample: Taking into considerations of different measurement scales. 2012 International Conference on Statistics in Science, Business and Engineering (ICSSBE). Langkawi: IEEE; 2012. pp. 1–5. 10.1109/ICSSBE.2012.6396605
- 22. Revelle W, Zinbarg R. Coefficients alpha, beta, omega and the glb: comments on Sijtsma. Psychometrika. 2009;74: 145–154.
- 23. Zinbarg R, Revelle W, Yovel I, Li W. Cronbach’s Alpha, Revelle’s Beta, McDonald’s Omega: Their relations with each and two alternative conceptualizations of reliability. Psychometrika. 2005;70: 123–133.
- 24. Zinbarg R, Yovel I, Revelle W. Estimating omega for structures containing two group factors: Perils and prospects. Appl Psychol Meas. 2007;31: 135–157.
- 25. Reise SP. The Rediscovery of Bifactor Measurement Models. Multivar Behav Res. 2012;47: 667–696. pmid:24049214
- 26. Reise SP, Moore TM, Haviland MG. Bifactor Models and Rotations: Exploring the Extent to Which Multidimensional Data Yield Univocal Scale Scores. J Pers Assess. 2010;92: 544–559. pmid:20954056
- 27.
R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; 2015. https://www.R-project.org
- 28.
RStudio Team. RStudio: Integrated Development for R. Boston, MA: RStudio, Inc.; 2015. http://www.rstudio.com/
- 29.
Revelle W. psych: Procedures for Psychological, Psychometric, and Personality Research. Evanstone, ILL, USA: Northwestern University; 2015. http://CRAN.R-project.org/package=psych
- 30. Kaiser HF. A second generation little jiffy. Psychometrika. 1970;35: 401–415.
- 31.
Mulaik SA. Foundations of factor analysis. 2nd ed. Boca Raton: CRC Press; 2010.
- 32. Kaiser HF, Rice J. Little Jiffy, Mark IV. Educ Psychol Meas. 1974;34: 111–117.
- 33. Zijlmans EAO, Tijmstra J, van der Ark LA, Sijtsma K. Item-Score Reliability as a Selection Tool in Test Construction. Front Psychol. 2019;9. pmid:30687144
- 34. Bock R, Gibbons R, Muraki E. Full-Information Item Factor Analysis. Appl Psychol Meas. 1988;12: 261–280.
- 35. Horn J. A rationale and test for the number of factors in factor analysis. Psychometrika. 1965;30: 179–185. pmid:14306381
- 36. Reise SP. The Rediscovery of Bifactor Measurement Models. Multivar Behav Res. 2012;47: 667–696. pmid:24049214
- 37. Zinbarg R, Yovel I, Revelle W, McDonald R. Estimating generalizability to a universe of indicators that all have one attribute in common: A comparison of estimators for omega. Appl Psychol Meas. 2006;30: 121–144.
- 38.
Stone PW, Harrison MI, Feldman P, Linzer M, Peng T, Roblin D, et al. Organizational Climate of Staff Working Conditions and Safety—An Integrative Model. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville (MD): Agency for Healthcare Research and Quality; 2005. pp. 467–481. https://www.ncbi.nlm.nih.gov/books/NBK20497/
- 39. Chaboyer W, Chamberlain D, Hewson-Conroy K, Grealy B, Elderkin T, Brittin M, et al. CNE Article: Safety Culture In Australian Intensive Care Units: Establishing A Baseline For Quality Improvement. Am J Crit Care. 2013;22: 93–102. pmid:23455858
- 40. Haerkens MH, van Leeuwen W, Sexton JB, Pickkers P, van der Hoeven JG. Validation of the Dutch language version of the Safety Attitudes Questionnaire (SAQ-NL). BMC Health Serv Res. 16: 385 (8 pp). pmid:27528393
- 41. Taylor JA, Pandian R. A dissonant scale: stress recognition in the SAQ. BMC Res Notes. 2013;6: 302. pmid:23902850
- 42. Zygmont C, Smith MR. Robust factor analysis in the presence of normality violations, missing data, and outliers: Empirical questions and possible solutions. Quant Methods Psychol. 2014;10: 40–55.