Burnout among midwives—the factorial structure of the burnout assessment tool and an assessment of burnout levels in a Swedish national sample

In 2020 a web-survey was distributed to all Swedish midwives who were members of either of two unions: the Swedish Association of Midwives or the Swedish Association of Health Professionals. These two unions organize the majority of the midwives in Sweden. An invitation to participate was sent out to 5,076 midwives and 2,060 choose to participate, generating a participation rate of 41%. Of the participating midwives 99.6% were women with a mean age of 48 years. The gender and age distribution of the participants was in line with public national statistics on midwives. The workplaces of midwives in this study included labour ward (44%), maternity care (32%), postnatal care (29%), gynaecology (11%) and youth clinic (8%).

The Swedish Ethical Review Authority approved the study in 2019 (Dnr 2019–03,776) and it was performed in accordance with the Declaration of Helsinki from 1964 and its later amendments regarding research involving human subjects. All participants were assured of confidentiality. The participants gave informed consent before taking part in the study.

A professional data collection company generated individual links to the web survey. These links were distributed by the two unions to all midwives with registered e-mail addresses. Three reminders were sent out. It was made clear in the information to the participants that the unions were not involved in any way in the study, other than distributing the invitation and reminders with individual links to the web-survey. Collection of data began on 04/02/2020 and ended on 20/04/2020. Note that the data collection was done before the full effect of the Covid-19 pandemic hit Sweden. A database was created by the data collection company and handed over to the researchers for analysis.

The Burnout Assessment Tool (BAT) is a self-report questionnaire consisting of 23 items (see Supplementary Information – (Supplementary file 1) for the Swedish version used in this study) grouped in four subscales: exhaustion (EX, 8 items), mental distance (MD, 5 items), cognitive impairment (CI, 5 items), and emotional impairment (EI, 5 items). All items are expressed as statements with five frequency-based response categories (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always). The total burnout score, as well as scores for each subscale is calculated as a mean value of all responses, where a high score is indicative of high level of burnout (range 1–5). In the present study Swedish version of the BAT was used. The translation of the BAT to Swedish was done using both the original (Dutch version) and the English version of the instrument by means of dual-panel translation including the bilingual panel and the lay panel. For the English-Swedish translation, the panel consisted of a professional translator, trained clinicians with a long experience of treating patients with stress-related problems as well as researchers in the field, all fluent in both Swedish and English. For the Dutch-Swedish translation the panel included a trained physician who is fluent on all three languages. In the first step all involved worked with their own translations. In the second step the panel members worked together in a series of meetings and had opportunity to discuss alternatives until they agreed on the most appropriate and conceptually equivalent translation. In the final step, a lay panel consisting of a small group of employees in Region Västra Götaland, tested a Swedish version of the BAT and had the opportunity to give their comments. The translation process took place at the Institute of Stress Medicine in Gothenburg, Sweden.

The goal of the Rasch analysis is to evaluate whether the observed data satisfy the requirements of the Rasch model, in which case the measurement shows construct validity, and the total score is a sufficient statistic for the latent construct that is being measured by the questionnaire. Important requirements in Rasch analysis are unidimensionality, monotonicity, invariance, absence of differential item functioning (DIF) and local independency, which are briefly explained here.

Unidimensionality is a basic assumption for combining a set of items into a single burnout score, i.e. all items should represent a common latent trait. Monotonicity implies that the item responses are positively related to the latent trait. The response structure required by the Rasch model is a stochastically consistent item order; i.e. a probabilistic Guttman pattern [35]. This implies that people experiencing higher levels of burnout are expected to get higher scores on the BAT and people with lower levels of burnout are expected to score lower on the BAT. Increasing levels of severity of burnout across response categories for each item need to be reflected in the data.

The assumption of invariance implies that the items need to work invariantly across the whole burnout continuum for all individuals, i.e. the ratio between the location values (positioning of items on the latent burnout logit scale) of any two items must be constant along the latent construct. Invariance also implies that the items need to work in the same way (invariantly) for all comparable groups, which is known as absence of DIF. Absence of DIF implies that given the same level of burnout, the scale should function in a similar way for comparable groups of respondents (e.g. women and men, younger and older).

Finally, local dependency implies that, having extracted the unidimensional latent trait of burnout, there should be no other meaningful patterns in the residuals. Assumption of local dependency may be violated by response dependency and/or multidimensionality [36]. Understandably, multidimensionality occurs when items are measuring more than one latent dimension. Response dependency occurs when items are linked in some way so that the response to one item depends on the response to another item. A pedagogical example from rheumatology is when several items assessing walking ability are included in the same questionnaire. If a person is able to walk several miles without difficulty, then that person is also able to walk 1 mile or less without difficulty [32]. Another form of response dependency, known as redundancy dependency, may be caused by the degree of overlap of the content of two items, so that a particular rating for one item logically implies the similar rating for another item, e.g. two items reflecting reversed statements such as “I feel tired” and “I feel alert” [37]. Local dependency affects the fit to the model. Response dependency can result in increased similarity of the responses of people across items, so that responses are more Guttman-like than they should be under no dependency. Contrarily, multidimensionality results would result in responses being less Guttman-like than they should be under no dependency [38].

Rasch analysis incorporated a cross-validation strategy. This means that the data was split into two random samples and analysed twice, to check the robustness of the results. This is also a convenient way to adjust for possible problems associated with χ² statistics in large sample sizes, where even minor levels of misfit become statistically significant. Rasch analysis was done in RUMM2030 [39]. Two random samples each consisting of 800 respondents were drawn using a function for random sampling in RUMM2030 and all analyses were done using these random samples The items’ fit was analysed using the partial credit model for polytomous cases [40]. To control for the large number of comparisons, the significance level was set at 0.01 and Bonferroni adjusted. The differences between the model-expected values and the observed values (i.e. residuals), are scrutinized in several ways in order to evaluate whether the data fit to the Rasch model.

In the first step, the first Rasch analysis included responses to all 23 items, to evaluate whether the items within each subscale would cluster together in a residual correlation matrix in a pattern that is consistent with the underlying conceptualization of the BAT. When instruments consist of a bundle of items measuring different aspects of the latent trait, it is expected that the correlation matrix of residuals reveals the clustering of the items within each subscale [37]. Any residual correlation between the items 0.2 above the average observed correlation is indicative of local dependency [37]. In the same analysis, the functioning of each item is evaluated in terms of (a) discriminant ability (item fit residual within range of ± 2.5); (b) threshold ordering (i.e., appropriateness of the response categories, evaluated graphically and by thresholds estimates for each item); (c) the non-significant item χ² statistic; (d) local dependency (residual correlation matrix), and (e) absence of DIF for age (above/under the median age of 47). DIF was tested by conducting ANOVA of standardized residuals, which enables separate estimations of misfit along the latent trait, uniform and non-uniform DIF. Disordering of the thresholds was tested by the hybrid approach proposed by Salzberger [41].

The overall fit to the model was evaluated by the mean and standard deviations of person and item residuals (expected values are 0 and 1 respectively), the model χ2 statistic (expected to be non-significant and indicates invariance). The internal consistency of the scale and the power of the BAT scale to discriminate among respondents with different levels of burnout were evaluated with the Person Separation Index (PSI). The PSI ranges from 0 to 1 and is similar to Cronbach’s alpha.

Smith’s test of unidimensionality was used to investigate the assumption of unidimensionality [42]. For this test, first a principal component analysis (PCA) on residuals was performed. Items loading positively and negatively on the first principal component were used to obtain an independent person estimate. Finally, independent t-tests for differences in these estimates for each person were performed [42]. Less than 5% of such tests being outside the range of ± 1.96 support the unidimensionality of the scale. A 95% binomial confidence interval of proportions [43] was used to show whether the lower limit of the observed proportion is below the 5% level.

In the second step, (after the local dependency was confirmed in the first step) we followed the method of combining correlated items into testlets, as recommended by Marais et. al. and Andrich [36, 38, 44]. This method combines correlated items into one or more testlets (preferably based on theoretical considerations) and the data were re-analysed using testlets instead of individual items. Thus, the second model was run with the four testlets based on the BAT’s four subscales.

In the case of testlets, the total variance in the data can be divided into three components: a) the total common true score variance (across all items), b) the total unique variance (the sum of the unique variances among the subscales) and c) the total error variance of the scale. The testlets’ model fit was compared with the fit obtained from the initial analysis of the individual items [44]. The coefficient alpha from the initial analysis corresponds to the proportion of the non-error variance (the sum of a) and b) above) relative to the total variance and is compared to the coefficient alpha from the testlet analysis, which corresponds to the proportion of the common true score variance relative to the total variance. The proportion of the common true score variance to the non-error variance is also calculated as well as the average latent correlation among the subscales corrected for attenuation for error.

Finally, targeting (distribution on a logit scale) of the BAT items and persons in the sample was evaluated graphically in a person-item-threshold graph. Targeting evaluates how well the items are targeted for severity levels of burnout as reported by the respondents. For a well-targeted instrument, the mean location for persons would be around the value of zero. The person parameter estimates are compared with the distribution of the item thresholds. In that way, thresholds which are extreme compared to persons can be identified, as they provide little information in the population. This is important for the precision of person parameter estimates. In other words, responses to such items will have little impact on the precision of the person estimates as these items are out of target [45].

Burnout levels among midwives are presented in terms of median and first and third quartiles (Q1 and Q3 respectively) for the BAT total score (BAT) and well as for the four subscales (EX, MD, CI and EI). These measures were calculated in the total sample. The number of participants who completed all items (regardless of gender and age) was as follows: BAT n = 1,693, EX n = 1,711, MD n = 1,702, CI = 1,714 and EI n = 1,714. Differences in burnout levels between age groups were tested with the Mann–Whitney U test. The results are visualized with boxplots.

The first analysis on the random sample 1, included all 23 items and the overall fit statistics are shown in Table 1. As expected, given the subscale structure of the BAT, fit to the model was not obtained. Item and person fit residuals mean and SD deviated from the expected value of 0 and 1 respectively and the value of χ2 statistic was high and significant (454.96, p < 0.0001). The value of PSI was high (0.94) but the Smith´s test indicated problems with dimensionality as the percentage of significant tests was 20.7 and the lower confidence bound was > 5%.

Item fit statistics (residuals and thresholds) are shown in Table 2. As seen in Table 2, item fit residuals outside the predefined range of ± 2.5 were observed for several items.

Evaluation of thresholds ordering showed that all items besides item CI5 had ordered thresholds. Item CI5 (I make mistakes in my work because I have my mind on other things) had a problem with thresholds three and four being reversed (Table 2) which correspond to the two highest response categories (often and always). Further analysis of the item CI5 was done using the Salzberger hybrid approach to evaluate true ordering (H₁) or true disordering (H₂) of thresholds three and four, with undecided results since the null hypotheses (no difference between the thresholds) could not be rejected (data not shown). Frequency distributions for all items show that the two highest response categories were used very rarely by the respondents in this study, see Supplementary file 2. Consequently, thresholds’ standard errors for these categories were much higher compared to the first two categories, especially for the item CI5 as well as for the EI items (Table 2). DIF for age was evaluated and no problems were observed for any of the items. A table showing the results from DIF analysis are given in Supplementary file 3.

The residual correlation matrix shown in Supplementary file 4 indicated problems with local dependency as correlations higher than expected under the condition of local independence (in this sample a value > 0.16) were found for many pairs of items. The pattern of the residual correlations corresponded to the underlying conceptual structure of the BAT with the four subscales, with high residual correlations found between pairs of items within the same subscales. The only exception was one pair of items between different subscales with an observed residual correlation of 0.19, namely items EX5 (When I get up in the morning, I lack the energy to start a new day at work) and MD1 (I struggle to find any enthusiasm for my work).

As a part of cross-validation strategy, the same analysis with all 23 items was repeated in random sample 2 and the results were similar. The overall fit statistics showed the same pattern as in the previous analysis (Table 1, Sample 2 BAT 23 items). For the item fit statistics (results not shown), significant item fit residuals outside the predefined range were found for the same two items, EX2 and MD2. Threshold disordering was observed for the item EI4 (I get upset or sad at work without knowing why) with two highest thresholds being reversed. The results of hypothesis testing for true ordering or true disordering were once again undecided. The pattern of residual correlations once again showed clusters of items within each subscale. The same pair of items (MD1 and EX5) was the only pair between the subscales with high observed residual correlation (0.20). No DIF for age was observed for any of the items. (Details about analyses for sample 2 can be obtained upon request from the first author.)

In the next step, items within each subscale (exhaustion, mental distance, cognitive and emotional impairment) are combined into four testlets as a way of handling problems with local dependency between the items, and the Rasch analysis was re-run on the four testlets as input variables. This analysis resulted in a good fit to the Rasch model in both subsamples (Table 1, BAT 4 testlets analysis). Compared with the initial analysis, item and person residuals in the testlet analysis were closer to the expected values of 0 and 1. The value of the χ2 statistic dropped to 44.53 and 35.34 in the two samples respectively and was no longer significant. The percentage of the significant t-tests also dropped from 20.7% to 5.4% in sample 1 and the lower confidence bound was below the predefined value of 5%. Corresponding figures for sample 2 were 21.2% and 4.9% respectively.

As expected, due to the lower number of variables in the model, the value of PSI decreased to 0.84 and 0.85 in the two samples respectively. In the BAT23 analysis, the coefficient alpha, i.e. the proportion of the non-error variance out of the total variance in the data was 0.94 in both random samples. After accounting for local dependency by the testlet analysis, the value of alpha was 0.81, which corresponded to the proportion of the common true score variance (Table 1). In both random samples, the proportion of the common true score variance out of the non-error variance is 0.86, meaning that most of the non-error variance is explained by the true common score variance. Average latent correlation between the four subscales was 0.61. In all, these results justify that the 23 items can be combined into one measurement of total burnout. No DIF for age was observed.

The targeting is visualized in Fig. 1 and shows the distribution of item thresholds (lower part of the figure) and persons (study participants) levels of burnout (upper part of the figure) along the common logit scale (higher values indicate higher burnout levels). The person mean was -0.95 (SD 0.63) on a logit scale and was lower than the items mean (constrained to 0). This means that rather low levels of burnout are observed in the sample, which was also seen in the frequency distribution of the items (Supplementary file 2). As explained above, the highest response categories were very rarely used by this occupation group.

Given the fit of data to the Rasch model, burnout levels in Swedish midwives were calculated. Frequency distributions for each item are shown in SI 2 (Supplementary file 2). Although the replies were distributed between all five response categories on all items, the frequencies of the two highest categories (often and always) were low on most of the items and especially for the items belonging to the MD, CI and EI subscales. Consequently, the first two response categories (never and seldom) were used by the majority of the respondents on these three subscales.

Median levels of burnout together with the first (Q1) and third (Q3) quartile for the overall burnout level (BAT-tot) as well as for the four subscales (EX, MD, CI, EI) for the total sample and for the two age groups (classified by the median age of 47) are presented in Table 3. The median burnout level for the total sample on the BAT-tot was 2.0. The Q1 was 1.6, meaning that the 25% of participants with the lowest scores rated their burnout levels in the range of 1 up to 1.6. In the same manner, 25% of the participants with the highest scores were observed in the range of 2.4 up to 4.4. The range for the total scores on each subscale for the BAT-tot are also shown in Table 3. Although there was a statistically significant difference between younger and older respondents in all five measures (p < 0.0001 for BAT-tot, EX, MD and CI and p = 0.006 for EI), their actual levels of burnout were comparable (Table 3).

The scores of the four subscales can be used to differentiate the picture. The response profiles of midwives were visualised by boxplots in Fig. 2. Although the median level of BAT-tot was 2.0, average levels on the exhaustion scales were higher, where the value of 2.0 corresponded to the Q1. The opposite result was seen for the MD and EI, where the value of 2.0 corresponds to the Q3. The median value for the CI did not deviate from the median value of BAT-tot.