The challenge of change in acute mental health services: measuring staff perceptions of barriers to change and their relationship to job status and satisfaction using a new measure (VOCALISE)

Mental health nurses perceive many objective barriers to change in acute ward settings, including limited resources (e.g., beds, staff), the process of bed management, poor/indeterminate leadership, and violence[10, 11]. Despite these perceptions, wide-scale changes to services have been pushed through in recent years, driven by political and economic factors. However, there is an increasing body of evidence in healthcare showing poor uptake of innovations and evidence-based findings into practice[12‐14].

During periods of change, there are reciprocal interactions between the process and those involved[15]. The literature shows that low levels of involvement in change and changes that are not perceived to be beneficial can negatively impact the workforce. Organisational change in the UK National Health Service (NHS) is often imposed via a top down approach, which may not take into consideration the views of the majority of nursing staff. Indeed, change process issues such as poor involvement in planning, implementation and control of the project have been highlighted as potential barriers to success[16]. Staff in leadership roles may view changes more positively than those in direct care roles because they form part of the consultation process[17, 18]. In our study, we will expand on this theme in a new context - acute mental health wards - hypothesising that nursing staff occupying managerial roles will view change more optimistically than more junior direct care staff.

In a study examining organisational justice, Haar et al.,[18] found that those employees of a local government organisation who were likely to directly benefit from work-family policies viewed them more positively than those who were not affected, who saw them as unfair. We will develop this theme to illuminate how organisational justice is perceived in mental health settings.

Other studies linked to acceptance of change have been carried out[19]. In particular, ‘personal resilience,’ characterised by perceived control over proposed changes, optimism, and self-esteem, was a predictor of acceptance to change. Poor job satisfaction was an outcome for staff who were less open to changes (i.e., who perceive more barriers)[19], a finding that has been replicated in the wider healthcare literature[1, 2]. Poor job satisfaction is not only damaging for the individual, but has organisational consequences as it has been linked to low staff retention[20]. We will explore the link between perceptions of barriers to change and job satisfaction in nursing staff in acute mental health settings.

Involving all staff in changes would present significant logistical problems. However, staff opinion might be accessed directly before changes are implemented using a measure that captures staff sentiment on the front line. In this study, we generate such a measure using a participatory method of measure development[21], which is known to improve participation and engagement. We believe that this participatory approach produces items that are of importance to the group under study[22, 23].

The findings from studies looking at private sector organisations or U.S. healthcare organisations may not be generalisable to U.K. healthcare organisations, which are funded through taxation and which have a constitutional aim to consider the best interests of the client group[24]. There are currently no psychometrically robust measures designed to capture nursing staff perceptions of barriers to change in acute in-patient wards, nor has the idea of staff’s direct participation in measure development been explored in this context. There are several measures developed in healthcare that focus on the uptake of evidence-based practice in clinical areas by mental health providers and nurses[25‐32]. However, there are no measures which focus on general changes in a mental health setting.

In this study, we first develop a psychometrically robust measure specific to the context of acute mental healthcare that includes a qualitative exploration of the emotional responses of staff in relation to themes of organisational unfairness. Then we use the new measure to explore how characteristics identified in the organisational literature are linked to perceptions of barriers to change. Specifically, we explore: whether staff in more senior positions have more positive perceptions of barriers to change than staff working in direct contact with service users; and whether staff with negative perceptions of barriers to change also have poor job satisfaction. This will clarify the impact of change on nursing staff.

Ethical approval for this study was awarded by a local NHS Research Ethics Committee (07/H0809/49). Staff were assured that confidentiality and anonymity would be maintained unless any concerns about patient safety arose during the process. All staff gave informed consent to participate and were paid five pounds for their time.

The process of measure development occurred across three stages which are outlined in Figure 1.

The participatory framework for VOCALISE was adapted from the User-Focused Monitoring method developed by Rose[21] and Rose et al.,[33, 34]. This model is ideal because it promotes involvement from nurses as ‘stakeholders’ of their clinical areas. The core principle is to translate participant views into items to ensure strong item content validity. This is accomplished by maintaining a direct contextual link between the researcher and participant through shared experiences. This reduces the usual power differentials between interviewer and interviewee, resulting in data that more accurately represent participant views[35]. In this research, the shared experience was being a psychiatric nurse.

The criterion sample (N = 32) was drawn from a large London mental health trust and comprised eight individuals from each level of acute in-patient nursing staff (nursing assistants, entry level qualified nurses, senior nurses and ward managers).

Each staff member participated in a 30-minute interview based on themes drawn from the literature and from a reference group that included two nurses, a service user researcher, and a clinical psychologist. As the concept of ‘perceptions of barriers to change’ is abstract, the interviewee was asked to consider a scenario in practice where a significant change to clinical practice had occurred. Successive thematic analyses[36] were conducted using a qualitative software package (NVivo). This allowed new themes to be incorporated into subsequent interviews and ensured that the content domain was explored broadly, enhancing content validity.

Themes with the highest numbers of references and which were therefore the most important issues to staff were included as items that also made use of participants’ language. Consideration was also given to themes that were expressed with vehemence. Independent raters compared how similarly they interpreted the themes and codes of one interview using a function within NVivo to minimize subjectivity and increase how reliably the data were interpreted.

Two rounds of feedback from acute ward nursing staff were then used to improve the item wording. This expert validation process also formed another reliability check for the accuracy of the final questionnaire items and ensured high face validity. At the end of the development process, VOCALISE contained 23 items, which were answerable on a 6-point Likert scale ranging from strongly agree to strongly disagree.

Figure 2 outlines the psychometric testing process based on the Health Technology Assessment programme recommendations[37] and Streiner & Norman[38]. Analyses were conducted using STATA 11 and SPSS Statistics 20. The results of the psychometric tests were used to improve the measure, which meant that modifications occurred during the analyses and the final number of items was reduced.

All mental health acute ward nursing staff were invited to participate in this phase (ward managers, senior nurses, entry level qualified nurses and nursing assistants). Four sets of questionnaire data were collected from these staff, from all four boroughs of a large London trust. Each set of data was for a different purpose:

Preliminary tests included an intraclass correlation, which was computed to establish that ‘between ward’ variance was greater than ‘within group’ variance[5]. Random effects regression models, clustering on ward, were then used to explore the effects of all staff characteristics on VOCALISE. Random effects models were appropriate because the data were multi-level. Random effects models meet the assumption that differences in the data relate to subsets of the population (in this case, that groups were located within wards). They control for unobserved heterogeneity so that significant effects can be said to be true effects rather than those observed due to differences in the sample. This method of analysis was also appropriate because despite differences in the scores of individuals, within each ward, the distribution was roughly symmetrical, suggesting general consensus.

Perceptions of job satisfaction were measured using the Index of Work Satisfaction (IWS)[39]. VOCALISE captured staff perceptions of barriers to change. Negatively phrased items were reverse scored so that higher scores indicated more negative perceptions.

Feasibility and acceptability were assessed on the 23-item measure. For feasibility, two additional questions assessing whether the measure was easy to complete and understand were asked of group one (N = 40), and for acceptability, a further two questions were added to the assessment of group four (N = 125).

Group two (N = 42) completed the questionnaire at two time points. Test-retest reliability was assessed on total scores using Lin’s concordance coefficient[40] and paired t-tests to investigate whether consistent bias existed. Item reliability was assessed using weighted kappa coefficients[41] on individual item scores at the two time points. Generally, scores of 0.21 to 0.4 are said to indicate ‘fair agreement’, scores of 0.41 to 0.60 indicate ‘moderate’ agreement, and scores above indicate ‘substantial’ agreement[42]. As data were often clustered on particular responses, this meant that a high kappa was not achievable irrespective of the level of concordance between the two time points. Therefore, kappa max was computed, which is the proportion of the maximum kappa attained[43]. The assessments were six to ten days apart, which was appropriate because the measure is of individual perceptions, which are subject to change over time. Items were removed on the basis of poor test-retest reliability.

The internal consistency of the scale using Cronbach’s alpha[44] was then investigated with items that had performed poorly on the basis of test-retest removed.

Construct validity was examined through an exploratory factor analysis on responses from group three (N = 275), following preliminary checks (Bartlett’s test of sphericity and the Kaiser-Meyer-Olkin measure of sampling adequacy), to determine the degree of correlation in the data.

Using data from group three (N = 275), the internal consistency of the subscales were tested using Cronbach’s alpha[44].

Exploratory analyses of the effects of staff perceptions were undertaken to assess criterion validity on the data from group four (N = 125), using random effects regression models, clustering on ward, to predict VOCALISE total scores in relation to two hypotheses:

The IWS scores were dichotomised to create the two groups, using an a priori point (171). This was the average from four studies of nurses’ job satisfaction, which used IWS[45‐48]. Positive perceptions were therefore defined as scores of 171 or below.

There was a good spread across the demographic criteria of age, ethnicity, borough and gender of participants in this phase (see Table 1).

During the interviews, staff focused on changes that were large scale and intended as long term. Examples include: the introduction of protected therapeutic interaction time for key workers and their clients, the introduction of the smoking ban in hospitals, and the introduction of the ‘Knowledge and Skills Framework’, which is an NHS framework designed to monitor and support the management of career development. Within these examples, all staff also described the effects of incremental local changes and daily ward changes.

Data reduction produced seven over-arching domains: ‘communication’, ‘generation of ideas’, ‘outcomes of changes’, ‘resistance’, ‘strategy’, ‘support and monitoring’, and ‘team dynamic’. Themes with fewer than 20 references were not included. Items were constructed around perceptions of the team/ward and perceptions of the self in relation to change. During the inter-rater reliability exercise, the raters achieved an 87% level of agreement in their interpretation of the main themes in the data. Those who participated in the expert validation process confirmed that the content of the measure was relevant and that the content domain had been widely explored.

The qualitative data illustrated the emotional responses of staff toward changes. Quotations that focus on the theme of organisational unfairness are presented in Table 2.

VOCALISE emerged out of a novel method of measure development that emphasizes the participation of nursing staff as a stakeholder group within mental health settings. As a brief self-report tool, VOCALISE can be used to capture staff perceptions of large scale changes to practice, concisely. VOCALISE is advantageous over previous measures (such as OSC, ORC, BARRIERS, EPBAS) because it combines aspects of the organisational, social/emotional and psychological context with staff perceptions of barriers to change, rather than treating each construct in isolation[3, 25, 26, 52, 53]. This type of measure might therefore usefully inform decisions for innovation made at the organisational level, by highlighting the negative impact of change on individuals.

The participatory method yielded items that were descriptive and evaluative, suggesting that both are important in allowing staff to express cognitive and affective responses to change. Further, the benefits of the participatory model used to develop VOCALISE are visible in both the breadth of the construct investigated and the rich content of the items. The scale addresses barriers arising from ward/environmental factors as well as from social factors. The content of some items (e.g., eight, nine and eleven – see Table 4) are of interest because they exemplify feelings of organisational unfairness relating to work pressures, which appear to have arisen as a result of poor consultation around new changes. Although the idea of work pressure as a barrier to organisational readiness for change is not new[53], VOCALISE makes the link between work pressure and the way that innovation is carried out explicit, thereby specifically addressing perceptions of implementation climate. These more focused features may have been missed using a non-participatory method, and item 11, which addresses nurses’ perceptions about the effects of change on service users, is a new development. The links between how staff experience change and the feasibility of new changes in clinical practice may therefore be exposed successfully using this type of method.

The psychometric properties of the final measure were promising. VOCALISE showed acceptable agreement, according to Cohen’s kappa[41], and the total score showed good test-retest concordance. A total of 13 staff (35%) altered their score by more than 5 points in the direct care group but not in the managerial group, which indicates their sensitivity to instability in the inpatient environment in mental healthcare rather than rater bias. Indeed, during the qualitative phase of this study, staff suggested that their views might alter depending on the day, the staff on shift, and the client group. In practice, this suggests that VOCALISE data should be collected during a prescribed time frame, and the score should represent an aggregate of perceptions during that time.

Researchers in the field of implementation science have argued that items capturing ‘implementation climate’ should not be value-based and should comprise the shared perceptions of the workforce using descriptive, rather than evaluative items that focus on the organisations’ implementation practices and policies[7, 54]. Using descriptive statements that focus on the team rather than the individual might increase within group consensus[7], because individuals’ views of themselves are likely to vary more than individuals’ views of the team. However, in an environment where both positive and negative approaches to implementation exist, the use of purely descriptive statements is a challenge[55]. The participatory method used to develop VOCALISE generated eight ‘I’ and ten general statements containing the issues that staff referenced most frequently and therefore felt to be the most important when it came to describing the barriers to delivering significant changes to practice.

How statements of perception function as a means of measurement in an environment that is subject to dynamic changes on a daily basis is worthy of further clarification. A mixture of ‘I’ statements as well as general statements may be of use because, as this study shows, ‘I’ statements are less likely to change over time. There were also less missing data in the ‘I’ statements, perhaps because staff found these items easier to answer. Further, despite a mixture of both descriptive and evaluative statements, aggregated scores were successful in reflecting a ward view, because there was higher consensus in each group than between groups. The internal consistency of the entire scale was good, and the internal consistency of the subscales was adequate.

The exploratory factor analysis indicated three latent psychological dimensions. Factor one links unsuccessful change to components of acute ward working that staff perceive as beyond their control. This highlights a sense of powerlessness that may be linked to poor work-related autonomy or to how staff prioritise change. Factor two appears to relate to staff confidence in how the process of change is managed. Factor three is characterised by a sense of disillusionment and loss of motivation that impacts on the team dynamic, perhaps creating a barrier in group commitment to the process of change.

Other studies have shown that staff with higher organisational status[18, 56] view changes more favourably than those in more junior positions. Further, job satisfaction has been linked to staff who express more open views about new changes[1]. These results were replicated with VOCALISE, which demonstrates criterion validity.

Issues of consultation and fairness in relation to changes may be linked to organisational justice theory, which suggests that managerial staff perceive change more favourably because of the way that organisational outcomes are determined[56]. Staff in leadership roles are more likely to be involved in the planning stages of new changes and therefore have an increased sense of control and responsibility over them. The qualitative findings showed that direct care staff frequently described poor involvement in new changes, which supports previous findings in the literature[17, 18]. A lack of interest in innovation existed amongst the client group, which affected staff motivation to deliver some changes. This increased feelings of blame and resentment among the team, and also emerged in response to issues of unfairness as linked to innovation. In the qualitative data, this theme was most frequently referenced by nursing assistants.

As the qualitative data identified, the direct care staff also made links between powerlessness and feasibility issues, feeling a sense of injustice because they were not able to deliver changes if neither their clients nor they perceived much benefit. Emotional responses such as blame and resentment also emerged in response to unfair task allocation, or low commitment to change by colleagues. This may have ramifications for the quality of care delivered as those in direct care roles are generally those who deliver changes and who spend the most time with patients.

More research is therefore required to explore whether emphasising the involvement of nursing staff in such changes might prove an effective method of managing changes more successfully.

Other studies of staff views of organisational changes have emphasised psychological concepts such as work related empowerment[1]; control over proposed changes[19, 57]; motivation[1]; and levels of optimism and self-esteem, which have been combined into ‘resilience’[19, 57]. Context-specific variables that capture information about how staff regard the change process, such as the provision of information, staff involvement and social support, have also been found to influence openness to change[16, 56]. As a measurement tool, VOCALISE is novel because it incorporates many of these concepts into one measure (powerlessness over proposed changes, confidence/optimism and de-motivation). The individual items identify barriers/openness to change and perceptions relating to the change process as well as context specific issues.

VOCALISE was designed for use in a randomized controlled trial to implement nurse-led psychological therapies on acute in-patient wards, to assess how staff respond to general changes. Although this was a project to implement evidence-based practice, VOCALISE was not designed to rate clinician responses to the importance and relevance of evidence-based practice, specifically.

The data for this study were collected within one mental health trust. This was a very large service covering different demographic and geographically diverse areas, but this still limits generalisability.

As the amount of variance explained by the factors in this study was low and some factor loadings were relatively low, a confirmatory factor analysis on a larger sample would be useful to confirm the measure structure.

The participatory method used to construct VOCALISE revealed items which staff felt to be important. When considering complex concepts such as staff responses to innovation, this method proved effective in illuminating both staff perceptions of barriers to change but also some of the psychological and emotional effects of change on staff. This stakeholder-focused method of measure development has potential in other areas of implementation science where the relationship between change and those involved are integral to the successful uptake of change. It may, for example, usefully illuminate service user responses to innovation or to assess the barriers in sustaining innovation.

Although VOCALISE was developed to capture staff responses to significant, general changes in mental health settings, it could also measure local and project-based change. The scale was not specifically designed to assess incremental change, but the emotional and psychological aspects of the scale also reflect how staff experience the ongoing nature of change in the NHS, and therefore some effects of incremental change might be usefully picked up by VOCALISE. The item content of VOCALISE could be generalisable to other health settings, but more work should be undertaken to assess how VOCALISE performs in different areas.

The findings suggest that those members of the nursing workforce who are in direct care positions are ambivalent about new changes. This may explain why changes are difficult to embed, and staff motivation, confidence and feelings of ownership over new changes should be addressed to improve uptake.