Longitudinal evaluation of a knowledge translation role in occupational therapy

The aim of this study was to evaluate the impact of a leadership position for knowledge translation in occupational therapy in the first 2.5 years. To achieve this aim, the study sought to describe:

The setting of this study was a public mental health service for people living in a major Australian city. This service delivers a comprehensive range of specialist, community and hospital-based mental health services for youth, adult and older people who are experiencing, or are at risk of developing a serious mental illness. The service comprises 6 programs, located in 32 separate sites across the catchment area. It also includes tertiary specialist services for neuropsychiatry and eating disorders. The current occupational therapy workforce numbers 90 individuals, or the equivalent of 70 full time positions. An organisational diagram to illustrate structure of the occupational therapy service at this service has been included as supplementary information for context.

This study used multiple methods to measure the impact of the occupational therapy leadership role in research and knowledge translation, and received ethics approval from the service’s Office of Research (QA2014029). Data was collected via workforce surveys in March–April 2014 and November–December 2016. Workforce surveys had been undertaken previously at the service around recruitment and retention, workplace culture and safety issues [19]. However, the two surveys reported here focused specifically on outcomes related to research, evidence based practice and knowledge translation.

Participants were recruited via an email sent by local Chief Occupational Therapists in each of the 6 programs. The email included a link for online completion of the survey using an institutional Survey Monkey account. Paper copies of the survey were also distributed at monthly discipline meetings, to provide an alternative method of participation. These paper copies were returned to the principal researcher, and entered into the Survey Monkey account, before being securely shredded. A follow up email was sent after two weeks, to prompt further participation and thank those who had returned their survey. The survey remained open for responses for a further two weeks (four weeks in total).

The first page of the survey provided a plain language statement, which highlighted the voluntary nature of participation along with the context of the study. If the participant provided consent, they were taken through to the rest of the questions. The survey had four sections, and began with open and closed demographic questions (About You). One of these questions asked participants if they had participated in a quality assurance or research activity in the previous years, focusing on participation in projects that had been formally reviewed by the service’s Office of Research. Participants were then asked to complete the Evidence-Based Practice Attitude Scale (EBPAS-15) [20]. The third part of the survey collected data about the participant’s evidence based practice social network, with questions asking for identification of key contacts and frequency of contact. Finally participants were posed open and closed questions about the Lead Research Occupational therapist role, and what they wanted to see in the next iteration of the occupational therapy services’ research strategy. A series of 18 characteristics of the role were surveyed, derived from Wenke and Mickan’s systematic review [15], with participants asked to rate the incumbents performance on a scale of 0–5 (0 = poor and 5 = excellent). This section of the survey was only administered in 2016, as the role had not been established long enough in 2014 to gain meaningful valid data.

The majority of the survey consisted of bespoke questions, but two standardised outcome measures were included – the Evidence Based Practice Attitude Scale (EBPAS-15) [20] and the Evidence Based Practice Implementation Scale [21]. The EBPAS-15 measures mental health clinician attitude towards evidence based practices in healthcare, and was administered in both 2014 and 2016. The scale uses a Likert format, with 15 questions related to four subscales – appeal (4 questions), requirements (3 question), openness (4 questions) and divergence (4 questions) [20]. The 5-point scale for each statement is rated from 1 (not at all) to 5 (very great extent) with higher scores indicating more positive attitudes. This scale has confirmed internal consistency reliability.

The psychometric properties of the EBPAS-15 have been established in several studies [22‐24]. Moderate to excellent reliability is reported for the subscales and total scale score in a sample of mental health service providers [25], and was confirmed in a further study with a more geographically diverse sample [22]. The content validity of this tool was established during its development from a literature review and consultation with both mental health service providers and researchers [20]. Good construct and convergent validity has also been reported in relation to measures of mental health organisational structure [20], and organisational culture and climate [23, 26].

The Evidence Based Practice Implementation Scale [21] measures changes in practice around research and knowledge translation, and was only administered in 2016 for benchmarking [21]. The original scale has 18 items, but only the 10 knowledge translation items were utilised in this study. Research into the psychometric properties of this scale has provided some evidence for its reliability and validity [21]. In a heterogeneous sample of nurses, the scale was found to have excellent internal consistency; good construct validity and very good criterion validity [21]. Versions of the scale translated into Slovak and Czech also demonstrated good to excellent internal consistency and established cross cultural validity [27].

All data analysis for this study in based on description of the sample characteristics, and was completed using SPSS Version 25 software and socnetv. Means and frequencies have been used to describe the workforces participation in quality assurance and knowledge translation activities, attitudes towards evidence based practice, perceptions of the knowledge translation role, changes in the social network and outcomes for key performance indicators.

Distribution was assessed using Q-Q plots and found to be non-parametric. Therefore, Mann Whitney U Tests and Chi Square Tests were used to analyse any differences between the samples in 2014 and 2016. Due to the data not being normally distributed, Cliff’s Delta was used to analyse changes in quality assurance and knowledge translation activities. Cliffs Delta is suitable for non-parametric datasets, as it makes no assumption about the shape or spread of the data distribution [28]. This statistic cannot be calculated in SPSS currently, and so calculations were made using a specially designed Excel macro which also calculated the equivalent effect size using Cohen’s d [29]. A small effect for Cohen’s d is a result between 0 and 0.2, a medium effect is 0.3 to 0.6 and a high effect is 0.7 and above [30].

Social network analysis has emerged from social science in recent years, and aims to consider social relationships in terms of network theory [31]. In particular, it focuses on the relationships between groups of individuals and the resources to which membership of these groups facilitate access [32]. It is a method which highlights the relationship between individual behaviour and systemic change [33]. While social network analysis has been used in Australian mental health services before [34], this study is one of the first to apply it to knowledge translation in health. De-identified data from the survey was entered into Social Network Visualiser (socnetv). This program converted the data into an adjacency matrix and network graph, showing connections that were analysed as a complete network (i.e. including contacts both in and outside of occupational therapy). This is a binary directional network, showing the direction of communication between nodes, and the cognitive social structure within the service associated with evidence based practice [33]. This network is only relevant to communication around evidence based practice and knowledge translation, and the representation of this service may be significantly different that that found in other services for the same topic.

The following metrics were analysed descriptively – nodes, arcs, density, distance, diameter, average clustering coefficient, and centrality. Nodes are represented as circles on the network graph and represent individual clinicians [32]. Arcs are the lines that connect each node, which represent the direction of each relationship. Density is calculated by the total number of arcs divided by the total possible number of arcs, and expresses how aware clinicians are of each other [32]. Distance measures the minimum number of arcs between two clinicians, while diameter refers to the longest of the possible paths between clinicians [35]. The average clustering coefficient indicates the degree to which clusters of clinicians are evident within the network, while centrality illustrates how often clinicians interact with others in the network [36].

Finally, the outcomes of this position were reported against a set of key performance indicators established at its commencement. These indicators were developed from both an academic perspective (reflecting the traditional metrics of this sector) and a knowledge translation perspective. The indicators have evolved over time, and are now formalised in the annual occupational therapy research strategy at this service. This strategy is aligned with the service’s organisational mission and its research committee multidisciplinary research plan. These outcomes include a further year of data collection, and reflect the impact of the position over 3.5 years.

A total of 86 responses were received from occupational therapists participating in this study; 42 in 2014 and 44 in 2016. Some of the participants participated in both surveys, but due to workforce turnover, some only completed one survey. It was not possible to track individual participant answers due to confidentiality requirements. Participants came from all 6 programs, and the majority were working in the community. Participants were also drawn from a range of experience levels, with some working in management and generic positions also participating. The majority of participants had Bachelor qualifying degrees, and a substantial number were studying for higher degrees (n = 21, 50% in the first sample n = 27, 61% in the second sample). Chi square and Mann Whitney U analysis resulted in no significant differences between the two samples on any of these variables, which are illustrated below in Table 1. Please note that not all participants answered all demographic questions, and some participants worked across several settings. As shown in the Table, there were relatively few demographic differences between the samples that responded to the two surveys, although the second sample did have more variation in regards to years of service as both an occupational therapist and in mental health.

The average number of formal quality assurance and research activities undertaken by the workforce was 0.74 in 2014 (SD = 1.18, Range = 0–4), and 1.60 in 2016 (SD = 1.64, Range = 0–5). A medium effect size was identified with Cliff’s Delta = 0.44 (95% CI [0.22, 0.62]). A greater proportion of participants self-identified as research generators (26 to 34%) and leaders (7 to 11%) in the second sample, but these changes were not statistically significant.

As shown in Table 2, at the time of the second survey participants engaged in the majority of knowledge translation activities surveyed using the Evidence Based Implementation Scale 1–3 times in the previous two months. However, no participants evaluated the impact of evidence based practice on patient, clinicians and systems outcomes formally, or shared the outcomes of quality assurance and research with patients and carers.

There were no significant differences between the EBPAS-15 scores over time. However, all subscales scores changed in a positive direction over time (as illustrated in Table 3), as indicated by either increased or decreased mean scores.

As shown in Table 4, the average ratings provided by the workforce after it had been established for 2.5 years indicated a positive perception of the Lead Research Occupational Therapist role.

The workforce generally requested contact with the Lead Research Occupational Therapist on an occasional or monthly basis, and these preferences did not change significantly from the roles initial months in 2014. However, there was a significant shift in the method of contact requested over time. A Chi Square analysis confirmed that the number of occupational therapists preferring online contact rose significantly between 2014 and 2016 - X² (130.00, N = 131) = 8.19, p = .04. In the first 2.5 years of the position, preferences for face-to-face, telephone and meeting contact decreased, although not to a statistically significant degree.

As shown in Figs. 1 and 2, the social network around evidence based practice showed more connections between clinicians, and less bottlenecks (where a single clinicians is the only point of contact between areas of the network). The Lead Research Occupational Therapist position is highlighted in grey, and has moved from the periphery to the centre of the network over time.

As shown in Table 5 below, less nodes (i.e. evidence based practice contacts) were identified in 2016, but a comparable number of arcs. This reflected an increase in the links between, and awareness of clinicians within the network. A considerable amount of homophily was demonstrated by the clinicians within the inner circle of the network, who were all Chief Occupational Therapists and therefore similar to each other [33]. The diameter of the network also increased, along with the degree of clustering of clinicians. The metrics also indicate a modest increase in the frequency of interaction between clinicians in the network.

As shown in Table 6, all key performance indicators have been met over the first 3.5 years of the Lead Research Occupational Therapist position. The 28 active research projects currently underway involve the active participation of 41 individual clinicians, which represents 46% of the total service workforce.

There are several limitations to acknowledge in this study, the foremost of which is its containment within a single service. The size of this service supports an unusually large and diverse occupational therapy workforce, and its size undoubtedly contributed to the service’s ability to create and maintain the Lead Research Occupational Therapy role. Context is recognised as a crucial factor in knowledge translation, and this will limit the ability to generalise the findings reported here more broadly. Around 50% of the available workforce responded to these surveys, however a more representative sample would have provided further information about the social network, which could have been important to understanding its structure. The samples were different across time (with staff attrition influencing the membership of the workforce), and this may also have influenced the findings, as the design did not provide for matched pre-post sample and measures. Tracking outcomes over time for clinicians in a matched sample would enable the exploration of the potential impact of these roles at the individual level. The workforce in this service is very interested and keen in research and knowledge translation, and this may not be true of other workforces in other services. Finally, the analysis here does not take into account the specific activities and proportional time spent by the Lead Research Occupational Therapist to achieve these outcomes. A greater understanding of this ‘active ingredient’, described by some as the ‘black box’ of implementation science [51], is required to fully understand the longer term sustainability and viability of this position.