Evaluation of health care providers’ use of the ‘Exercise and Depression Toolkit’: a case study

Ethical approval was obtained from the University of British Columbia Ethics Board. Participants were recruited from across Canada through purposive sampling. HCPs that had participated in interviews during a previous phase of toolkit development were contacted and invited to participate in the study. This was done considering the DoI theory, as these individuals were considered likely to be innovators [8] of adopting the toolkit (the first to use it) as they have an identified interest in physical activity and mental health, and would be receptive to implementing the toolkit in their practice. Recruitment emails were also shared through private practice and public health authority networks.

To be eligible for participation, HCPs had to be currently working with adults with depression, working a minimum of 2 days per week, and not have any planned vacation (for more than a week) during the four-week evaluation period. The toolkit was created for a variety of end users: HCPs who work with adults with depression such as Family Physicians, Psychiatrists, Counsellors, Mental Health Workers, Occupational Therapists, Recreation Therapists, Nurses, and Social workers. The aim was to recruit a sample inclusive of different end users from diverse professional designations.

Case studies are commonly used to understand and explore professional practice behaviours [18] and in particular nursing practice in Canada [19]. A case study can be defined as an intensive, systematic investigation of a single individual, group, or community in which the researcher examines in-depth data [18]. Case studies are used to explore complex phenomena in a natural setting and are recommended as a research design to address how and why questions, and situations where a researcher has little control over manipulation of a behaviour [18]. The case study design allowed consideration of the individual health care providers and their context (e.g. family physician in primary care), as well as the providers as a group of ‘mental health professionals’. Within this design, the authors opted for multiple data collection methods to better understand health care providers’ practice behaviour in-depth.

Semi-structured telephone interviews were conducted by the first author (KG) pre- and post-intervention (after use of the toolkit for 4 weeks) scheduled at times convenient for participants. Participants were also given an electronic weekly log (that could be filled out by hand or online). The log was used to track use of the toolkit, as well as satisfaction with respect to their interactions with individuals with depression, and details of the ‘end point’ of what occurred after using the toolkit. Data collection occurred July 2019–October 2019 with rolling admission into the study. Interview guides and the weekly log are provided for reference as supplementary materials.

A pre-intervention interview guide was developed to help understand HCP’s practice before using the toolkit. During the pre-interview, a brief overview of how to use the toolkit was provided. Participants were then given four weekly logs to use at their discretion in their practice settings for 4 weeks. The log provided participants with an option to check and track the days that the toolkit was used and if a referral to an exercise program was made. Participants were asked to rate their perceived success of interactions with the individuals they work with when using the toolkit on a scale of 1–7 (1 being extremely low success, and 7 being extremely high success). Prompts were provided to consider success including usefulness, client receptiveness, and satisfaction. In-depth phone interviews were then conducted with all participants at the end of 4 weeks once all weekly logs were received. Questions addressed acceptability and prompts were included for each of the constructs of acceptability and the perceived attributes of an innovation. For example, for the acceptability construct ‘affective attitude’, participants were asked: what did you like about the toolkit, and what did you dislike about the toolkit? Final questions concerned the intent to continue using the toolkit, recommending it to colleagues, and possible dissemination strategies. Data collected from pre-intervention phone interviews and the weekly logs were used as prompts for these final interviews. Pre and post-intervention interview guides, as well as the weekly logs are available for reference as supplementary materials.

All interviews were audio-recorded and transcribed verbatim. Weekly logs were transferred or transcribed into one document for each participant. Logs, interviews, and transcriptions were assigned participant ID numbers to maintain confidentiality. Names were removed, and participants are referred to as their provider designation. Interviews ranged from 30 to 60 min in length. A content analysis was conducted to prepare, organize, and report on the data [20, 21]. A codebook was first created using the seven component constructs in the TFA [13] as well as the innovation attributes of the DoI [8] and their respective definitions. The concepts of adoption and dissemination were also included. Although coding was primarily deductive, there was flexibility in creating new codes if necessary. QSR International’s NVivo 11 Software was used to manage data and support qualitative data analysis. Coding was conducted collaboratively by the first and second author (KG and DZ) on one transcript and one weekly log to ensure understanding of the codes and concept definitions. After this collaborative approach, KG and DZ coded all transcripts independently and then met to review and discuss the coding framework. Any discrepancies in coding were discussed until agreement was reached, and when needed, the last author GF was consulted. Units of data were coded as statements which ranged from a sentence to a paragraph. Statements were considered new ideas or concepts, hence the variability in length of what was coded. For theoretical constructs, the authors coded statements as negative or positive. This was determined based on the participants’ description of views, context, and tone. A negative statement was considered something that would negatively impact adoption, and a positive statement would positively influence adoption. For example, if a participant felt that the toolkit was difficult and hard to understand and use, this was coded under complexity as a negative statement. The authors attempted to use one code to isolate the meaning of the text, versus double-coding passages of text. For example, participants were asked how they felt about the toolkit (their likes and dislikes). An example of a response commonly heard was “I just liked the simplicity, so, not too many pages, because that gets overwhelming for patients”. Such a response, stating it was simple or easy to use, was coded to complexity (as a positive statement) rather than complexity and affective attitude (how an individual feels about an intervention).

Various criteria were adopted to ensure ‘trustworthiness’ of this qualitative approach [21]. A critical reference group was consulted prior to conducting the study for data collection methods, evaluation of interview guide questions, and purposive sampling strategy [20‐22]. Throughout the analysis process, two independent coders (KG & DZ) utilized a detailed pre-determined codebook and discussed meaning units and categorization throughout the analysis process [21]. Based on recommendations [21] the authors attempted to describe the analysis process in detail, with results reported systematically, carefully and with transparency as to how connections were made between the data and results.

All participants (n = 6) self-reported personally meeting the Canadian Physical Activity Guidelines. All participants viewed the toolkit and recommending exercise as aligning with their personal beliefs and values, as well as their beliefs and values as HCPs. Some participants also explained that the design (format and layout) and some general content of the toolkit aligned with how they were already practicing so this made it easy to transition into using it regularly. The Family Physician explained:

“The toolkit goes into my style of like how I like to run my office anyways, like I use a lot of drawn diagrams. I do like to make sure people understand, and I usually write down things for them. So, it wasn’t like, for me that whole kind-of process is not like – I prefer to practice like that, so I think – that’s why I found like, I had good, good experience with it.”

All participants stated that the toolkit was easy to use, understand and not complex. Some participants also identified that individuals with depression they were working with found it easy to use and understand as well, which reinforced the HCP’s positive evaluation of the toolkit and desire to continue using it. The Occupational Therapist explained:

“I liked that it was very simplistic. It’s not complex, it’s not too long, so, when you have a client, or have the client review the information, it’s not too challenging … like it was simple to use.”

Another participant identified that when she first received the toolkit, she had to concentrate when using it and that it was harder for her to make the connection between exercise and depression. She went on to explain in her post-interview that with time and continued use of the toolkit, it became easier.

Whether HCPs viewed the toolkit as being adaptable to their practice context was considered for trialability. All participants identified that different toolkit sections could be used as desired based on their context. The Nurse Practitioner included an unprompted comment in her weekly log after using the toolkit, “Individual previously had finances required for gym membership – this no longer the case. We discussed alternative plan which can be done @ home @ 0 cost: i.e. push-ups, crunches, squats, lunges, jump rope, biking, running, fast-paces walking”. This demonstrated her ability to use the toolkit with an individual in her practice, despite the fact that the individual she worked with had financial barriers and she was not able to refer to a structured exercise program or facility.

Given the nature of the case study approach, it was not possible for participants to observe their peers use the toolkit in practice. In this context, we defined observability as ‘the extent to which the results of an evidence-based program become visible’ [23]. Participants identified that some of the individuals with depression reported they went and tried to exercise whether it was at home or in a structured class. From this, HCPs considered patients receptive to the idea of exercise, and some identified changes in mood and conversations on follow-up visits. The conversation with the Nurse highlighted this:

“I had another patient who wasn’t on the log, who brought me back some of the stuff filled in, yesterday actually.. . very basic, like she could only manage once a week walking to go get her kids from school. But that was better than zero, you know?”

Although the majority of statements within this construct were coded as positive, reasons for not observing any changes included lack of follow-up visits with people, individuals’ severity of depression with symptoms such as being unmotivated and tired.

For this construct, HCPs were asked specifically about their likes and dislikes. An overwhelming number of positive statements about the toolkit were expressed with and without prompting questions throughout the interviews and the logs. The Family Physician expressed:

“… the toolkit and like exercise and all that, is a good way to start that kind of – tap into all like the needs of the patient, as well as like, now with like evidence that can actually provide benefits … I think that was interesting. Like there’s something that you can like show them, you know, that it’s going to … like it’s proven to help, you know? Instead of just saying, ‘you exercise, you’ll feel better, you know?”

The Psychotherapist also expressed her positive feelings:

“But it was nice to actually have a handout on it, and actual tools that I can actually give to them, rather than say, ‘you know, exercise is great for depression’, it’s actually to have that toolkit at hand … and I think it was very beneficial for the clients too, because then they have something to take home.”

Three negative statements from two participants were about parts of the toolkit they personally did not use or find helpful rather than something they specifically disliked about the toolkit overall.

Views of the HCPs were divided on how much time and effort was required to use the toolkit in practice. Participants were asked if they felt the toolkit required a lot of time and effort to use. Three participants viewed the toolkit as requiring little time and effort, as the Social Worker explained:

“No, because it’s just almost like bullet points. So, then when I’m talking to someone, I can say, ‘how does exercise help’, and then, there’s another sentence that I can just follow after that, you know. And when I’m talking to someone, having just those there very short bullet points, makes it a bit easier.”

Conversely, two participants felt that it does require time and effort, although this was minimal. The Occupational Therapist explained her views on the toolkit requiring more time and effort when working with new individuals:

“… where she was as a client, who was brand new, it might take a little – it would probably take a little bit more time in terms of being like, ‘okay, well, you know, this is how you’re feeling, these are the potential benefits of exercise, this is why it can add to your life’.”

For opportunity costs, HCPs were asked if they felt that using the toolkit took away from other priorities they had (what they give up to use the toolkit in practice). Almost all (n = 5) participants felt that the toolkit did not take away from other priorities, and some explained that they felt exercise was a priority that they should be discussing. The Psychotherapist explained her priority of exercise: “Well, I always thought exercise is a priority of treating depression. So, I felt it enhanced that, because I would just have that conversation, and set some, you know, realistic goals with them”. On the other hand, the Nurse Practitioner said she felt that it did take away from other priorities:

“yes, it’s a little bit reprioritizing some of the workload … the time you take – yes. But just even the fact of, you know, pulling out the sheet, that’s – even if it’s embedded in your system, just going through that process of learning that, clicking on the right places, all that does cause some, you know, barriers. You know, we don’t like to change our ways”.

HCPs were asked about their confidence in using the toolkit to discuss and recommend exercise, as well as if the toolkit had changed their confidence levels in general to discuss and recommend exercise with individuals with depression. The Family Physician explained that the toolkit helped her gain confidence in discussing exercise by providing conversational pieces:

“Yes, it kind of gives me like the … ‘trigger words’, you know, the important kind-of … like the CANMAT guidelines, like I can use like certain things when I am explaining to patients. And then they can go and do more reading, right. So, I think it explains everything to them.”

Other practitioners did not feel that the toolkit specifically helped to improve their confidence. These practitioners also felt confident in discussing exercise prior to receiving the toolkit.

In general, the participants demonstrated at some point in the interview that they understood the purpose of the toolkit. Only one participant did not seem to understand its purpose until the post-interview period. She explained a lack of understanding with regards to the intended population to use the toolkit with someone (with mild-moderate depression), and in making a connection between exercise and depression:

“So I don’t know if my population is what would normally use this toolkit … because, my population was not as severe as someone that - and my population has some awareness of the relationship between exercise and depression. But I didn’t actually bring that together with the connection using the toolkit”.

Throughout the post-interview she recognized in hindsight how she could better implement the toolkit in her practice and acknowledged she would do so moving forward through promoting the connection between mood and physical activity and not just recommending exercise.

A strength of this study is the novel theoretical underpinnings using the DoI theory and the TFA. Using the hybrid approach was helpful for understanding perceptions of important aspects of the toolkit, as well as cognitive and emotional responses to using it. Important factors may have been missed if both theories were not utilized. In terms of limitations, due to the case study design, observability could not be explored in the context of peer behaviour with regards to use of the intervention. Some participants did not always observe changes in people with depression they were working with. The study length was 4 weeks and one participant noted this was not enough time for her to follow-up with patients. Thus, little is known about secondary outcomes of the toolkit for individuals with depression such as mood and physical activity changes. Participants were recruited from a previous study to inform development of the toolkit. While this was done purposely considering the DoI theory, the two participants involved in both studies may have been predisposed to perceive the toolkit favorably. This study had a small and homogeneous sample with regards to gender, ethnicity, and age and participants only represented two provinces across Canada. Participants were physically active and likely receptive to exercise as a treatment consideration. This may explain little impact of the toolkit in enhancing self-efficacy to discuss exercise with individuals. The authors do not know how the toolkit would be received by health care providers who do not have time or interest in considering exercise as a treatment for depression. While there are no current plans to further evaluate the toolkit itself, a team is monitoring perceptions of its implementation within a campus-based exercise intervention for students seeking help for depression.