The development of the assessment instrument was divided into four stages: literature search, concept mapping, drafting of the assessment instrument, and pilot testing and instrument finalization. Ethical approval was granted by the Research Ethics Committee of GGzE (registration number IMBB/2017022). The concept map sessions and the pilot testing were conducted in Dutch. The concept maps and assessment instrument were translated to English by the authors for publication.
Stage 1: literature search
The literature was comprehensively reviewed for working alliance assessment instruments and for studies regarding the relationship between working alliance and treatment outcomes for (F) ACT clients with SMI. The search terms were as follows: ‘working alliance’, ‘working relationship’, ‘therapeutic alliance’, ‘therapeutic relationship’, ‘helping alliance’, ‘alliance’ AND ‘instrument’, ‘questionnaire’ AND ‘case management’, ‘assertive case management’, ‘flexible assertive case management’, ‘community mental health care’, ‘severe mental illness’ AND ‘(treatment) outcome’, and ‘recovery’. Articles cited by the identified studies were also reviewed for inclusion in this study. The topic, target population, and method of analysis for each study were recorded to identify common themes. .
Stage 2: concept mapping
Two concept mapping sessions were organised to identify the key components of the working alliance between clients and (F) ACT teams. Concept mapping is a mixed qualitative-quantitative participatory approach that results in a graphical representation of the concepts, the way these concepts are organised and their relative importance [
37]. Concept mapping has previously been used in FACT research to measure quality indicators [
38,
39]. Both sessions were led by the first author and the scenarios for the sessions were written by the first author with feedback from the third author.
The first concept mapping session was conducted with team members of (F) ACT teams from GGzE (a large institution for mental health care in Eindhoven and surrounding cities, a region in the south of the Netherlands) and national (Dutch) experts in the field. The second session was conducted with clients and former clients of (F) ACT teams and peer support workers who received care from one of the teams in the Eindhoven region. GGzE has eight FACT teams which are regularly audited by a certification board (CCAF) and one ACT team. Each FACT team has a caseload of approximately 200 clients with SMI. In total, approximately 12,000 clients are cared for by GGzE each year.
Flyers and information letters were made and distributed among the teams to recruit participants. The care professionals provided clients with information letters; interested clients were contacted by the researchers with further information. After a reflection period, clients could agree or decline to participate. Professionals were also informed of the study through oral presentations in team meetings by the first author. Experts in the field were contacted by email. Information about the study was also provided to a client coordination centre visited by clients, former clients and peer support workers. Finally, information was posted on a website that promotes research at GGzE (
www.ggzei.nl). Two field experts and 13 GGzE mental health care professionals participated in the first concept mapping session. Five current clients, one client in training to become a peer support worker and one former client participated in the second concept mapping session. A purposive sampling procedure was used to generate maximum variation in responses. Both groups were diverse in terms of gender, age, level of education and years of experience with multidisciplinary outpatient teams with shared caseloads (Table
1). In the client and peer support worker concept mapping session we guarded against perceived coercion by emphasizing that the researchers were independent of the (F) ACT teams the participants receive(d) care from and that all answers would be treated confidentially.
Table 1
Characteristics of the concept map session participants
Gender | 8 females, 7 males |
Age, mean (range) | 45.2 years (34–59 years) |
Profession | Psychiatrist (1) |
Clinical psychologist/psychotherapist (2) |
Mental health care psychologist (2) |
Case manager (4) |
Social psychiatric nurse (3), in training (1) |
Director of FACT certification board in the Netherlands (1) |
Professor specialized in community care (1) |
Mean number of years of experience working in (F) ACT teams (range) | 8.4 years (1–26 years) |
Clients and former clients (N = 7) |
Gender | 4 males, 3 females |
Age, mean (range) | 41.4 years (26–56 years) |
Level of education | 3 secondary vocational education, 2 BSc, 1 MSc, 1 primary education |
Living situation | 7 independent living |
Employment status | 3 paid work, 3 no work or daily occupation services*, 1 daily occupation services |
Mean number of years receiving care from (F) ACT teams (range) | 3.2 years (1–8.5 years) |
At the start of each session all participants completed informed consent forms and provided personal demographic information. The sessions then followed a written scenario based on literature regarding concept mapping [
37,
40‐
42]. The goal of the study was explained, after which the generation of statements started. All participants were given a piece of paper with the agenda for the day on one side and the focus of the meeting and the definition of a working alliance on the other side. The focus of the meeting and the definition of a working alliance were also written down on a whiteboard in view of all participants. The definition of a working alliance was formulated as broadly as possible to minimize influence from the researchers. Specifically, we defined a working alliance as ‘the relationship between a professional/treatment team and the client who receives care from this professional/team’. The focus of the concept map session was formulated as follows: ‘the following topics are important in the working relationship between clients and a team that consists of multiple disciplines and caregivers who are all involved with the client and who also visit the client at home’. During the brainstorming, participants were asked to think outside of the box and consider all possible elements of care, including relationships and treatments, that could possibly influence the working alliance.
After one hour, the participants were asked to individually sort their statements regarding the working alliance into categories and to rate them based on their importance using a five-point Likert scale ranging from ‘not very important’ to ‘very important’. Ariadne was used to analyse the data [
43]; first, a binary symmetric similarity matrix was computed for each participant. Subsequently, the software calculated the similarity between any two statements in the same pile to create an aggregated group matrix. A high value in the group matrix indicates that many participants grouped those statements together and implies that the statements are conceptually similar in some way. The aggregated similarity matrix was used as the input for a principal component analysis that translated the correlations between statements into coordinates in a multidimensional space. Subsequently, cluster analysis was used with the coordinates to further classify the statements and group statements that were similar into clusters [
41,
42]. During the final phase of the concept mapping session, the participants were involved in interpreting the concept maps in a structured group discussion.
Each concept mapping session took approximately 4 h. Professionals were allowed to register the session as work time, and all travel expenses were covered for both groups. All participants received a box of chocolates after the session.
Stage 3: development of the assessment instrument
The two concept maps formed the basis for the construction of the assessment instrument. The first step in designing the assessment instrument was the identification of larger domains common to both concept maps and integration of the concepts. The interpretation of the participants was used to guide this task and further built upon by the researchers.
The second step was searching for the specific statements within the identified domains that were considered the most important by both clients and professionals. To this end, an overview of the domains including the underlying concepts from each of the concept maps and the matching statements was compiled; the statements were then categorised by their priority score. When generating items for the assessment instrument, a differentiation was made between statements from the concept map sessions with importance scores from 3.5 to 4, 4 to 4.5 and 4.5 to 5. All statements that were scored as 4.5 or higher were included in the preliminary item list. The statements were rewritten into items for the assessment instrument.
The third step was the construction of the assessment instrument. Visual Analogue Scales (VAS) were chosen as the response scales, ranging from totally disagree to totally agree and from very unimportant to very important. VAS scales were used because of their relative ease of use and their sensitivity to small differences in scores [
44‐
46]. Participants were asked to rate their degree of agreement with each item as well as their perception of the importance of each item.
All included items were formulated for a client version and a professional version of the assessment instrument. An introduction to the assessment instrument and an open-ended question that asked the participant if the assessment instrument lacked important details of the working alliance were also added. The items were randomised to prevent items from the same domain being scored in a row to avoid influencing scores because of priming or bias.
In the final step, the assessment instrument was reviewed by the second and third author and a registered nurse who was not involved in this study.
Stage 4: pilot testing & cognitive interviews
The clients and professionals that participated in the concept mapping sessions and clients who volunteered for the concept mapping sessions but were unable to attend were contacted through email and invited to participate in testing the assessment instrument. Six professionals attended both the concept mapping session and the pilot testing. Five clients participated in the pilot testing, including two former clients who did not participate in the concept mapping sessions. Two gift certificates (€10) were raffled among the participants (one in each group). Participants differed with respect to gender, age, level of education and years of experience in/with (F) ACT teams (Table
2). Participants were interviewed by the first author following a think-aloud procedure in which they were asked to state all their thoughts while reading and responding to the instrument. Advantages of cognitive interviews with a think-aloud procedure over other forms of testing questions such as expert review and behaviour coding are its usefulness in identifying problems with questions and its ability to explicitly assess the participants ability to comprehend, recall, judge and respond to the questions. Also, this method is particularly helpful in assessing ambiguities that come up during the cognitive process of answering a question [
47,
48]. Part of the instruction to participants was as follows: ‘I want to ask you to read this questionnaire and verbalize your thoughts out loud. I’m not necessarily interested in your answers, but I would like to understand how you get to your answer and which problems and inconsistencies you come across in your thought process. [ …] This is not a test of your skills, but of the questionnaire. You can be open in your critique of the questionnaire; I want to know if there is anything wrong with the questions and if so, what. There are no right or wrong answers’. Following the instructions, the interviews started with a small exercise in which participants were asked to visualize their house and recount what they see to the interviewer.
Table 2
Characteristics of the cognitive interview participants
Gender | 4 females, 2 males |
Age, mean (range) | 42.5 years (38–49 years) |
Profession | Clinical psychologist/psychotherapist (2) |
Mental health care psychologist (1) |
Casemanager (1) |
Social psychiatric nurse (2) |
Mean number of years of experience working in (F) ACT teams (range) | 6.8 years (3–10 years) |
Clients and former clients (N = 5) |
Gender | 3 males, 2 females |
Age, mean (range) | 43.4 years (34–56 years) |
Level of education Living situation | 4 secondary vocational education, 1 BSc 5 independent living |
Employment status | 2 daily occupation services*, 2 no work or daily occupation services, 1 paid work |
Mean number of years receiving care from (F) ACT teams (range) | 5.4 years (1.5–11 years) |
Participants were asked about several topics related to the assessment instrument including the introduction, individual items, scoring of the items, and the use of the VAS rating scale. All interviews were recorded and additional notes were made by the first author. The recordings were transcribed and analysed by the first and third author to reveal general strategies for answering the questions and difficulties with particular questions. All feedback from the interviews was summarized for each item and items were subsequently adjusted in response to the feedback in consultation with the third author.