Development and psychometric properties of the client’s assessment of treatment scale for supported accommodation (CAT-SA)

As recommended by Cronbach [35] any statistical assessment of internal consistency should be preceded with a logical analysis of the item content and method of administration, addressing any issues that could potentially invalidate the instrument. To assess the suitability of content for supported accommodation, we conducted focus groups with service users to review the seven CAT question items and the 11-point visual analogue response format [28]. We also explicitly asked service users whether they thought any additional items should be added. Service users for the focus groups were recruited in North London from the three main supported accommodation service types (i.e. residential care, supported housing and floating outreach). In addition, members from the North London Service User Research Group were invited to participate.

One researcher (SS or JK) facilitated the focus groups, with a second researcher taking notes (JK or PM). In addition to the in situ annotations, all focus groups were audio recorded and further details were extracted from these recordings and added to the annotations to ensure all comments were captured. The facilitator canvassed opinions and suggestions about the structure, terminology and suitability for use in supported accommodation services for each of the seven CAT items. Opinions were also sought on the response format and general appropriateness for use within supported accommodation services. The adapted CAT was named the CAT for Supported Accommodation (CAT-SA).

Following the item inspection and subsequent modification, the CAT-SA was administered to a sample of adults (aged 18 and over) with a primary mental health diagnosis who were receiving supported accommodation services. This data was collected as part of a national survey in England. Services were selected from a nationally representative sample of 14 Local Authorities (areas) across England. The sampling strategy for selecting these areas was based on an index score that factored in: local mental health morbidity, social deprivation, level of urbanicity, provision of community mental health care, provision of residential care, mental health care spend, and housing demand [36].

We aimed to recruit users from 90 mental health supported accommodation services, with approximately 30 services from each type (residential care, supported housing and floating outreach). Services were stratified by service type and area. Researchers attempted to recruit two services from each area-service type cluster, commencing with the area with the lowest index score (i.e. negative z-score with corresponding raw scores on mental health morbidity etc. below the mean) as derived from the area sampling strategy [36]. Where two services could not be recruited from a cluster, an attempt was made to recruit an additional service from the cluster with the closest index score. Where services agreed to participate, users within these services were block randomised and approached to participate in an interview survey where the CAT-SA was administered. We also assessed subjective quality of life on the MANSA [34], a widely used PROM in mental health care containing questions on satisfaction with various life domains (i.e. accommodation, finances, friends, family relations, leisure, mental health, people that service user lives with, overall life satisfaction, physical health, safety, sex life, and work). The focus on multiple domains of social life is relevant to some of the domains service users receive support with in supported accommodation, although it should be noted that the MANSA has a broader remit (i.e. domains related to personal relationships etc.). The MANSA is also one of the most frequently used scales in recent years for assessing subjective quality of life in adults with schizophrenia. It has also been recognised as suitable for generic use across disorders [37, 38]. Information was also collected on the service user’s age, gender, length of stay and primary diagnosis as reported in case notes.

Comments and suggestions from all focus groups were collated into a summary document containing the consensus changes that appeared in more than one focus group. This summary document was then reviewed by the QuEST Programme Management Group and Service User Reference Group to gain final agreement for any suggested changes.

For the survey data, if one or more items of the CAT-SA were not completed, listwise deletion procedures were applied to account for missing data prior to assessing for internal consistency. The internal consistency of the CAT-SA items was investigated using Cronbach’s alpha [39], with corrected item-total correlations used to assess for the interrelatedness of each item to the total mean. We used analysis of variance (Kruskal–Wallis test) and Mann Whitney U to inspect for differences in CAT-SA scores by service types, primary diagnosis, and gender. Spearman’s ranked correlation coefficients were used to assess for mean CAT-SA score relationships with age and length of stay in the service. For assessing the convergent validity with subjective quality of life, Spearman’s correlations were calculated with the total MANSA mean score. Additionally, we computed the correlation with the specific MANSA item on satisfaction with accommodation.

Four focus groups were conducted in total (n = 21). Three of these focus groups consisted of mental health supported accommodation services users in two North London boroughs. These focus groups were service-specific and each provided a different level of support. Service users forming the residential care focus group (n = 6) were living in a service staffed 24 h a day, seven days a week, with a high level of support available. The supported housing focus group (n = 5) comprised of service users in a service where staff were available most of the day for support needs, but clients had limited access to support in the evenings and overnight. Service users with the lowest level input from supported accommodation services were represented in the floating outreach focus groups (n = 5), where service users held their own tenancies and staff supported clients through personalised one-to-one visits in their own homes or local communities. The fourth focus group consisted of members from the North London Service User Research Forum (n = 5), which consisted of individuals with lived experience of mental health services, as well as some knowledge of research practices and procedures.

Consistent changes were recommended across two or more of these four focus groups. These recommended changes related to the terminology used across items. Where items contained the term ‘believe’, this was replaced with ‘think’. Where the term ‘treatment/care’ was used, this was changed to ‘support/care’. Similarly, changes were made to the use of ‘therapist/case manager/key-worker’ to ‘support worker/key worker’ to better reflect the role of the staff working in supported accommodation services. A minor change was also made to the wording of the item “Do you feel respected and regarded well?” to “Do you feel respected and well regarded?” Where recommendations for changes where mentioned in one focus group, by a signal member or two from that group, they referred to alternative terms for a more frequently mentioned substitute term. For example, two members of the supported housing focus group suggested replacing the term ‘believe’ with ‘feel’, as opposed to the more frequently suggested ‘think’, and one member of the floating outreach focus group recommended replacing ‘treatment’ with ‘aftercare’, as opposed to the more common suggestions of ‘support’ or ‘care’.

All seven items were regarded as relevant to assess the satisfaction and perceptions of the appropriateness of supported accommodation services. However, three of the four focus groups debated the appropriateness of the item “Do you believe you are receiving the right medication for you?” Some participants questioned the applicability of this item to all supported accommodations settings. Some participants argued that decisions about medication were more appropriate for hospital settings, while others insisted on its relevance because staff often supported service users to take their medication. Also, satisfaction with medication, side effects, and being able to pick up prescriptions easily from the general practitioner or pharmacist, can influence the way the client experiences their accommodation and the function of the service. This item was therefore retained. The number of items and visual analogue response format were deemed appropriate and accessible by all four focus groups therefore no changes were recommended. In addition, no recommendations were made for additional areas or items to be included in the scale. The list of items forming the CAT-SA is presented in Additional file 1.

Eighty-seven mental health supported accommodation services agreed to participate in the national survey, of which 22 were residential care, 35 were supported housing, and 30 were floating outreach services. Of the 619 clients who participated in the national survey, 618 completed the CAT-SA, of whom 159 were living in residential care services (26 %), 251 were living in supported housing services (40 %), and the remaining 208 were receiving floating outreach services in their own tenancies (34 %). See Table 1 for a breakdown of participant characteristics by service type.

Two-thirds of participants were male (n = 410). The participants ranged in age from 18 to 85, with a mean age of 46 (s.d. = 13.5). The majority defined their ethnicity as white British or white other (n = 499, 81 %), and reported their current relationship status as single, which included widowed, divorced and separated (n = 517, 92 %). The mean length of contact with mental health services was 17 years (s.d. 11.7). A primary diagnosis was available for 551 participants (89 %), of which approximately two-thirds had schizophrenia (n = 324) or schizoaffective disorder (n = 57). The remainder either had a diagnosis of depression or anxiety (n = 130), or bipolar affective disorder (n = 39), whilst only one participant was noted as having a personality disorder. These socio-demographics and clinical characteristics are largely in accordance with the findings from a previous national survey of supported accommodation providers in England [27].

Following listwise deletion procedures, Cronbach’s alpha for the seven items of the CAT-SA was 0.89 (n = 568). The internal consistency of the CAT-SA when an individual item was removed is presented in Table 2, along with corrected item-total correlations to assess for the interrelatedness of each item to the total mean.

No single item substantially decreased or increased the internal consistency of the CAT-SA if it was removed. The removal of item 4 (“Do you believe you are receiving the right medication for you?”) increased the internal consistency only slightly (α = 0.91). In terms of the interrelatedness of each item to the total, all of the corrected item-total correlations were between 0.72 and 0.78, with the exception of item 4 which was 0.44. Examination of the inter-item correlation matrix indicated that individual item correlations were all above 0.33, indicating a degree of relatedness above the recommended 0.3 cut off [40]. Mean averages for the CAT-SA items are presented in Table 3.

Mean scores were slightly higher for less supported settings, with the mean total score for floating outreach (8.09, s.d. 1.82) being slightly higher than that for supported housing (7.80, s.d. 1.91) and residential care (7.71, s.d. 1.92). Overall, the difference in CAT-SA scores by service type were not significant (Kruskal–Wallis K = 4.83, p = 0.09). The difference in means for the CAT-SA between residential care and supported housing services was not significant (Mann–Whitney U = 18941, z = −0.49, p = .63), but was significant between residential care and floating outreach services (U = 14220, z = −1.95 p = 0.05), and a trend towards significance between supported housing and floating outreach services (U = 23346, z = −1.80, p = .07).

There was no significant difference in the CAT-SA by gender (females 7.94, s.d. 2.01 vs. males 7.83, s.d. 1.82, U = 39179, z = −1.37, p = 0.17) or correlation with age (r _s  = 0.06, p = .13). However, there was a significant difference by diagnosis (U = 32566, z = −3.24, p ˂ .001), service users with a non-psychotic disorder rated their supported accommodation service more favourably on the CAT-SA (mean 8.13, s.d. 1.96) than those with a psychotic disorder (mean 7.75, s.d. 1.83). This significant difference could not be attributed to the medication item on the CAT-SA (item 4) alone as there was no significant difference between service users with (mean 7.44, s.d. 2.85) or without a psychotic disorder (mean 7.38, s.d. 2.77) on this item (U = 34773, z = −0.41, p = 0.68). There was no significant correlation between years of contact with mental health services and CAT-SA scores (r _s  = 0.001, p = .97).

As a test of convergent validity, high scores on the CAT-SA were positively correlated with subjective quality of life as measured by the total for the MANSA (r _s  = 0.35, p ˂ .001) and the satisfaction with accommodation item (r _s  = 0.37, p ˂ .001). However, when the accommodation item was excluded from the MANSA total, there was no significant correlation between subjective quality of life, as measured by the MANSA, and the CAT-SA (r _s  = 0.03, p = .47). Convergent validity was substantially due to the accommodation item alone.

The study produced a PROM (CAT-SA) for supported accommodation services that appears relevant, is easy to use and acceptable, and shows good psychometric properties. Focus groups with service users from the three main types of supported accommodation service in the UK (residential care, supported housing and floating outreach) aided the development of a measure that is brief and relevant to the users of these services. Each item of the CAT-SA was completed by no less than 592 of the 619 service users that participated in the survey, indicating high usability. When tested in a large nationally representative sample of services across England, the CAT-SA demonstrated good internal consistency and satisfactory convergent validity with self-reported satisfaction with accommodation.

There was near significant difference in CAT-SA scores between service types, and the direction tended towards higher scores in less supported settings, although service users of residential care scored the CAT-SA item on appropriateness of medication higher than those of the other two service settings. This reduced the interrelatedness of this item to the total CAT-SA score, and to the other items. Since not all users received medication, the response rate to the medication question was slightly lower than to the other items. However, all correlations still remained within reasonable confines for inferring internal consistency within the seven items [40]. Moreover, service users expressed the importance of this item to their experience of the service, and its inclusion did not compromise the overall good psychometric properties. This variation in response also had face validity and reflected a real difference in the function and role of the services in terms of monitoring and administering medication. For these reasons it was retained.

In line with previous surveys using the CAT, no differences or associations were found in scores on the CAT-SA by gender, age, or length of time in receipt of mental health services [30]. However, differences existed in CAT-SA scores by diagnosis, with service users with non-psychotic disorders rating their supported accommodation service more positively than those with a psychotic disorder.

A limitation of this study is the lack of available data on the service users’ symptoms in order to elucidate any potential associations between symptoms, in particular depressive symptoms, and the ratings on the CAT-SA.

Temporal changes in the CAT-SA were not measured as test-retest reliability and sensitivity to change have yet to be assessed. The CAT-SA scores also have a tendency towards ratings at the positive end (high scores), an issue that has been noted in a review of service user reported outcomes more generally [9]. Therefore, the CAT-SA is likely to have a lower ability to discriminate appraisals at the lower end than at the higher end of the scale, a common problem of PROMs in mental health care [9]. The CAT-SA’s appropriateness for use as a PROM in other countries with similar provisions for transitional mental health supported accommodation would require additional validation. However, given the international applicability of the original CAT, one might assume that the CAT-SA will also provide valid results in different countries.

Although it is important to note that the CAT was originally developed for use in inpatient setting, where all services and potential influences on treatment satisfaction are contained within a relatively closed environment. For supported accommodation services this in not necessarily the case. The variation in what constitutes supported accommodation (i.e. from residential care to floating outreach) can influence the amount of direct service input and contact hours a service user receives. In addition, the direct and indirect involvement of other support services readily available in the wider community may dilute the attribution that can be placed on the supported accommodation service itself. Therefore the CAT-SA, unlike the CAT, needs to be interpreted taking the lack of specificity inherent in community-based services into account. A potential solution to counter the lack of specificity when responding to items in the CAT-SA would be the inclusion of “here” or “in this service” in all items to focus the response to the support received from the supported accommodation service, as it currently is for the item “do you think the other elements of support/care here are right for you?” This may in part address the evident reduced relatedness of item 4 (the medication item) compared to the other items in the CAT-SA. The administering and monitoring of medication varies between service types, and item 4 could potentially be modified in future applications to reflect this variation. For example, the item could be reworded to specify support from the service (e.g.“do you think you are receiving the right support here with any medication you are taking?”), with an added option to select “not applicable where the service user has not been prescribed any medication”. This would refine responses to the support the service user is receiving within the service from any wider views held about the efficacy of the medication and those that prescribe and disperse it.

The CAT-SA was developed in a systematic and stepwise manner, with the involvement of supported accommodation service users. It produced a PROM that is simple to complete and valid for use in a large nationally representative sample, across three service settings attesting to its use in supported accommodation services in England. This provides a good basis for testing the CAT-SA in other samples and countries, including types of supported accommodation that differ from those available in England. The advantage of using a mean score for the overall CAT-SA, as opposed to the sum, allows for an item to be incomplete without affecting the overall score, by adjusted the mean to the six completed items instead for example. We recommend this flexibility when applying the CAT-SA in future research to ensure appropriate responding.

The anonymised data set on which this analysis is based can be requested from the corresponding author.