Developing a proxy version of the Adult social care outcome toolkit (ASCOT)

While recognising that proxy-report is not equivalent to self-report, the questionnaire development sought to draw upon the same principles and concepts as the ASCOT-SCT4 [10, 12]. The eight attributes captured by the ASCOT-SCT4 are: Control over daily life; Occupation (‘doing things I value and enjoy’); Social participation and involvement; Personal safety; Personal cleanliness and comfort; Food and drink; Accommodation cleanliness and comfort; and Dignity. The instrument has four response options for each item to distinguish between the ideal state, no needs, some needs and high-level needs in relation to each quality of life attribute [10] (see Table 1).

The key challenges of developing a proxy-report instrument were explored in an earlier phase of research, which involved a rapid literature review, focus groups and in-depth one-to-one interviews [21, 24]. In the English Adult Social Care Survey, the majority of proxy respondents are family or friends who live with the respondent or elsewhere and proxy-report by care staff is relatively infrequent (5.9% of proxy respondents in the 2014/15 ASCS) [29]. It should be recognised that some people may have no or limited contact with family or friends, so care staff may be called upon to act as proxy respondents, especially if they have frequent contact with individuals and in-depth knowledge of the person’s needs and preferences through their practice of care (for example, with support staff who work intensively one-to-one or with small groups of adults with learning disabilities). While some studies have found that proxy-reported quality of life varies by ‘type’ of proxy, for example, health or social care professionals compared to family [20, 30‐33], other studies have not found significant differences by type of proxy [34‐36]. There is also evidence that this may be accounted for by differences in proximity, intimacy and frequency of contact between the proxy and the individual rather than the ‘type’ of proxy per se [37, 38]. Therefore, in this study and also the earlier phase of research, we considered both care support staff and also family or friends as potential proxy respondents.

The key challenges in proxy response identified in this earlier phase of research were: (1) care workers’ resistance to the idea of acting as a proxy respondent; (2) the perceived difficulty of judging care recipients’ internal subjective state (i.e. how they ‘think’ and ‘feel’); and (3) proxy respondents’ wanted to express that their own response (as a proxy) differed from how they thought the care recipient would respond based on the proxy respondents’ judgement of the care recipients’ perspective [24]. Proxy respondents also commented that they found it difficult to differentiate between the top response options of the ‘ideal state’ and ‘no needs’ [24], whereas earlier work on self-report had found that the inclusion of the ‘ideal state’ supported comprehension and judgement of response [10] (see Table 1). Since the ‘ideal state’ represents that an individual’s needs and preferences are met to his/her desired level [10], it was hypothesised that this was a proxy-specific issue related to the proxy perspective adopted in formulating a response.

In response to the issues summarised above, the authors with advice from the ASCOT development team at the University of Kent (www.​pssru.​ac.​uk/​ascot), explored four significant variations in the format and content of the draft proxy version of the questionnaire. First, the questionnaire was formatted to allow the collection of proxy-ratings of quality of life for each attribute from the two proxy perspectives identified by Pickard and Knight [39]. The proxy-proxy perspective represents the proxy’s view based on their own preferences and values, while the proxy-patient perspective asks the proxy to answer based on the proxy’s best attempt at a reconstruction of the individual’s internal mental state based on their knowledge of the individual and their preferences. It was hypothesised that allowing proxy respondents to express both proxy perspectives may improve the acceptability of a proxy-report instrument, especially for paid carers. Furthermore, the specification of the proxy perspective within the questionnaire was anticipated to reduce the potential for bias that may arise if different types of proxy respondents (i.e. paid or unpaid carers), without clear instructions as to which proxy response to use, systematically refer to different proxy perspectives in order to formulate their response to the questionnaire.

The second modification was to include a comments box after each question. In the first phase of work, care staff reported that they would feel more comfortable with acting as a proxy if they had the opportunity to add further detail or explanation of their responses, especially if quality of life had been rated as some or high-level needs [24]. The inclusion of the comments boxes aimed to improve the acceptability of the questionnaire, especially for care workers.

Third, during the development of the original ASCOT-SCT4, it was found that some respondents understood the Dignity item to refer to the impact of having help on their self-esteem and self-perception, rather than the effect of the way they are supported or treated by care staff [10]. Based on this evidence, an additional item was developed for the ASCOT-SCT4 to allow respondents to express any difficulties they have with coming to terms with needing help [10]. This item is not considered in scoring the ASCOT-SCT4. In the development of the proxy-report questionnaire, by contrast, it was initially decided to include only one Dignity item because of the additional complexity of asking respondents to adopt two different proxy perspectives.

Finally, based on the proxy respondent’s report that they found it difficult to judge how the individual thinks or feels [24], the wording of some items was modified in the draft questionnaire to be more ‘objective’ (version 1.0): for example, Occupation was revised to refer to what the person ‘wants to do’ rather than ‘values and enjoys’, Social participation referred to any type of social activity rather than specifically to social interaction with ‘people s/he likes’ and the items for Personal safety and Personal comfort and cleanliness were modified to refer to ‘being’ rather than ‘feeling’ safe or clean and comfortable. In the first phase of interviews with the version 1.0 questionnaire, we sought to explore whether the use of more ‘objective’ criteria would improve the acceptability of the proxy-report questionnaire. The research team were, however, also aware that these modifications were in tension with the underlying conceptual basis of the ASCOT measure to capture social care-related quality of life with reference to the individual’s preferences, values and attitudes.

A summary of the process of the initial development and refinement of the questionnaire is shown in Fig. 1.

Participants were recruited purposively via three social care provider organisations across three local authorities in South East England. The study inclusion criteria were adult care workers or unpaid family carers (aged 18 years or over) in regular contact with an adult who uses social care services and who would be unable to answer the ASCOT-SCT4 on his/her own behalf due to cognitive and/or communication impairments. Potential participants were sent an invitation letter and information sheet by the care provider, which explained the purpose of the research. Those who were interested in participating were invited to contact the research team to arrange an interview and to answer any questions about the purpose or nature of the research. A total of 25 proxy respondents (13 care workers, 12 family carers) were recruited and completed a cognitive interview.

The study was approved by the national social care research ethics committee in England (reference: 13/IEC08/0020). Written informed consent was obtained from all participants prior to interview.

All interviews were conducted between June 2015 and May 2016 by three trained qualitative interviewers (SR, GC, JC), two of whom had prior experience of cognitive interviews in the development of self-report questionnaires (SR, JC). The interviews lasted between 40 and 75 min and took place at a location convenient for the respondent – usually, in a private office or meeting room in the workplace or at the respondent’s home. The proxy respondents were asked to complete the ASCOT questionnaire using a think-aloud method of cognitive interviewing with concurrent follow-up probes [26]. Think-aloud requires the respondent to speak aloud their thoughts as they read the instructions and complete each item of the questionnaire. Following this, the interviewers used verbal probing to further explore the comprehension, clarity and relevance of each item and its response options (for example, ‘why did you choose this response rather than the level above/below?’) [26] (see Table 2 for further examples). The probes were used flexibly by the interviewers to explore the respondents’ understanding and ability to respond to the questions, as well as to identify any potential issues with the acceptability of item wording, format or layout. For some items, alternative versions were presented to assess the proxy respondents’ preferences and to evaluate whether the different content or format would affect the respondent’s comprehension, judgement or response. After the cognitive interview, the respondents were asked to reflect on their overall experience of completing the questionnaire with a focus on clarity and the acceptability of answering the questionnaire as a proxy respondent (see Table 3).

The interviews were audio-recorded and transcribed for analysis. The written interview transcripts were analysed using NVivo version 10 software [40]. While it is possible to informally analyse data from cognitive interviews to identify and resolve issues with item wording, layout or format, the use of systematic thematic analysis may provide a more rigorous way of analysing think-aloud cognitive interviews ([26] p.157). The researchers agreed on an initial analytical coding structure based on the framework of cognitive interviews as a technique to explore the understanding and acceptability of questionnaire item wording, format and layout (see Table 4) ([26] pp.164–7). The data was coded by two researchers (SR, GC). The coding structure was iteratively refined based on themes and subthemes that emerged from the data, which allowed the identification of key issues and insights.

The questionnaire was refined through an iterative process of cognitive interviewing ([26] p.146). Each round of interviews had a minimum of six and a maximum of twelve participants, which is within the guidelines suggested by Willis [26] (p.138). The researchers met during the fieldwork to review the data, to discuss any emerging themes, and to agree on any modifications to the questionnaire.

After six interviews with questionnaire version 1.0, the data were analysed to identify any issues with comprehension, response or acceptability that would justify modification of the questionnaire at that point. The edits were applied to generate version 2.0 of the questionnaire, which was tested in the next round of interviews (n = 7). The second interim analysis considered whether any further changes were required based on the interview data. The questionnaire with these changes is referred to as version 3.0, which was then tested in the final set of interviews (n = 12). During this final set of interviews, the research team agreed that we had reached saturation (i.e. no new concepts or issues emerged). At this stage, the third and final analysis was conducted of this final set of interviews, both separately and also alongside the two earlier phases.

A summary of the iterative development process is shown in Fig. 1.

The demographic characteristics of the sample are shown in Table 5. Of the 25 respondents, 60% were female. The sample comprised care staff (n = 13) or family carers (n = 12). Half of the family carers responded on behalf of their spouse or partner (n = 6). The other carers responded on behalf of other family members (their adult children, parents or siblings). The majority of proxy respondents reported that the care recipient’s primary support reason was an intellectual disability (n = 9) or autism (n = 7). The remaining interviews were conducted with proxies who spoke on behalf of someone with dementia or another long-term condition that affected cognitive ability, comprehension or ability to respond.

With the iterative modifications outlined in detail below, the questionnaire items were found to be understandable and acceptable to care staff and family carers asked to act as proxy respondents. A summary of the findings in relation to the understanding of concepts and acceptability of each item to proxy respondents is presented in Table 6.

Based on analysis of the first round of interviews, the modifications outlined in Table 7 were applied to the draft questionnaire. For Occupation and Social participation, it was found that the wording of the original ASCOT-SCT4 item or response options were preferred to the version 1.0 proxy questionnaire because they were perceived to be more person-centred (“looks at the individual”). One formal carer, for example, compared the two versions of the Occupation item, ‘what s/he values and enjoys’ (original ASCOT-SCT4) to ‘what s/he wants to do’ (version 1.0), as follows:

“I think that’s worded better… because it’s talking about her values and her enjoyment… When we’re looking at the people we support we’re looking at their values and what they like to do, their likes and dislikes - so I just think it’s easier to answer”. [GC_FC_01]

Because of this, the items of Occupation, Social participation, Personal safety and Personal comfort and cleanliness were modified to correspond to the original ASCOT-SCT4 item wording to improve acceptability and comprehension (see Table 7).

Based on the earlier research that explored care staff and carers’ views on the ASCOT-SCT4 questionnaire, the version 1.0 proxy questionnaire sought to avoid the use of ‘enough’ because of potential issues with comprehension or acceptability [24]. The first round of interviews identified that respondents found the alternative wording used in version 1.0 (‘It’s ok’) to be too informal and imprecise:

“I just don’t like how that looks, just remove the ‘it’s ok’… it’s too vague”. [SR_FC_02]

The first round of interviews explored whether the adaptation of the Personal safety and Personal comfort and cleanliness items to be more ‘objective’ (i.e. being rather than feeling safe) would improve acceptability of the measure to proxy respondents [24]. Interestingly, despite the finding of the earlier phase of work that some respondents were uncomfortable with rating the cared-for person’s internal subjective state, the respondents spontaneously answered the questions by attempting to construct the cared-for person’s subjective perspective based on their observations and knowledge of the individual. This was prompted by the questionnaire format of asking respondents to rate the two different proxy perspectives (proxy-proxy and proxy-patient).

“It took me a little bit longer to answer this one I thought, I don’t know whether you thought exactly the same but because obviously I’m trying to think how this individual is thinking, and also I'm thinking about what she does, and is that because she doesn’t feel 100 percent safe, and what does safe mean to her as well”. [GC_FC_01]

Based on this finding, the Personal safety and Personal comfort and cleanliness were adapted to correspond to the original ASCOT-SCT4, which refers to feeling rather than being safe or clean and presentable. In subsequent rounds of cognitive interviews, the respondents were able to understand and respond to these two modified items with no evidence of any issues with acceptability.

Based on advice from the ASCOT development team, the researchers asked respondents in the second round of cognitive interviews to compare two versions of the questionnaire (see Table 8). Cognitive interviews with a carer version of the ASCOT, the ASCOT-Carer, had identified that some respondents were not able to understand the word ‘adequate’ used in some response options [41]. The aim of comparing the two versions in this study, which used either ‘adequate’ or ‘enough’ in the response options for six of the eight items (see Table 8), was to further test the acceptability and comprehension of ‘adequate’ and also to determine whether the alternative item wording (‘enough’) would affect comprehension, judgement or response in the context of a proxy-report questionnaire. For each of the six items, the respondents were asked to respond to the version using ‘adequate’ before being asked to review the version using ‘enough’. The interviewer probed to explore whether the respondent preferred one version to the other, and also whether the different wording affected the respondent’s comprehension, judgement or response.

There was no clear preference between the two versions. Two respondents showed a consistent preference for ‘adequate’ [GC_FC_05; SR_FC_03], while three respondents preferred ‘enough’ [GC_FC_04; GC_FC_06; SR_IC_01; SR_IC_02] and one respondent indicated a preference for ‘adequate’ or ‘enough’ that varied by item [SR_IC_03]. The findings indicate that ‘adequate’ was understood by some respondents to be an objective baseline related to social services’ or other external agencies’ standards, while ‘enough’ implied a subjective standard.

“If you’re social services… they’ve got very different opinions of what’s adequate to bring a child up in a house. Most of us would say definitely not, but because there’s levels, there’s only concerns on certain levels, isn’t there? So adequate makes me feel that that’s what it’s looking at, whether it’s adequate. Whereas this is more about--, if you’re asking my opinion then it’s whether it’s enough for me”. [SR_IC_01]

While the intention of the ASCOT tools is to ask respondents to rate quality of life from a subjective perspective rather than an objective standard, it was decided to use the term ‘adequate’ rather than ‘enough’ to maintain comparability between the ASCOT-SCT4 and the proxy-report version.

Pickard and Knight’s (2005) conceptual framework of proxy response was used to develop proxy questionnaire items that asked: (1) respondents to rate their own view (proxy-proxy perspective); and (2) what they thought the care recipient would say (proxy-patient perspective). This dual response approach was found to be acceptable to care worker and family carer proxies. The format allowed the respondents to express differences between their opinion and their understanding of the care recipient’s perspective. These differences were explained by reference to different personal preferences influenced, for example, by social or cultural norms or health condition-related factors:

“Her generation don’t do all this changing sheets every day malarkey and they don’t believe in so much washing. That’s just alien to them. And she used to say, “That’s fine, it doesn’t need a wash”. But to me it would be like reeking and I’d be like, “No, you must”. [SR_IC_10]

While the results suggest that the two proxy perspectives improved the acceptability of the questionnaire, some respondents were confused by the questionnaire layout with two columns that correspond to the two proxy perspectives (see Fig. 2). In the first two rounds of interviews, four of the thirteen proxy respondents answered the first question (Food and drink) by ticking multiple boxes in each column or writing yes/no to indicate whether each statement applied or not. They were, however, able to correctly complete the remainder of the questionnaire once they had realised that the format required one tick per column. Based on these findings, the questionnaire for version 3.0 was re-formatted to include ‘please tick one box’ above the response boxes (Fig. 2).

In the third round of interviews (n = 12), two family carers were initially confused by the layout. The interviewers had been briefed to allow the respondent to continue the questionnaire without prompts to highlight the questionnaire instructions to tick one box per column. In one of the two cases ([SR_IC_05]), the respondent spontaneously realised that their initial response with multiple ticks had been incorrect and amended their response. There were no issues with the remaining questions. In the other case ([SR_IC_07]), the respondent directly asked the interviewer whether they needed to tick multiple boxes in each column. Once they had seen the instruction, with a prompt from the interviewer, the respondent understood that only one tick per column was required to indicate their rating:

“Tick, ‘Please tick one box for each,’ oh, ‘Tick one box for each column.’ Oh I see, so [sigh], so it’s just the top one then isn’t it?” [SR_IC_07]

The ASCOT Dignity item aims to capture the positive and negative psychological effects of formal support and care on the service user’s personal sense of significance [10]. In this study, it was found that the Dignity item was generally understood to relate to the effect of care workers' working practices and interpersonal interactions on the cared-for person’s sense-of-self. The interviews often focussed on interpersonal interactions associated with personal care:

“I would say dignity is like--, if someone is having a bath and you’re bathing them, like for instance they get out of the bath you turn away or you give them towels so they can hide themselves so they’ve, you know, privacy”. [SR_FC_02]

Other respondents, however, reflected on the wider implications of care practice and the quality of interpersonal interactions with care staff on the individual’s self-perception and dignity:

“I would say the quality of care is what results in dignity and respect… If the person who is going in to support them ultimately has that person’s--, as top most priority their needs, they will always think better of themselves at the end of that”. [JC_FC_03]

In one interview with a family carer, although the carer initially answered the question in relation to paid care, the boundary between the support given by family and paid carers became blurred:

“I would probably tick the top one and say he thinks and feels better about himself because we always do, we always praise him up. We always tell him he looks good”. [SR_IC_03]

As observed in the development of the ASCOT-SCT4 [10], although less frequently, three proxy respondents (care worker (1), family carer (2)) spoke of the effect of needing help on the individual’s sense-of-self.

“Have I ever met anyone who has been quite positive and upbeat? I don’t think I ever have. I think if I did they most probably wouldn’t want our support … it’s the very act of receiving care that may make them feel undermined”. [JC_FC_03]

Importantly, however, the ‘think aloud’ indicated that the three proxy respondents made their final judgement and response based on the way in which the care recipient was treated by care workers, rather than the effect of needing help on the individual’s sense-of-self.