Public involvement in health outcomes research: lessons learnt from the development of the recovering quality of life (ReQoL) measures

ReQoL was developed by a core team of seven academics and a scientific group (which included the core team) comprising seven expert service users, five clinicians, five academics and two clinical academics; these were the main decision-making bodies. They were supported by four advisory groups who provided opinions and recommendations at different stages of the research: (1) The expert service user group included two expert service users from the scientific group, plus other five other expert service users. All were purposively chosen in a number of ways. First, the research team approached people in their existing networks; second, expert service users recommended other service users to the research team and third, one expert user responded to a request circulated through a mental health network. Some had an academic background and were familiar with PROM development, others had varying experiences of research. (2) The psychometrics group comprised six ‘psychometricians’ who are experts in the science of measurement and the development of outcome measures. (3) The stakeholder group included 32 policy-makers and clinicians, while (4) the advisory group consisted of 33 national and international academics. Figure 1 summarises the involvement of service users in the three development stages and the implementation and dissemination of ReQoL. The top and bottom parts of the figure outline the role of service users as research participants and PI respectively. Full details of the development of ReQoL are available elsewhere [18, 22, 23].

At the first meeting of the expert service user group the 122 items were explored in-depth systematically and additional items were suggested by the group increasing the number of items to 180. Among these 58 additional items, nine were completely new items about missing sub-themes (Table 1); 22 had been dropped at much earlier stages along the selection process; and 27 items were dropped from the previous pool. As a result of the addition of items at that stage, three extra core team meetings had to be scheduled to carefully consider these items and comments.

After a comparable exercise was subsequently undertaken by the scientific group, the items were then reduced to 101. The core team further reduced the number of items to 88 for use in the subsequent stage through a similar exercise guided by the Streiner and Norman criteria [24]. It was important to reduce the number of items to make the face and content validity stage practically manageable without imposing unnecessary burden to participants.

In order to test face and content validity of the reduced pool of 88 items, 40 individual interviews, four paired interviews and two focus groups (n = 11) were carried out, obtaining the views of 59 service user participants and 19 service user participants aged 16–18. Important issues emerged from the interviews concerning the perceived irrelevance, complexity and ambiguity of certain items. Potentially distressing and judgmental items were also highlighted [22] . Mid-way through data collection the three interviewers, in conjunction with the scientific group and the expert service user group, agreed to add 12 more items as a result of feedback from the study participants.

At the scientific group meeting, the feedback received from the study participants on each item was discussed. In some instances, there were conflicting views between the feedback received from expert service users in the previous stage and that received from study participants. One example of disagreement concerned the item ‘I felt guilty’ which the expert service users found to be an important item. Study participants felt that at times it could be a positive thing to feel guilty in some circumstances (for instance, one is well enough to appreciate what one might have done when experiencing a serious episode), whereas in other circumstances it could be a negative experience of being too critical towards oneself. This item was discussed and it was agreed to drop it because the item not only enhances quality of life but also takes away from it. At this stage it was necessary to review the feedback from expert service users from the previous stage with the new evidence from study participants. In cases where there were disagreements (n = 20 items), these were highlighted by the qualitative researchers in advance of the meeting and more time was devoted to discussing such items to reach consensus on whether the item should be omitted, retained or re-worded. Therefore, as well as service users as participants, in terms of PI, an expert service user was also involved in collecting data and all the expert service users were involved in re-shaping the interview topic guide. Thus, the contribution of the expert service users was not only in terms of the contribution to the wording of items but to the underlying conceptualisation of the scale.

During the final stage of development after the second quantitative study, the expert service user group met separately before joining the scientific group later the same day to examine all the data and to finalise the short form (ReQoL-10) and the longer version (ReQoL-20). The combined group considered which items were most appropriate for each domain, and agreed that no additional items were needed. The expert service users contributed to the final item selection, and while this group was happy with the short ReQoL measure to contain 10 items, clinicians were of the view that six items would be sufficient. This was debated and the group agreed that 10 items offered better psychometric properties than six items. Because of the simplicity of the items of the ReQoL, the additional burden of four questions was minimal. It was also decided that the physical health item should be included in both versions of the PROM. As shown in Table 2, the result of inputs from expert service users in the decision-making process meant that the items with the strongest psychometric properties were not automatically chosen for the final measure. Instead, a compromise was reached between psychometric strength and content validity.

A short video describing the ReQoL project was co-developed prior to the launch of the PROM (http://​www.​reqol.​org.​uk/​p/​overview.​html). Expert service users helped to devise an information sheet about ReQoL, and they also attended the launch event during which barriers and facilitators surrounding the use of ReQoL were discussed. Furthermore, members of the scientific group met to discuss the possibilities of translating the PROM into different languages. Finally, expert service users are co-authors of published papers (including this one) and conference presentations arising from developing ReQoL.

This paper gives an account of one of the few examples in the literature of PI at every stage of a PROM development: recruitment to studies, collecting data, interpretation and dissemination of the findings. The service user voice was heard not only from the data sources (items from existing outcome measures; qualitative interviews; face and content validity testing; and psychometric testing), but also by expert service users being actively involved in decision-making regarding the domains and the items of the PROM through their membership in the scientific group and the expert service user group. Crucially, expert service users were key collaborators in the design of the PROM. The following section presents the value added and key issues of embedding PI in the project.