Best practice framework for Patient and Public Involvement (PPI) in collaborative data analysis of qualitative mental health research: methodology development and refinement

A critical literature review [40] was conducted by HJ, involving evaluation and synthesis of included papers. The aim of a critical review is to identify the most significant articles in a field, and it was chosen over more systematic approaches as a proportionate review approach suitable for generating a model. Databases searched were AMED, PubMed, CINAHL, MEDLINE and PsycARTICLES, with serendipitous searching [41] to identify grey literature. A scoping search indicated that terms like ‘coproduction’ were insignificantly specific, so search terms specifically focussed on the collaborative analysis element of coproduction were used. Inclusion criteria were (a) primary focus on qualitative collaborative data analysis; (b) published in journal or book; (c) focused on mental health research; (d) aligned with the principles of PPI and a democratic approach to public engagement [7]; and (e) published since 2007 to maximise relevance to current PPI frameworks. The initial concept search strategy comprised (“collaborative data analysis” OR “interpretation workshop” OR “participatory action research”) AND (“service user” OR “public patient involvement” OR “PPI” OR “co-researcher” OR “co-investigator” OR “expert by experience” OR “client” OR “consumer” OR “survivor”) AND (“mental health” OR “mental illness” OR “mental distress” OR “psychological distress”) AND “qualitative”. The initial search identified few papers specific to mental health, so inclusion criterion (c) was amended to include any health research, and the search strategy amended accordingly. The first author extracted data relevant to design and procedure, and characteristics of successful CDA. Findings were synthesised by the research team, and organised into methodology options and success characteristics.

The Stage 1 synthesis was considered by academic researchers and PPI co-researchers to select a methodology for CDA in RECOLLECT, in relation to investigation about mechanisms, outcomes and change models for Recovery Colleges. Evaluation criteria were that the methodology: ensured PPI co-researcher interpretations of qualitative data collected in RECOLLECT were generated; ensured these interpretations were used to inform findings; aligned with a democratic approach to PPI [7]; were practical to apply within the constraints of a one-year study; and incorporated the design techniques that were associated with successful CDA. Decisions were amalgamated by HJ into a final synthesised methodology and procedural session plans, which were then refined by the other authors. The CDA methodology was then piloted. PPI co-researchers met with the academic researchers in four 4-h meetings, each with a minimum of 8 co-researchers present who were paid for their preparation and attendance.

The aim of this part of RECOLLECT was to develop an understanding of how RCs work (mechanisms of action) and the impact on students (outcomes). Framework analysis [42] was the underpinning methodology. Summarising the content of CDA meetings (described in Table 3): Meeting 1 ensured the ecological validity of the document analysis carried out by the academic researchers; Meeting 2 refined the preliminary, academic researcher-developed coding framework to co-produce the final coding framework; Meeting 3 created a mental health service user-led change model (organising and linking coded mechanisms of action and outcomes); and Meeting 4 enabled final verification of the change model. Each meeting was facilitated by two RECOLLECT academic researchers; one to help navigate the narrative of the discussions and the other to support group dynamics and ensure all PPI co-researchers were able to contribute. Tasks were predominantly broken down into small group or pair work initially, with all PPI co-researchers then coming together to share what they had produced. Exercises were presented both verbally and on paper, with opportunities for questions and challenge. Adequate understanding of task requirements was ascertained through check-back and observation. Insights from occupational therapy research [43, 44] were used to inform grading and adapting activities to align with an individual’s capacities, in order to maximise their performance. This improves the likelihood the PPI co-researcher experience will be positive, rather than frustrating, marginalising or demoralising [45]. Exercises were paced to allow time for breaks and refreshment. Data interpretations by PPI co-researchers were captured by two academic researchers observing and writing down PPI co-researcher verbal contributions, collection of written and photographed outputs, and completion of field notes by the academic researchers.

Several strategies to enhance quality were used, such as peer examination, triangulation of researchers and PPI co-researcher perspective, triangulation of data sources (PPI, documents, qualitative interviews), and use of reflexivity in the academic researcher field notes [46]. Additionally, in the final meeting informal feedback was obtained and the Quality Involvement Questionnaire was completed by PPI co-researchers [47]. This 31-item self-report measure (each item rated 0 (low involvement) to 4) assesses perceived involvement, and has six sub-scales in two parts: Personal Factors (Your ability (range 0–28), Your potential (0–20), Your sense of being (0–20)) and Research Contexts (Research relationships (0–24), Ways of doing research (0–16) and Research structures (0–16)). Total score ranges from 0 (low involvement) to 124. As normative data are not available, it was also completed by the academic researchers to provide a comparison group.

The most useful components of the methodology were synthesised by the academic researchers to produce best practice, based on data collected during Stage 2.

The critical review identified 10 publications for inclusion, shown in Table 1 [30‐33, 48‐53].

One publication was a methodology chapter [31], and the others were empirical studies using qualitative methodologies: thematic analysis [30, 33, 49, 52], qualitative content analysis [33, 48], interpretative phenomenological analysis [51], grounded theory [32] and framework analysis [48, 50, 53]. There was strong consensus across all studies that including co-researchers with lived experience in qualitative research data analysis produced richer, more in-depth and alternative understandings of the material that the academic researchers could not have developed on their own. Four methodological approaches to involving PPI co-researchers in data analysis were identified.

Approaches 1 and 2 were integrated for use in RECOLLECT, as shown in Table 3. As some of the academic researchers were people with dual identities as researchers and with lived experience, concerns around tokenism in Approach 1 were less relevant. Combining this with Approach 2 allowed more meaningful involvement in understanding the data. The intention was that this combination would offer an effective way of conducting CDA within the time and resource boundaries of the study. Approach 3 was not used as it was least collaborative, and Approach 4 was not feasible in a one-year study. The authors incorporated success characteristics shown in Table 2 into planning of the CDA meetings. The CDA focus and the content of each meeting are detailed below.

Informal feedback was broadly positive, with one suggestion to use more action-focussed, concrete and illustrative tasks, rather than giving too much information about theory. For example, demonstrating the use of string and self-adhesive arrows to show a causal relationship between a mechanism card and an outcome card if the PPI co-researcher believes the mechanism produces the outcome.

Two process findings emerged from conducting the Stage 2 CDA, and from reflections from the involved academic researchers and PPI co-researchers. First, academic researcher facilitators felt that more time was needed for PPI co-researchers to complete the CDA tasks, especially regarding areas of ambiguity and achieving rationalisation and reconciliation in the analyses. Second, a balance was needed between over-planning sessions (risking stifling creativity and encouraging PPI co-researcher passivity) and carefully planning the structure and content of sessions so as to generate the deepest and most collaborative analysis. On balance, we concluded that CDA sessions need to be more structured and formalised than general PPI advisory meetings.

Based on this pilot, and for studies with limited time and financial/human resources (i.e. most funded studies, where compromises between quality and pragmatism are required), we identify a three-stage approach to CDA: Preparation, CDA and Application. Figure 1 shows a proposed best practice framework using these three stages to co-produce a qualitative coding framework.

In RECOLLECT we also used these stages to develop a service user-led model of change, developed by the PPI co-researchers and based on the co-produced coding framework. The Preparation phase was unchanged. In the CDA phase the small group work focussed on how model components (from the co-produced coding framework) relate to each other, using active and practical methods (e.g. laminated cards, sticky tape, string, self-adhesive arrows) to facilitate creativity and easy adjustments. In the Application phase a formatted version of the change model was validated with PPI co-researchers, then verified by the academic researchers with new research participants before being finalised with PPI co-researchers.

The National Involvement Partnership has developed the 4Pi national involvement standards: principles (respect, equality); purpose (potential and limits of change), presence (at all decision-making levels), process (engagement, communication, support, practical issues) and impact (ethos/culture, policy/practice, delivery, outputs/outcomes, diversity and equality of opportunity, PPI experience) [54]. Similarly, INVOLVE published a PPI values and principles framework emphasising respect, support, transparency, responsiveness, fairness of opportunity and accountability [55]. These principles informed RECOLLECT, although the aim of this study was to incorporate these principles whilst recognising the challenges of time-limited studies. If resources were less limited, desirable extensions would be involvement of PPI co-researchers in all elements of the data analysis, training and supporting PPI co-researchers to undertake the integration activities, and involvement (e.g. through online consultation) of a much larger heterogeneous and more representative PPI group. The decision to have the academic researchers develop the preliminary framework was made because the challenges of having PPI co-researchers undertake primary data coding using qualitative analysis software were deemed too complex. The potential bias introduced by this decision in the overarching interpretation could have been reduced by involving a sub-group of PPI co-researchers at the data coding stage.

The review identified only ten studies of PPI in analysis of qualitative data in mental health research. Despite the increasing PPI literature, empirical studies on involvement at this critical research stage of making sense of qualitative data were relatively few, necessitating extension of the search strategy to include non-mental health research studies. There is more published research about PPI in other research stages, such as the early stage of prioritising important research questions [56, 57]. Established methodologies now exist, such as James Lind Alliance priority setting partnerships [58]. PPI impact assessment studies consistently recommend that PPI needs to occur throughout all research stages. For example, in relation to PPI in quality improvement, guidance concludes ‘To be effective, PPI should run through the full cycle of every quality improvement project, as an integral part of the fabric’ [59] (p. 26). The absence of methodologies for collaborative data analysis is an important knowledge gap.

Four approaches to CDA were identified. Although they broadly move from lower (Approach 1) to higher (Approach 4) involvement, they cannot be simplistically allocated to a level of involvement. For example, in RECOLLECT the academic researchers included people with a range of lived experience, several study applicants self-identified as having lived experience, and the PI has published and been influenced by PPI impact assessment studies [60, 61]. Similarly, the PPI group had members with a range of research experience. A particular challenge was for academic researchers to avoid viewing PPI co-researchers as ‘just’ bringing a lived experience perspective – the strategic essentialism issue of reifying ‘people with lived experience’ as an undisputed and cohesive category [62]. PPI co-researchers cannot represent the full range of lived experience, and they all bring lived experience as one identity component among many.

Both collaborating parties therefore included members with hybrid identities [63]. Given this complexity, even a consultation approach may be effective, depending on the relational context, which from a PPI perspective is central [64]. The experience of PPI co-researchers will be influenced by whether they trust the academic researchers, whether they perceive the academic researchers really do want their views, even if challenging, and whether changes happen in response to PPI input. The best practice framework adheres to democratic principles, by involving people within research related to their health needs to produce a mutually agreed interpretation of the data [27]. Collaboration was increased by the involvement of people with lived experience as academic researchers, the ongoing relationship between the academic researchers and the PPI group (allowing trusting relationships to form), and repeated contribution verification and incorporation of PPI input [2].

Qualitative paradigms encourage researchers to focus on what being human is like, how people understand their worlds and the ‘lived experience’ [65]. Including PPI co-researchers in analysis supports this goal. Evidence from our pilot confirms that PPI co-researchers interpret data through an overlapping but different lens from academic researchers, so are a resource for highlighting assumptions and reducing interpretive bias [31, 32]. For example, in RECOLLECT the PPI co-researchers proposed that one particular mechanism of change (related to environment) was a precursor to all others in the change model, which was a new – and incorporated – proposal to the academic researchers. This may indicate that a deep understanding of the importance of the social, relational and environment is more easily available to PPI co-researchers than to academic researchers. As a second example, the preliminary framework included a code relating to identity change, but PPI co-researchers highlighted the implicit though unintended implication of ‘moving from bad to good identity’. This may reflect encultured clinical research beliefs about improvement meaning transition from sickness to health. Again, the framework was changed to reflect this insight, and researchers also grew in self-awareness about their own values and assumptions. Overall, our findings were consistent with existing evidence [4] that PPI involvement in collaborative data analysis improves quality of the findings and beneficially impacts on academic researchers. The same finding emerges from ‘inclusive research’ – involving people with intellectual disabilities in research – that acknowledging differences and uniqueness enriches research outcomes and supports reflective practice by researchers [66].

An average of four to five days of academic researcher time was spent in planning each meeting, which involved ensuring that meeting tasks created a ‘just right’ activity challenge [37], developing training approaches that incorporated andragogic principles [67, 68], and developing supporting materials and pre-circulating these to PPI co-researchers for absorption, comment and refinement. Along with the involvement of other academic researchers (n = 4) and PPI co-researchers (n = 9) in the four whole-day meetings, overall around 20 person-days were used per meeting, i.e. around 80 person-days extra time was involved in collaborative rather than academic researcher-led data analysis. This reinforces previous warnings not to underestimate the scale of resources involved in CDA [31], and is necessary given the challenges of collaborating on complex mental health topics [69].

CDA studies identified in the review did not emphasise the importance of adjusting tasks to build on PPI co-researcher strengths and reduce the impact of needs, other than practical suggestions such as pre-circulating accessible material, allowing time for tasks, not overwhelming PPI co-researchers with data, and using aids like post-its [30, 48‐50].

The goal in RECOLLECT was for the collaborative data analysis to inform the interpretation of the findings. There was little in the published literature about how to measure the extent to which this specific goal was met, beyond the recommendation of verifying findings with PPI co-researchers [48] and obtaining feedback [2, 7]. Available guidance is more general, e.g. PPI reporting guidelines [70] or ethical frameworks [25]. There is also a considerable literature on overarching frameworks for involvement in research, both relating to psychosis [71] and wider frameworks such as community-based participatory research [72], which focus more on principles and examples than on developing a best practice framework as in this study. Future research could: (a) develop a standardised measure of CDA impact (including both costs and benefits); (b) refine through piloting and then manualise the best practice framework, including resources such as relevant warm-up tasks and training exercises and potentially incorporating elements from CDA approaches 3 and 4; and (c) conduct a PPI impact assessment to evaluate the impact of CDA on study findings. This would inform decision-making in studies about whether the benefits of collaborative data outweigh the costs.

This study has developed the first theory-based and piloted best practice framework for collaborative analysis of qualitative data. Strengths include the use of a wide range of health research evidence to inform the framework, the significant investment of human resources, the repeated respondent validation, and the collection of formal and informal feedback demonstrating active involvement by PPI co-researchers.

The findings could be strengthened by conducting a systematised rather than critical review [40], with a stronger focus on obtaining grey literature, use of broader search terms (e.g. coproduction, co-design, emancipatory research, inclusive research, partnership research) and a more formal deductive analysis approach. More consistent involvement of a sub-group of PPI co-researchers in planning the meetings would have increased co-production in the process, and might for example have led to a more structured assessment of accommodations needed by individual PPI co-researchers to allow them to fully contribute. Further piloting, especially in studies which are not as positive towards the use of lived experience as RECOLLECT, would helpfully refine and extend the best practice framework.