Description of interviewees
We approached 26 researchers and interviewed the 18 who agreed. These 18 interviewees came from three sources in our wider study: they were authors of journal articles reporting qualitative research undertaken with trials (n = 9); they were applicants on studies reporting the use of qualitative research with trials which we identified on a trials database (n = 6); and they were applicants on studies which used qualitative research with trials but did not report this on a trials database, identified through a survey of researchers listed on a trials database (n = 3).
As described earlier, we undertook purposive sampling of two types of researcher working on these sampled studies: lead investigators (who were likely to be leading the trials) and qualitative researchers. Of the 18 interviewees, 11 led the trial, 7 led or undertook the qualitative research, and 2 led both the trial and qualitative research.
We offer further description of the sample so that readers can understand the characteristics of the sample. The 18 interviewees were male (n = 5) and female (n = 13). The 18 interviewees described themselves as a quantitative researcher (n = 4), qualitative researcher (n = 5), mixed-methods researcher (n = 1), lead investigator (n = 6), and both lead investigator and qualitative researcher (n = 2). Some interviewees had experience of undertaking qualitative research to address more than one aspect of trials identified in our wider study[
3]: they used qualitative research to focus on the intervention (n = 11), the trial conduct (n = 6), outcomes (n = 1), measures (n = 1), and the health problem under study in the trial (n = 2). The studies we selected them from were funded from a range of sources, particularly the National Institute for Health Research and the UK Medical Research Council.
The value of the qualitative research
Interviewees described the direct utility of qualitative research for the specific trial. We devised labels for the different types of utility they described and display and explain them in Table
1. Interviewees’ views of the value of the qualitative research often focused on value for the trial and were similar to descriptions in the literature on the contribution of qualitative research to trials, including identifying problems at the feasibility or pilot stage of a trial to prevent them occurring at the full trial stage (problem solver), helping to explain the trial results (explainer), and helping research users to understand the relevance of the trial findings in different contexts (translator). Interviewees also identified a benefit for the specific trial that had not been intended when they had started their study - engaging stakeholders important to delivering the trial and thus ensuring the successful completion of the trial (engager). However, for some interviewees the value of the qualitative research lay beyond the specific trial, impacting on future trials by informing the research team’s future practice in developing and evaluating complex interventions, allowing junior researchers the opportunity to obtain academic qualifications (trainer), and offering insights into the patient experience that had the potential to improve health service delivery generally (knowledge generator).
Table 1
Value of using qualitative research with trials
Problem solver | Identifying problems and solutions prior to the full trial by exploring trial feasibility, informing the intervention, developing data collection instruments. | Saving the effort of undertaking trials which prove to be unfeasible, making a full trial viable by ensuring its ability to recruit or retain participants, optimising interventions so that an expensive trial is undertaken of the best intervention. |
Explainer | Explaining trial results which are null, disappointing, surprising or confusing. | Offering complementary findings which supplement or modify the conclusion of the trial. |
Translator | Understanding the intervention implementation and context in the full trial in order to facilitate use of the evidence in the real world. | Increasing the utility of the evidence generated by the trial for changing practice in the real world. |
Engager | Securing stakeholder engagement so they have ‘buy in’ for the trial and remain enthusiastic about the trial. | Facilitating trial viability. |
Trainer | Offering an opportunity for trial managers to undertake PhDs or researchers to undertake dissertations for masters. | Increasing numbers of academically qualified and experienced researchers. |
| Helping trialists working in a specific field to develop understanding of a body of interventions. | Improving evaluations of interventions in the future. |
Knowledge generator | Identifying issues about the experiences of patients with different health conditions. | Improving the knowledge base about how to improve the quality of services for patients with different conditions. |
Some interviewees also described an unintended consequence - the potential of qualitative research to impact negatively on a trial by acting as an intervention and thus contaminating or damaging the experiment. Concern about the therapeutic effect of qualitative research existed where the interviews and observation were considered to be more intensive than the intervention under study:
‘particularly where the intervention you’re evaluating has got a psycho-social component, you do worry a little bit about interviewer effect … a therapeutic effect which can water down the impact of the actual intervention within the trial.’ (T17, qualitative researcher).
As well as specific benefits described in Table
1, interviewees valued the flexibility of the qualitative research to emergent situations - that it could be undertaken with a specific purpose in mind but end up offering insights about other issues. They described the need for qualitative research to evolve over time within a study rather than be constrained by the original research design, for example, in order to address difficulties that arose within the wider study. This flexibility and responsiveness was discussed as a strength and as quite different from qualitative research that was unplanned or not thought through. This flexibility was seen as complementary to the heavily protocolled approach taken to trials.
Three models of the relationship of the qualitative research to the trial
Within our sample we identified three models of the relationship of the qualitative research to the trial based on how interviewees described their studies. The first model we call ‘the peripheral’, where the intention was not to add value to the trial but to attain a value unrelated to the trial. For example, where the trial offered an opportunity to provide a higher degree for a researcher (‘trainer’ in Table
1), or to explore an area of interest to a qualitative researcher to facilitate understanding of patients’ experiences of a disease (‘knowledge generator’ in Table
1). For some of the 18 studies that interviewees discussed in detail, a number of qualitative components were undertaken. We categorised at least one qualitative component of three of these 18 studies as undertaken within a ‘peripheral’ model.
The second model we call ‘the add-on’, where the qualitative researcher believed in the value of the qualitative research to the trial but perceived that this belief was not shared by the lead investigator of the study or key team members. They described how the team viewed the trial as the central study component, with the qualitative research as an interesting but ultimately separate activity. The intention of the lead investigator or wider team was described by interviewees as not to add value to the trial but to generate knowledge that was complementary to the trial:
‘more quantitatively oriented trialists […] might be interested in the qualitative results in their own right, but I don’t know how interested they’re going to be in thinking about what it tells us about the trial. […] They provide a path of least resistance, if you want.’ (T11, qualitative researcher).
We categorized at least one qualitative component of four studies as ‘add-on’.
The third model was where the lead investigator viewed the qualitative research as integral, that is, essential to the evaluation because of uncertainties around the trial, the intervention, the outcome they wished to affect, or the patient group receiving the intervention. Indeed the term integral was used commonly by interviewees. These lead investigators described being driven by the need to undertake research which was applicable to the complex world in which health and healthcare operates and they could not conceive of undertaking a trial of the types of issues they were interested in without also using qualitative research. However, the practice of the study did not always match this intention. We categorised 14 qualitative components from our 18 studies as integral. There were two subgroups of this model: ‘integral in theory’ (7 qualitative components) and ‘integral-in-practice’ (7 qualitative components). We categorized studies as ‘integral in theory’ when interviewees described some added value of the qualitative research to the trial but that it was limited because the qualitative research was under-resourced, there was a lack of integration between the qualitative research and the trial:
‘it can be used in much more creative ways than it is being I think, or it has been doing, because I think there’s still this idea that it’s just something on its own whereas it can be integrated.’ (T1, lead investigator),
or the qualitative research was perceived as poor quality in terms of a lack of conceptual thinking or ‘intellectual intensity’:
‘I think one of the problems that we face with qualitative research is that it is talked up a lot, but quite often the product that is delivered isn’t nearly as good as the aspirations for the discipline […] I would say that most of it falls very far short of this sort of conceptual framework, real contribution to understanding that we think it will be, […] and you come out the other end and you think, ‘I don’t know if I’ve learnt anything from this’. And when you meet really good qualitative research, you know what […] we can achieve with it, but so often, I think, the intellectual intensity that’s required to do it well isn’t applied.’ (T8, lead investigator).
We categorized studies as ‘integral-in-practice’ when interviewees described the qualitative research impacting on the trial to the satisfaction of the interviewee by changing the outcome measures to be used or explaining the trial findings.
Issues important to ‘integral-in-practice’ qualitative research
Where we categorised the qualitative research as ‘integral-in-practice’, interviewees described resources which maximised the potential added value of qualitative research to the trial. These resources included senior qualitative expertise on the team from beginning to end, and staff and time dedicated to the qualitative research.
Qualitative researcher as ‘full’ team member from the start
Both the quality of the qualitative research and its integration with the trial could be affected by who was in the team from planning through to completion of the study. Where qualitative research was integral-in-practice the qualitative researcher was the principal investigator, a joint principal investigator with a quantitative colleague, or a co-applicant. The qualitative research was designed in the original study proposal with senior qualitative expertise on the team from the outset. This contrasted with the ‘add-on’ and ‘integral-in-theory’ models where the qualitative researcher could be junior and/or have been brought into the team once the trial had started. They might have access to a senior qualitative researcher in theory but in practice this person was too busy to offer them supervision. One interviewee had been the ‘added on’ junior team member in the past:
‘I was brought in to help finish off those interviews and then I led the focus groups […]. Because I was brought in at a late stage, and I basically came in, finished off a bit of a job and then wrote the sections for the report, but didn’t really bring it together, I’m not sure how the links were made between the two’ (T13, qualitative researcher).
On another study this interviewee felt themselves to be a ‘full’ team member from the beginning, which they felt enhanced the quality of research produced by ensuring that a better research proposal was written.
Where we categorised the qualitative research as ‘integral-in-practice’ the lead investigator had mixed-methods expertise or there was joint leadership between the quantitative and qualitative experts:
‘I was the PI and I’m a qualitative researcher, so it’s quite unusual in that most complex intervention trials are designed by the quantitative person and the qualitative researcher is sort of bought in.’ (T9, lead investigator and qualitative researcher).
This shared expertise at the top of the team was not necessary to ensure quality qualitative research and integration with the trial but the presence of senior qualitative researchers was reported as ensuring good quality qualitative research by facilitating depth analysis and write-up. This was described in contrast to an ‘add-on’ model seen in grant applications:
‘… there certainly are an awful lot of grant applications I see where it is quite clear that the little bit on process evaluation is just there as a bit of a token, maybe got a junior qualitative Research Assistant from a sociology department down the road to write a paragraph to go in the bid and then they cost it in for a few hours a week for the duration of the trial and clearly they are not an integral and powerful part of the study team’ (T18, lead investigator).
Time to undertake the qualitative research
Our ‘integral-in-practice’ studies had senior qualitative researchers as applicants with enough of their time funded to ensure good quality research was undertaken, as well as qualitative researchers to undertake the research rather than trial managers doing it alongside running the trial. This contrasted with ‘integral-in-theory’ and ‘add-on’ models where qualitative components were described as under resourced:
‘That’s the ideal situation - where the work is properly funded’ (T3, qualitative researcher).
Time, as well as staff, was an important resource, with enough time allocated to allow for an in-depth qualitative analysis:
‘…. allowing adequate time for analysis of qualitative data is important because it does take much longer than analysing quantitative so therefore there is a cost implication, so that’s the only thing that could be challenging when working with people who are from a more quantitative background is helping them understand the process of qualitative analysis and the time that it takes.’ (T17, qualitative researcher).
A negative case concerning the importance of resourcing the qualitative research was an interviewee describing a study which we categorized as an ‘integral-in-practice’ study as having had little money available for the qualitative research. However, the trial was also described as under-resourced. That is, there was not a disparity of resource with perceptions of under-resourced qualitative research and an adequately resourced trial. It was also the case that within the study they described there was senior qualitative research commitment and time invested in delivering the qualitative research in depth regardless of the funding available.
Investing time to communicate
In addition to the way teams were structured, the way they were described as operating on a day-to-day basis and how members communicated with each other was seen to contribute to the way qualitative research could add value to the trial. ‘Integral-in-theory’ and ‘integral-in-practice’ studies had integrative team practices. Where the qualitative research was an add-on, team closeness could develop over time, for example, as a colleague who was described as not ‘overly impressed’ with the qualitative research became more interested once the qualitative findings were reported (T10, qualitative researcher). Engaging the whole team and working in a collaborative way was described as facilitative to integrating different parts of the study. The use of meetings could encourage communication and reciprocal appreciation of the different work streams in the study:
‘… the main thing was openness of communication, fostering an environment that everybody's views counts and everybody’s methodology is on the same level, and about each member of the team facilitating qualitative or quantitative research to be done, at the time that it was required, so our team meetings were very open, we all knew we didn’t know everything, so we were very open about asking questions from each other, because none of us could cover the full spectrum that everybody else had, and, keeping those channels of communication open was seen to be very good.’ (T4, lead investigator).
The wider research environment can limit value
Interviewees discussed the wider research environment in which these mixed-methods evaluations were undertaken. Aspects of this wider environment could explain why qualitative research appeared to be peripheral, an add-on, or integral in theory. The wider environment could also affect ‘integral-in-practice’ studies in that the value of the qualitative research to the trial was not necessarily visible beyond the original research team because it was not reported in journal publications.
The drive to pursue academic careers
A qualitative researcher reflected on how trials could be vehicles for the career progression of some researchers:
‘They are not driven to find a treatment that is acceptable to patients, are they? They are essentially driven by the task of undertaking a highbrow randomized controlled trial, getting publications on it and moving on to the next big thing’ (T10, qualitative researcher).
Even researchers who spoke with passion about producing research evidence for use in the real world were driven by the need to be seen within their own institutions as someone doing ‘valuable research’. Valuable research was defined as producing articles that were published in high impact journals and if the qualitative research could not produce these then it was viewed as less valuable, or even worthless, within academia:
‘… if it’s not a three-star paper, which they define as something like obviously high impact and is top 10% of Web of Science categories, then it doesn’t count for anything at all, and you’re literally in their eyes wasting your time writing it. So, it’s difficult for me to justify the time writing up qualitative papers for minor journals when I could be writing high impact papers or getting more grant money in.’ (T5, lead investigator).
Pursuit of an academic career also included the need to bring in more funding and move on to the next project, resulting in teams breaking up before ‘non-priority’ papers were written, where the qualitative papers were described as the non-priority ones:
‘… our priority now, at the moment, is just to get the trial paper published. […] I’m being honest here, there is an issue of will here. Because we have learnt a lot by doing this, including by doing the qualitative research and we’ve moved on to new projects, and we’ve just started another massive project, and so the question is where is the appetite when we’ve moved on. Who in our team is going to develop the qualitative publications from the work that we did before?’ (T8, lead investigator).
The value of applied qualitative research
The disciplinary background of team members and the associated research paradigms were described as significant for the value placed on the qualitative research. Some clinical specialties were described by interviewees as more sympathetic to qualitative research than others - palliative care, public health, and primary care - where one might argue the complexity of interventions is more obvious than in other specialties. A lack of expertise in qualitative research by the quantitative researchers was described as not only affecting the quality of the qualitative research by imposing quantitative ideals on it (for example wanting large numbers of interviews), but could also affect the confidence quantitative researchers had in the methodological quality of the qualitative research and therefore their willingness to consider its interaction with the trial. These differences were perceived to be due to a lack of understanding of qualitative research by quantitative researchers in the team, or even a lack of understanding that there was anything to understand. The effect of not having expertise, or constraining that expertise, on the quality of the qualitative research was not necessarily evident until it was too late, as in a case where at transcription stage it became apparent that the reality of the ‘in-depth’ interviews was that the participants ‘ only got the choice of saying ‘yes’ or ‘no” (T15, lead investigator). Additionally, some qualitative and mixed-methods interviewees described how qualitative research, applied research, or applied qualitative research might not be valued within their discipline, making it difficult to publish in a high impact journal within their discipline or a ‘decent qualitative journal’ (T5, lead investigator).
Funding agencies
Interviewees noted that qualitative and mixed-methods researchers were now on research commissioning panels which they perceived as creating a more favourable condition for obtaining funding for this type of work. The UK MRC framework for the evaluation of complex interventions was identified as shaping the environment within which researchers worked, promoting the value of qualitative research to funding agencies to the point that they would sometimes request that qualitative research was included within a bid:
‘I think the commissioning process has really changed quite significantly since this was set up […] now the MRC has its famous complex trials framework. And that’s just been part of the general change across the whole community with the interdisciplinary understanding of applied health researchers coming to bear on the broader community to show that qualitative research is important, has important roles to play in trials … And so the commissions have improved dramatically.’ (T2, lead investigator).
However the availability of funds clearly exercised some interviewees. One interviewee described the funder’s view of the qualitative research as integral to the trial and their willingness to fund it fully as a key contributing factor to maximising its value. They then went on to say that ‘ to get a donor who’s willing to put the extra money in is exceptional’ (T15, lead investigator), and indeed another interviewee regarded having dedicated staff to undertake the qualitative research as ‘ quite a luxury’ (T3, qualitative researcher). It was not clear to what extent the funding agencies were contributing actively to this perceived lack of funding for fully resourced qualitative research and to what extent lead applicants were contributing to it by trying to keep the costs of a bid down, either by ‘second-guessing’ their chances of securing a large enough grant to fund the qualitative research properly or by trading on the goodwill of qualitative researchers to ‘squeeze it in’ without proper funding:
‘because it wasn’t that the funder said you cannot have this money, but we underestimated in order to keep the costs of the study down, and hence didn’t have the time or the resource in terms of the researcher to gather the data and to analyse the data as fully as you would have liked. So I suppose that’s the take-home message really [laugh], […] we shouldn’t have been quite so ambitious in the original proposal and should have tried to seek more funding’ (T12, lead investigator).
A related issue for some interviewees was the space on the funding application form being used to give details about the trial and little or no detail about the qualitative research, an issue we also identified in our wider study[
20]. Interviewees suggested that funders could help to promote the importance of describing the qualitative research by changing application forms to offer explicit space for the qualitative research:
‘the quantitative side tends to take predominance very often in terms of the application […] and the qualitative side is much often less fully described, because there’s less space left on the application form’ (T12, lead investigator)
to ensure it was planned in detail:
‘As a grant reviewer, you tend to get very tired of people saying ‘Oh yes, we’ll do a mixed-methods evaluation, including a process evaluation, because it’s very important, blah blah blah’ and then they say absolutely nothing about what exactly they’re going to do […] there are very few applications you get where they can say a lot because of the fact that the forms aren’t helpful to support that.’ (T18, lead investigator).
Publishing the added value
Regardless of the degree of integration of the qualitative research and the trial during the study, and its perceived value by all team members during the trial, communication of learning from this integration at the publication stage seemed to present a final challenge that proved insurmountable for some of our interviewees. They described fragmentation of teams that could leave the lower priority paper unpublished; the production of the findings of the qualitative and the quantitative research could occur at different times resulting in the communication of the value of the qualitative research in terms of explaining the trial findings being lost; or two separate sets of journal articles could be published to report the quantitative and qualitative findings separately without full attention to communicating the value of the qualitative research to the trial:
‘… ideally we would have put more stuff from the process evaluation in the main trial paper but we didn’t … because of the word length, you know, word count restriction meant that just by reporting all the usual stuff for the primary and secondary outcome analyses, there wasn’t really room for much else.’ (T18, lead investigator).