The next stage involves deciding what to include in your synthesis.
What is scope of search?
Defining the specific scope of the search is an important step in any systematic review. Your chosen area of study will influence the search strategy. For example, if very little is published about the topic, you may need to cast your search net more widely. Due to the sheer volume of studies exploring chronic pain, we defined very specific inclusion criteria and excluded a large body of research that did not meet our scope. For example, if the study did not allow us to disentangle the patient experience from that of others (e.g. carers, clinicians, and partners) then it was excluded. One of the challenges that we encountered was the absence of clear descriptions of study samples in the published abstracts. For example, we might not know the type of pain. This meant that we had to retrieve the full text of over 300 studies. A clearer description of the study sample in abstracts would facilitate more cost-effective and relevant qualitative research syntheses.
Do I need to do an exhaustive literature search?
Health research is proliferating and we have access to a vast and growing body of research. Researchers (and their funders) should consider whether or not an exhaustive search of the literature is necessary for qualitative syntheses. It could be argued that a disproportionate amount of time is spent searching for potential qualitative studies, and this time could have been better spent. However, researchers need to consider the prevailing research culture. We wanted to produce a conceptual analysis with a weight of evidence that would have resonance with the health research community who were more used to quantitative systematic reviews, and therefore chose to undertake a systematic search of the published literature. The Cochrane Collaboration has a major role in providing systematic reviews of high quality research. Part of their approach is a systematic search for all the evidence on a topic. Since expecting such a search is part of how high quality reviews are judged and used in practice, we felt it was important to follow this approach to ensure our findings were not dismissed as lacking rigour. A systematic search also gave us the unique opportunity to identify the qualitative studies published within our own area of interest and identify any gaps in knowledge, and to explore the usefulness of meta-ethnography for larger syntheses.
However, you could argue that systematic searching is not integral to high quality meta-ethnography. In their original text on meta-ethnography, Noblit and Hare do not advocate an exhaustive literature search, and the meta-ethnographies included in their core text include only 2–6 studies [
9]. Reviews of published qualitative syntheses show that the number of studies included in meta-ethnographies ranges widely [
2,
4,
6]. Meta-ethnography does not aim to summarise the entire body of available knowledge, or make statistical inference. Meta-ethnography focuses on conceptual insight, and including too many studies might make conceptual analysis ‘unwieldy’ or make it difficult to maintain insight or ‘sufficient familiarity’ [
6]. Whereas in quantitative meta-analysis, omission of a key paper can have a dramatic effect on statistically drawn conclusions, some would argue that this is not true of conceptual qualitative synthesis which aims to develop ideas. Consider Campbell and colleagues’ view that ‘omission of some papers is unlikely to have a dramatic effect on the results’ [
6]. Just as there is no consensus regarding the number of interviews necessary for a ‘good’ qualitative study [
13], there is no consensus about whether or not you need to search for, and include, all available studies for a ‘good’ qualitative synthesis. This does not imply that meta-ethnography is not a rigorous research methodology. Importantly, there is a fundamental difference between qualitative and quantitative analysis that affects the decision of whether, or not, to search for and include all available studies. Namely, in qualitative research, analysis does not begin when all data is collected: Analysis and data collection occur simultaneously, often to the point where no new ideas are developing. Therefore, it may be that sampling strategies compatible with qualitative methodologies are more appropriate to qualitative synthesis. For example, you could stop searching for new data when ‘theoretical saturation’ is reached i.e. when collecting additional data seems to add no more insight [
3,
14]. However, the concept of data or theoretical ‘saturation’ could have its limitations; importantly, how do we know that an additional study will not add important insight?
I am worried about not having all papers. “Would there be that one paper which had a new insight” is always in the back of my mind. This is probably my quantitative systematic review training, but also the feeling that a particularly insightful author could come up with something new. I support the “data saturation approach” and think if the next twenty papers don’t offer anything new, what’s the likelihood of the twenty-first (reflexive statement).
Perhaps the answer to ‘how many’ is that we cannot know, and that ‘it depends’ [
13]. If we can accept that there is a degree of uncertainty, what becomes important is that we include enough ‘data’ to ensure that our conceptual categories are robust, yet at the same time that the project remains grounded in the idiographic.
Did we need to include 77 studies? Would it have worked with 10, 20, 40 . . . .? Can we say any more on what would be ‘enough’? Maybe we can’t. Maybe this is where you need experienced qualitative researcher? That’s more questions than answers. Maybe this isn’t sortable and we have to live with ‘it depends’. Whilst this is true, it is also a bit of a cop out (Reflexive statement).
I suppose the skill in conceptual qualitative analysis is to make a decision whether or not you have enough ‘data’ to support a robust conceptual category. If you don’t then your analysis represents a catalogue or ‘aggregate’ of findings which may provide a lead for further sampling, but it is not necessarily conceptual (Reflexive statement).
Searching and screening
Another challenge specific to qualitative research is how to identify papers without being overwhelmed by the sheer number of hits. Strategies for identifying qualitative search can be unwieldy, and require ‘trade-offs’ between recall and precision [
15]. In an evaluation of search strategies for qualitative research synthesis, Shaw and colleagues found that 96% of the initial search yield was not relevant. This means that search strategies for qualitative research can be over inclusive, time-consuming and expensive [
15]. Accurate indexing and more explicit research designs in qualitative abstracts would facilitate more efficient searching. Our study supports the suggestion that screening for qualitative research syntheses will remain daunting [
15]. The initial search yield of 24,992 studies was title-screened by two team members. If they were uncertain whether or not to include, they next screened the abstract, followed by the full paper. If after reading the full study they were still uncertain two other team members read the paper and made a final decision.
There are some useful resources for qualitative search filters. For example the InterTASC Information Specialists’ Sub-Group (ISSG) Search Filter Resource is a group of information professionals supporting research groups producing technology assessments for NICE [
16]. From there you can access empirically-tested search filters for qualitative studies [
17‐
20]. Shaw and colleagues also provide useful search filters and discuss their relative usefulness for qualitative syntheses [
15]. We searched six electronic bibliographic databases (Medline, Embase, Cinahl, Psychinfo, Amed and HMIC) using the ISSG search filter resources. We did not use the
clinical query limits option for qualitative research, as we found that this limit filtered out relevant qualitative studies. Consider also whether or not you intend to supplement the database search with other strategies. Hand searching specific journals is recognised as important for comprehensively identifying all relevant qualitative studies [
15,
19,
21]. We identified specific journals that we knew agreed reported relevant qualitative research studies in full. These journals were: Journal of Advanced Nursing, Social Science and Medicine, Qualitative Heath Research, Sociology of Health and Illness and Arthritis Care and Research. Your own choices might differ depending on your topic. We subsequently added three journals that contributed the highest number of potential hits on the database searches. We further supplemented the search with citation checks. We did not search the grey literature and PhDs, partly due to the sheer volume of hits, and also because we aimed only to include peer reviewed and published reports. Decisions regarding search strategy and screening hinge upon your aims, resources, availability of studies and epistemological viewpoint. Importantly, do you think that a systematic search that aims to include every available study necessarily leads to more insightful knowledge? Our search strategy took six months of a two year study and 95% of the included studies were identified from three databases.
Quality appraisal
Although the use of quality criteria for qualitative research is debated, a growing number of researchers are choosing to appraise studies for qualitative systematic review. Hannes and colleagues report that the percentage of qualitative syntheses using quality appraisal increased from 40% (1988–2004) to 72% (2005–2008) [
4]. However, although there are many frameworks suggested for appraising quality, there is no consensus about what makes a study good [
6,
22]. The decision to appraise, or not, is confounded by the prevailing research culture where gold standard methodologies are the expectation.
I might not do a quality appraisal if we did it again but still feel constrained by what the health community would think, so would probably feel I had to do it to get our findings used (reflexive statement)
We aimed to explore the issue of quality appraisal for qualitative synthesis [
11] and used three methods of appraisal as a focus for discussion: First, the questions developed by the Critical Appraisal Skills Programme for appraising qualitative research (CASP) which has been used for appraising the quality of studies for meta-ethnography [
10,
23,
24]. We assigned a numerical score to each question to indicate whether we felt that the CASP question had [
1] not been addressed, [
2] been addressed partially or [
3] had been extensively addressed, thus giving a possible score range of 10–30. The CASP was useful in framing our discussions and encouraging us to read ‘carefully and systematically’ [
6]. Secondly, the Qualitative Assessment and Review Instrument (JBI-QARI) designed by the Joanna Briggs Institute for use in systematic reviews of evidence [
25]. Early in the appraisal process, we agreed the JBI-QARI did not add anything further than CASP to the final decision on inclusion. Finally, we categorised papers as either: a ‘key paper’ (‘conceptually rich and could potentially make an important contribution to the synthesis’); a satisfactory paper; a paper that is irrelevant to the synthesis; a methodologically fatally flawed paper [
22]. This method has been used to determine inclusion of studies into meta-ethnography [
26]. The concepts fatally flawed (FF) satisfactory (SAT) and key papers (KP) have not been defined, but are intuitive judgements made by a particular appraiser that comprise unspecified factors. Two team members appraised papers, and if they were unable to reach an agreement, the paper was sent to two other team members to make a final decision.
Our research supports the finding that where tools are used to appraise the quality of qualitative research, there is low inter-rather agreement [
22]. It was striking that although ‘fatally flawed’ papers consistently scored below 20 on CASP, we found it extremely difficult to decipher, or agree about, what made a paper ‘key’ as opposed to a ‘satisfactory’ one [
11]. This may illustrate that appraisal tools focus on methodological rather than conceptual strength. A common approach in quantitative research synthesis, recently adopted in qualitative synthesis, is to use sensitivity analysis to allow the reviewer to assess the impact of including ‘lower quality’ studies on the interpretation. For example, Carroll and colleagues used sensitivity analysis to show the possible benefits of quality appraisal for qualitative research synthesis [
27]. However, this remains a challenge for qualitative synthesis if we do not agree about what good quality is.
Appraisal is time consuming and researchers (and funders) should consider why we appraise qualitative research. For example, do we include methodologically weak studies if they are conceptual rich?
I am still uncomfortable including anything that doesn’t have at least a semblance of decent methods, even if conceptually rich (reflexive statement).
The process I found most difficult to develop a clear view on was the process of critical appraisal. I am still not sure of the value of this although I still feel that some sort of quality appraisal is important. . . I would I think still feel uncomfortable including studies that have significant methodological limitations, but feel it is difficult to make a judgment as to where the line should be drawn (reflexive statement).
Dixon Woods and colleagues exclude studies they judge to be ‘fatally flawed’, and give some guidelines for determining this [
3]. Others argue that excluding studies on the basis of quality criteria may mean that insightful studies are excluded [
6]. Campbell and colleagues include ‘classic’ studies in their meta-ethnography,
assuming ‘methodological integrity’ in the absence of fully reported methods. We felt that although conceptual richness was fundamental to meta-ethnography, the reported methods should be
good enough, and agreed several factors were integral to methodological quality [
11]. Importantly, does the study present a reflexive account of the research process that allows the reader to make a sound judgment about the authors’ interpretation? If we agreed that it did not do this, we did not include it in the synthesis.