Background
The global health movement is increasingly calling for qualitative research to accompany its projects and programmes [
1]. This demand, and the funding that goes with it, has led to critical debates among qualitative researchers, particularly over their role as applied or theoretical researchers [
2]. An additional challenge emanating from this demand is to justify research findings and methodological rigour in terms that are meaningful and useful to global public health practitioners. A key area that has grown in quantitative health research has been in quality assurance activities, following the social movement towards evidence-based medicine and global public health [
3]. Through the eyes of this movement, the quality of research affects not only the trajectory of academic disciplines but also local and global health policies. Clinical trials researchers and managers have led much of health research into an era of structured standardised procedures that demarcate and assure quality [
4,
5].
By contrast, disciplines using qualitative research methods have, to date, engaged far less frequently with quality assurance as a concept or set of procedures, and no standardised guidance for assuring quality exists. The lack of a unified approach to assuring quality can prove unhelpful for the qualitative researcher [
6,
7], particularly when working in the global health arena, where research needs both to withstand external scrutiny and provide confidence in interpretation of results by internal collaborators Furthermore, past and existing debates on what constitutes 'good' qualitative research have tended to be centred firmly within social science disciplines such as sociology or anthropology, and as such, their language and content may prove difficult to penetrate for the qualitative researcher operating within a multi-disciplinary, and largely positivist, global health environment.
The authors and colleagues within the ACT Consortium [
8] conduct qualitative research that is mostly rooted in anthropology and sociology, to explore the use of antimalarial medicines and intervention trials around antimalarial drug use, within the global health field. Through this work, within the context of clinical trials following Good Clinical Practice (GCP) guidelines [
4], we have identified a number of challenges relating to the demands for evidence of quality and for quality assurance of qualitative research. The quality assurance procedures available for quantitative research, such as GCP training and auditing, are rooted in a positivist epistemology and are not easily translated to the reflexive, subjective nature of qualitative research and the interpretivist-constructionist epistemological position held by many social scientists, including the authors. Experiences of spatial distance between collaborators and those working in remote study field sites have also raised questions around how best to ensure that a qualitative research study is being conducted to high quality standards when the day-to-day research activity is unobservable by collaborators.
In response to the perceived need for the authors' qualitative studies to maintain and demonstrate quality in research processes and outcomes, we sought to identify existing guidance for quality assurance of qualitative research. In the absence of an established unified approach encapsulated in guidance format, we saw the need to review literature addressing the concept and practice of quality assurance of qualitative research, as a precursor to developing suitable guidance.
In this paper, we examine how quality assurance has been conceptualised and defined within qualitative paradigms. The specific objectives of the review were to, firstly, identify literature that expressly addresses the concept of quality assurance of qualitative research, and secondly, to identify common narratives across the existing discourses of quality assurance.
Methods
Search strategy
Keywords were identified from a preliminary review of methodological papers and textbooks on qualitative research, reflecting the concepts of 'quality assurance' and 'qualitative research', and all their relevant synonyms. The pool of keywords was augmented and refined iteratively as the search progressed and as the nature of the body of literature became apparent. Five electronic databases-Academic Search Complete, CINAHL Plus, IBSS, Medline and Web of Science-were searched systematically between October and December 2010, using combinations of the following keywords: "quality assurance", "quality assess*", "quality control*", "quality monitor*", "quality manage*, "audit*", "quality", "valid*", "rigo*r", "trustworth*", "legitima*", "authentic*", "strength", "power", "reliabil*", "accura*","thorough*", "credibil*", "fidelity", "authorit*", "integrity", "value", "worth*", "good*", "excellen*", "qualitative AND (research OR inquiry OR approach* OR method* OR paradigm OR epistemolog* OR study). Grey literature was also searched for using Google, and the key phrases "quality assurance" AND "qualitative research".
Several relevant journals-International Journal of Qualitative Methods, International Journal of Social Research Methodology and Social Science and Medicine - were hand searched for applicable papers using the same keywords. Finally, additional literature, in particular books and book chapters, was identified through snowballing techniques, both backwards by following references of eligible papers and forwards through citation chasing. At the point where no new references were identified from the above techniques, the decision was made to curtail the search and begin reviewing, reflecting the practical and time implications of adopting further search strategies.
Inclusion and exclusion criteria
Inclusion criteria were identified prior to the search, to include:
Methodological discussion papers, books or book chapters addressing qualitative research with explicit focus on issues of assuring quality.
Guidance or training documents (in 'grey literature') addressing quality assurance in qualitative research.
Excluded were:
Publications primarily addressing critical appraisal or evaluation of qualitative research for decision-making, reviews or publication. These topics were considered to be distinct from the activity of quality assurance which occurs before writing up and publication.
Publications focusing only on one or more specific qualitative methods or methodological approaches, for example grounded theory or focus groups; focusing on a single stage of the research process only, for example, data collection; or primarily addressing mixed methods of qualitative and quantitative research. It was agreed by the authors that these method-specific papers would not help inform narratives about the discourse of quality assurance, but may become useful at a later date when developing detailed guidance.
Publications not in the English language.
Review methodology
A meta-narrative approach was chosen for the reviewing and synthesis of the literature. This is a systematic method developed by Greenhalgh et al [
9] to make sense of complex, conflicting and diverse sources of literature, interpreting the over-arching narratives across different research traditions and paradigms [
9,
10]. Within the meta-narrative approach, literature is mapped in terms of its paradigmatic and philosophical underpinnings, critically appraised and then synthesised by constructing narrative accounts of the contributions made by each perspective to the different dimensions of the topic [
9]. Due to the discursive nature of the literature sought, representing different debates and philosophical traditions, the meta-narrative approach was deemed most appropriate for review and synthesis. A process of evaluating papers according to predefined quality criteria and using methods to minimise bias, as in traditional, Cochrane-style systematic reviewing, was not considered suitable or feasible to achieve the objectives.
Each paper was read twice by JR, summarised and analysed to determine the paper's academic tradition, the debates around quality assurance in qualitative research identified and discussed, the definition(s) used for 'quality' and the values underpinning this, and recommended methods or strategies for assuring quality in qualitative research. At the outset of the review, the authors attempted to identify the epistemological position of each paper and to use as a category by which to interpret conceptualisations of quality assurance. However, it emerged that fewer than half of the publications explicitly presented their epistemology; consequently, epistemological position was not used in the analytical approach to this review, but rather as contextual information for a paper, where present.
Following the appraisal of each paper individually, the literature was then grouped by academic disciplines, by epistemological position (where evident) and by recommendations. This grouping enabled the authors to identify narratives across the literature, and to interpret these in association with the research question. The narratives were developed thematically, following the same process used when conducting thematic analysis of qualitative data. First, the authors identified key idea units in each of the papers, then considered and grouped these ideas into broader cross-cutting themes and constructs. These themes, together with consideration of the epistemologies of the papers, were then used to develop overarching narratives emerging from the reviewed literature.
Discussion
The literature reviewed showed a lack of consensus between qualitative research approaches about how to assure quality of research. This reflects past and on-going debates among qualitative researchers about how to define quality, and even the nature of qualitative research itself. The two main narratives that emerged from the reviewed literature reflected differing approaches to quality assurance and, underpinning these differing conceptualisations of quality in qualitative research.
Among the literature that directly discusses quality assurance in qualitative research, the most dominant narrative detected was that of an output-oriented approach. Within this narrative, quality is conceptualised in relation to theoretical constructs such as validity or rigour, derived from the positivist paradigm, and is demonstrated by the inclusion of certain recommended methodological techniques. By contrast, the second, process-oriented narrative presented conceptualisations of quality that were linked to principles or values considered inherent to the qualitative approach, to be understood and enacted throughout the research process. A third, minor narrative offered critique of current and recent discourses on assuring quality of qualitative research but did not appear to offer alternative ways by which to conceptualise or conduct quality assurance.
Strengths of the output-oriented approach for assuring quality of qualitative studies include the acceptability and credibility of this approach within the dominant positivist environment where decision-making is based on 'objective' criteria of quality [
11]. Checklists equip those unfamiliar with qualitative research with the means to assess its quality [
6]. In this way, qualitative research can become more widely accessible, accepted and integrated into decision-making. This has been demonstrated in the increasing presence of qualitative studies in leading medical research journals [
11,
12]. However, as argued by those contributing to the second narrative in this review, the following of check-lists does not equate with understanding of and commitment to the theoretical underpinnings of qualitative paradigms or what constitutes quality within the approach. The privileging of guidelines as a mechanism to demonstrate quality can mislead inexperienced qualitative researchers as to what constitutes good qualitative research. This runs the risk of reducing qualitative research to a limited set of methods, requiring little theoretical expertise [
52] and diverting attention away from the analytic content of research unique to the qualitative approach [
14]. Ultimately, one can argue that a solely output-oriented approach risks the values of qualitative research becoming skewed towards the demands of the positivist paradigm without retaining quality in the substance of the research process.
By contrast, strengths of the process-oriented approach include the ability of the researcher to address the quality of their research in relation to the core principles or values of qualitative research (see Table
2). For example, previous assumptions that incorporating participant-observation methods over an extended period of time in 'the field' constituted 'good' anthropology and an indicator of quality have been challenged on the basis that fieldwork as a method should not be conducted uncritically [
53], without acknowledgement of other important steps, including exploring variability and contradiction [
54], and being explicit about methodological choices made and the theoretical reasons behind them [
55]. The core principles identified in this narrative also represent continuous, researcher-led activities, rather than externally-determined indicators such as validity, or end-points. Reflexivity, for example, is an active, iterative process [
56], described as '
an attitude of attending systematically to the context of knowledge construction... at every step of the research process' [p484, 23]. As such, this approach emphasises the need to consider quality throughout the whole course of research, and locates the responsibility for enacting good qualitative research practice firmly in the lap of the researcher(s).
The question remains, however, as to
how researchers can demonstrate to others that core principles have guided their research process. The paucity of guidelines among those advocating a process-oriented approach suggests these are either not possible or not desirable to disseminate. Guidelines, by their largely fixed nature, could be considered incompatible with flexible, pluralistic, qualitative research. Awareness and understanding of the fundamental principles of qualitative research (such as those six identified in this review) could be considered sufficient to ensure that researchers conduct the whole research process to a high standard. Indeed, it could be argued that this type of approach has been promoted within qualitative research fields beyond the health sciences for several decades, since debates around how to do 'good' qualitative research emerged publically [
41,
43,
51]. However, the premises of this approach are challenged by increasing scrutiny over the accuracy and ethics of the generation of information through scientific activity [
57,
58]. Previous critiques of a post-hoc evaluation approach to quality, in favour of procedural mechanisms to ensure good research [
43], have not responded to the demand in some research contexts, particularly in global health, for externally demonstrable quality assurance procedures.
The authors propose, therefore, that some form of guidelines may be possible and desirable, although in a less structured format than those representing a more positivistic paradigm and based on researcher-led principles of good practice rather than externally-determined constructs of quality such as validity. However, first it is important to acknowledge some of the limitations of our search and interpretations.
Limitations
The number of papers included in the review was relatively low. The search was limited to publications explicitly focused on 'quality assurance', and the inclusion criteria may have excluded relevant literature that uses different terminologies, particularly as this concept has not commonly been used within qualitative methods literature. As has been demonstrated in the narratives identified, approaches to quality assurance are linked closely to conceptualisations of quality, about which there is a much larger body of literature than was reviewed for this paper. The possibility of these publications being missed, along with other hard-to-find and grey literature, has implications for the robustness of the narratives identified.
This limitation is perhaps most evident in the lack of literature in this review identified from the field of anthropology. Debates around concepts such as validity and what constitutes 'knowledge' from research have long been of interest to anthropologists [
55], but the absence of these in the publications which met the inclusion criteria raises questions about the search strategy used. Although the search strategy was revised iteratively during the search process to capture variations of quality assurance, anthropological references did not emerge. The choice was made not to pursue the search further for practical and time-related reasons, but also as we felt that limiting the review to quality assurance as originally described would be useful for understanding the literature that a researcher would likely encounter when exploring quality assurance of qualitative research. The lack of clear anthropological voice in this literature reflects the paucity of engagement with the theoretical basis of this discipline in the health sciences, unlike other social sciences such as sociology [
52]. As such, anthropology's contributions to debates on qualitative research methods within health and medical research have been somewhat overlooked [
59].
Hence, this review presents only a part of the discourse of assuring quality of qualitative research, but it does reflect the part that has dominated the fields of health and medical research. Although this review leaves some unanswered questions about defining and assuring quality across different qualitative disciplines, we believe it gives a valuable insight into the types of narratives a typical researcher would begin to engage with if coming from a global health research perspective.
Recommendations
The narratives emerging from this literature review indicate the challenges related to approaching quality assurance from a perspective shaped by the positivist fields of evidence-based medicine, but also the lack of clear, structured guidance based on the intrinsic principles of qualitative research. We recommend that the strengths of both the output-oriented and process-oriented narratives be brought together to create guidance that reflects core principles of qualitative research but also responds to expectations of the global health field for explicitly assured quality in research. The fundamental principles characterising qualitative research, such as the six presented in Table
2, offer the basis of an approach to assuring quality that is reflexive of and appropriate to the specific values of qualitative research.
The next step in developing guidance should focus on identifying practical and specific advice to researchers as to how to engage with these principles and demonstrate enactment of the principles at each stage of the research process while being wary of promoting unthinking use of 'technical fixes' [
6]. We recommend the development of a framework that helps researchers to identify their core principles, appropriate for their epistemological and methodological approach, and ways to demonstrate that these have been upheld throughout the research process. Current generic quality assurance activities, such as the use of standard operating procedures (SOPs) and monitoring visits could be attuned to the principles of the qualitative research being undertaken through an approach that demonstrates quality without constraining the research or compromising core principles. The development of such a framework should be undertaken in a collaborative way between researchers and field teams undertaking qualitative research in practice. We propose that this framework be flexible enough to accommodate different qualitative methodologies without dictating essential activities for promoting quality. Unlike previous guidance, we propose the framework should also respond to different demands from multi-disciplinary research teams and from external, positivist, audiences for evidence of quality assurance procedures, as may be faced, for example, in the field of global health research. This review has also highlighted the challenges of accessing a broad range of literature from across different social science disciplines (in particular anthropology) when conducting searches using standard approaches adopted in the health sciences. Further consideration should be taken as to how best to encourage wider search parameters, familiarisation with different sources of literature and greater acceptance of non-traditional disciplinary perspectives within health and medical literature reviews.
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
JR helped with the design of the review, searched for and reviewed the literature and wrote the first draft of the manuscript. JK, NE, PM and EA contributed to the interpretation of the results and the writing the manuscript. CC conceived of the review and helped with its design, interpretation of results and writing the manuscript. All authors read and approved the final manuscript.