Why an approach to assessing confidence in the evidence from reviews of qualitative research is needed
Aims of the CERQual approach
Strengths related to how the approach was developed: | |
• Developed by a diverse group of international methodologists, qualitative researchers, systematic review authors and guideline developers. A few members of the group were also involved in health care decision making | |
• Refined over several years through testing on a substantial number of qualitative evidence syntheses and through several rounds of consultations with academics and users in relevant fields | |
Strengths related to the design of the approach: | |
• Uses terminology, concepts and theoretical underpinnings that are sensitive to qualitative research | |
• Provides explicit guidance on which concerns/threats to consider that may lead users to lower their confidence in the evidence | |
• Makes judgements about confidence in qualitative evidence more transparent | |
• The approach is independent of specific primary qualitative research methods and methods of synthesis | |
• Assessments of confidence are based on multiple interdependent components | |
Strengths related to the uses of the approach: | |
• Assessments can be used within diverse decision making processes, including guideline development and health technology assessments, alongside GRADE assessments for other forms of evidence | |
• The approach is congruent with other GRADE approaches for other types of evidence, and so can be easily integrated with these other approaches in decision making | |
• The approach is well received and understood by stakeholders, when used in decision making processes including guideline development | |
• Within decision making processes, CERQual may facilitate the use of qualitative evidence to address a range of issues. These include which outcomes are important to stakeholders; the acceptability and feasibility of interventions, including differences in views across different stakeholder groups; considerations regarding implementation; and the unintended consequences of interventions |
Assumptions underlying the development of CERQual
-
We acknowledge that some within the qualitative research community have argued that synthesising data across multiple qualitative studies challenges the integrity of the contributing primary studies and that findings from this synthesis process may therefore not be trustworthy (e.g., [19‐21]). However, in our approach to qualitative evidence synthesis and the development of CERQual, we have adopted the ‘subtle realist’ position [22] which maintains that the existence of phenomena does not depend on our subjective perceptions of them. In other words, social reality is not entirely constructed. Based on this, we suggest that synthesis can potentially provide a deeper understanding of a phenomenon than is achievable by any one single study, that this understanding can be viewed as trustworthy, and that it is therefore desirable to synthesise data from multiple qualitative studies. Others have made similar arguments in relation to ethnography, noting that comparisons of different ethnographic studies ‘are fruitful because they lead to empirical generalisations, they expose analytical problems, and they allow for falsification of hypotheses’ ([23] page 207).
-
We acknowledge debates on the most appropriate methods for synthesis but argue that concerns regarding the synthesis process and its outputs should be addressed in how a synthesis is undertaken (for example, by using methods that help to preserve the context of the primary studies in the analysis process) rather than being seen as a complete barrier to conducting syntheses or to the practical usefulness of qualitative synthesis findings to decision-makers [24].
-
CERQual assumes that qualitative research, in addition to its more interpretative and exploratory functions, has an instrumental role to play in informing decisions. In other words, qualitative research holds the potential to produce knowledge that can directly inform decision-making processes.
-
CERQual acknowledges that a well-conducted qualitative evidence synthesis does not automatically produce useful findings applicable to a range of contexts. As with primary qualitative research, sophisticated processes of analysis and interpretation are required. CERQual aims to accommodate the interpretivist nature of qualitative synthesis by, for example, encouraging the review authors to examine possible theoretical contributions and to be sensitive to the importance of context when assessing confidence in the evidence
-
The CERQual approach is intended to be applied to well-conducted syntheses that report their methods and limitations in a transparent way. We believe that applying CERQual to the findings of a poorly conducted or poorly reported synthesis would be challenging and would not yield useful results. Paper 2 in the series provides guidance on assessing how well a review was conducted [25].
How was the CERQual approach developed?
Initial development of the CERQual approach
Further development of the CERQual approach
The role of dissemination bias
An overview of the CERQual approach to assessing confidence in the evidence
Component | Definition |
---|---|
Methodological limitations | The extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding |
Coherence | An assessment of how clear and cogent the fit is between the data from the primary studies and a review finding that synthesises that data. By ‘cogent’, we mean well supported or compelling |
Adequacy of data | An overall determination of the degree of richness and quantity of data supporting a review finding |
Relevance | The extent to which the body of evidence from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question |
Level | Definition |
---|---|
High confidence | It is highly likely that the review finding is a reasonable representation of the phenomenon of interest |
Moderate confidence | It is likely that the review finding is a reasonable representation of the phenomenon of interest |
Low confidence | It is possible that the review finding is a reasonable representation of the phenomenon of interest |
Very low confidence | It is not clear whether the review finding is a reasonable representation of the phenomenon of interest |
Applying CERQual across types of qualitative data and synthesis methods
Purpose and structure of this series of papers
Conclusions: the widening influence of qualitative evidence
The following steps are needed to further develop the approach: |
• To date, there is little collective experience of applying CERQual in the context of mixed method syntheses that include qualitative and quantitative data. Whether the approach needs to be adapted for this context needs to be explored. An important concern is whether assessing the quantitative and qualitative elements of a mixed-methods study individually, using separate approaches, risks under-valuing the contribution of review findings based on integrated data |
• In some decision making processes, CERQual assessments of qualitative evidence may be presented alongside other GRADE assessments for data on intervention effectiveness and resource use.User testing is needed to explore how best to present this range of assessments to evidence users |
• Our aim is that CERQual can be applied to review findings based on any kind of qualitative data. However, we do not have experience of applying the approach to syntheses where the primary material includes sources that are textual in nature but are not the output of formal qualitative research procedures. Such sources include blogs, online discussion group transcripts or newspaper reports. Further work is needed to examine how the approach can be used for such data |
• We need to gather experience and, if necessary, adapt CERQual for syntheses of primary studies outside the field of health and health care research |
• We need further work on whether CERQual needs to be adapted for application to more interpretive outputs from syntheses, such as logic models and findings from synthesis methods such as meta-ethnography |