Skip to main content

01.12.2019 | Protocol | Ausgabe 1/2019 Open Access

Systematic Reviews 1/2019

Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research

Systematic Reviews > Ausgabe 1/2019
Benjamin Hanckel, Mark Petticrew, James Thomas, Judith Green
Wichtige Hinweise

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Qualitative comparative analysis
Randomised control trial


Identifying the pathways that lead to the success of health interventions is crucial for public health research. However, assessing causal relationships between interventions and outcomes across contexts is challenging [ 13], as multiple interacting components come together in often complex ways [ 46] and ‘inevitable adaptations’ take place as health interventions get implemented within local contexts [ 3]. Recent work [ 7] has pointed to the limitations of using randomised control trials (RCTs) to examine complex health interventions, as they are often unrepresentative of everyday practices across contexts: there is increased input to achieve fidelity, participants in trials may be more committed to the intervention, and context plays an important role in shaping the intervention. Understanding complexity thus requires approaches that draw on real world case studies [ 2, 8] and can identify pathways to successful implementation of, and effective outcomes from, health interventions across diverse contexts.
Qualitative comparative analysis (QCA) is one approach that has been increasingly adopted in public health research and evaluation [ 812] to examine complexity. Developed by Charles Ragin [ 13, 14], QCA is a set-theoretic method, which ‘bridges’ quantitative (large-n statistical analyses) and qualitative (small-n case study) techniques to analyse complex causation across contexts [ 15]. A QCA analysis can involve a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases [ 16, 17]. QCA is often described as being suitable for ‘small-n’ scenarios and is less frequently used in large datasets, though some have used it with some success [ 18].
A QCA approach provides a tool for examining what conditions (alone or in combination with other conditions) are ‘necessary’ or ‘sufficient’ to produce some outcome [ 19]. QCA involves using Boolean algebra and formal logic to examine the presence or absence of conditions, and the effect on an outcome. QCA uses ‘truth tables’, which ‘list all the possible combinations of casual conditions and the empirical outcome associated with each configuration … to identify explicit connections between combinations of casual conditions and outcomes’ [ 19]. When first developed degree of membership was limited by nominal-scale variables, crisp sets, but has since been revised by Ragin to include ‘fuzzy sets’, which scale degree of membership from 0.0 to 1.0 [ 13, 19].
Warren et al. have argued that QCA offers an important contribution to mixed method public health evaluation methodologies [ 20]. Others have identified opportunities to use QCA for informing the translation of research/interventions into practice and policy [ 21, 22]. There is a growing body of scholarship in public health that does use QCA methods for these, and other, aims. However, to date, there has been no systematic review of the use of QCA in public health literature and how it is being adopted. This systematic review will address this gap, examining how it is being used in public health research, including the methods and cases used to explore complexity, the populations it is being applied to, and the types of intervention research that QCA has been used to examine. This is a timely point to assess the field, in order to identify strengths and weaknesses, gaps, and implications for how to better report QCA methods and the case studies on which future QCA studies might be based.

Study aim

The primary aim of the review is to understand what is known about the purpose, role, and use of qualitative comparative analysis (QCA) in research on the uptake and public health effects of interventions.
The proposed systematic review will answer the following research questions:
How is QCA used for public health evaluation? What populations, settings, methods used in source case studies, unit/s and level of analysis (‘cases’), and ‘conditions’ (often referred to as ‘variables’ in public health) have been included in QCA studies?
What strengths and weaknesses have been identified by researchers who have used QCA to understand complex causation in public health evaluation research?
How is QCA used to understand the contexts in which interventions increase or decrease health inequities?
What are the existing gaps in, and strengths and weakness of, the QCA literature, and what implications do these have for future QCA studies for public health?
The study will also identify implications for better reporting of QCA methods in public health research and for reporting case studies in general.


Our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 29 April 2019 ( CRD42019131910). The protocol was prepared using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement [ 23].

Eligibility criteria

Papers included in the systematic review will be published, be peer reviewed, and report on studies that have undertaken a qualitative comparative analysis (QCA) to examine casual complexity in the implementation, uptake, and/or effects of public health interventions. There will be no date restrictions.

Inclusion criteria

Types of participants
Studies will include any population (including national, geographical communities, population sub-groups). As the review will be examining the types of populations included in QCA studies, all study participants will be included. This includes youth and adult populations, their families, and public health professionals.
Types of interventions
QCA studies that incorporate case studies of any kind of public health interventions will be included. Public health interventions are defined here as those focused on the promotion or protection of health, or the prevention of ill health for populations. This is in contrast to clinical interventions, which focus on the prevention or treatment of illness in individuals [ 24]. This will include any interventions for where benefits to population health are the stated outcomes.
Types of outcome measures
To be included, QCA studies must identify an outcome (the phenomenon that the QCA is aiming to explain) that is relevant to public health, in which the effects on population health, health equity, or intervention uptake are being evaluated.
Types of studies
Papers reporting studies that use crisp and/or fuzzy sets will be included. QCA can be included as part of the analysis, or as the only tool used in analysing data.
Papers reporting both primary studies and systematic reviews which use QCA in their analysis will be included.

Exclusion criteria

Publications not in English.
Studies where no intervention is evaluated, including studies that use QCA to examine public health infrastructure (i.e. staff training for low-skilled health staff), or papers that report on prevalence of health issues (i.e. prevalence of child mortality).

Search strategy

The search strategy will use the following phrases ‘Qualitative Comparative Analysis’ and ‘QCA’, which will be combined with keywords (i.e. ‘health’, ‘public health’, ‘intervention’, and ‘wellbeing’). Database searches will be conducted in PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar. Study design must be QCA; no date limits will be imposed on the search. The search will be updated at the end of the review to ensure inclusion of all relevant studies. This will also be supplemented by reviewing reference lists of primary studies identified in the review, reviewing studies that cite these primary studies, as well as inviting experts to comment on whether the final list of studies omits any published studies known to them.

Study selection and data collection

Data and records will be maintained by the lead investigator (BH) and stored on a shared secure platform for access by all investigators.

Selection process

Titles and abstracts identified by the bibliographic search and additional sources will be single screened against the criteria above, with machine learning employed to check these results, acting as a ‘second screener’ [ 25]. ‘EPPI-Reviewer’ Version 4 software, or equivalent machine learning software, will be used to screen titles and abstracts. After removing duplicates, each study will be categorised as ‘include’, ‘exclude’, or ‘unsure’. The full texts of articles categorised as ‘include’ and ‘unsure’ will be retrieved and then assessed against inclusion criteria by reviewers. Those categorised as ‘unsure’ and/or where there are disagreements will be resolved through discussion, and if necessary, a third reviewer will mediate any unresolved differences.
Data extracted will be tabulated, using a standardised form approved by all investigators to ensure consistency (see Table  1). This will include extracting the following information for each included paper: title, author, rationale for QCA, context (setting, case level of analysis, and population characteristics), health intervention, health outcome/s, methods of data generation pertaining to the cases included (qualitative and/or quantitative), findings, and study conclusions. The reviewer/s will also extract any strengths and limitations outlined in each paper. Study authors will be contacted regarding any uncertainties.
Table 1
Study data collection
Study characteristic
Specific items
Study identifiers
Title, authors, journal, publication year, county of publication, and sponsorship
Study aims
Rationale for using a QCA approach
Country/region, population(s), setting
Intervention characteristics
Public health intervention evaluated
Phenomena explained by the QCA
Source case study methods
Methods of data generation pertaining to the cases included (qualitative and/or quantitative)
Strengths and limitations of approach identified by authors

Assessment of risk of bias

Quality assessment

The objective of this review is to examine how QCA is currently used in the public health literature. There are no established reporting criteria for quality, so no studies will be excluded on the basis of quality. However, there are suggestions as to how to best report QCA [ 15, 16]. We will utilise four key criteria from these recommendations to map current reporting of the method. These will be evidence of familiarity with cases, justification for selection of cases, availability of raw data matrix and ‘truth table’ outcomes, and solution formula reported.

Data synthesis

Data will be synthesised narratively, using thematic synthesis methods [ 2628]. Final studies included will, following a descriptive analysis, be grouped by outcome measures, as well as context of intervention (e.g. school). We will assess claims about the utility of the method for addressing complex causation in public health research by identifying how far the published QCA study was able to identify necessary and sufficient conditions for the selected outcome. We will analyse separately (as a sub sample) those studies with an outcome related to health equity or inequalities, to assess how far QCA has been used to date in this area.


QCA has been increasingly used in public health research, although the utility of the method for producing policy-relevant findings has been questioned [ 29]. The use of QCA in public health research therefore warrants further attention. We will map current use of the method, including identifying what types of intervention and setting have been addressed and what strengths and weaknesses researchers have identified in using QCA to address questions of complex causation. We will assess the extent to which current reporting in public health matches recommendations for best practice within the QCA research community, and the implications of these recommendations and current practice for the public health research community. As QCA relies on detailed knowledge of cases, we will also comment on the implications for public health researchers publishing stand-alone case studies, which might be used in future QCA studies. At present, there has been no systematic review of the use of QCA in public health literature and how it is being adopted; this systematic review will fill this gap, with the aim to inform, guide, and assess use of QCA in future complex health systems research.
There are several limitations to this study. Including only published studies from a rapidly emerging field may not be representative of the most recent applications of the method. In addition, we are only able to include those in English, which may underrepresent high-quality research from some regions.


The authors would like to thank and acknowledge the support of Sara Shaw, PI of MR/S014632/1.

Ethics approval and consent to participate


Consent for publication


Competing interests

The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://​creativecommons.​org/​licenses/​by/​4.​0/​), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://​creativecommons.​org/​publicdomain/​zero/​1.​0/​) applies to the data made available in this article, unless otherwise stated.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Über diesen Artikel

Weitere Artikel der Ausgabe 1/2019

Systematic Reviews 1/2019 Zur Ausgabe