We will follow the six stages in the following example, and because this is an early worked example of QCA in synthesis, we will retain stage 5: consideration of ‘logical remainders’. However, this stage was not necessary in our example, and further methodological work will be required to ascertain whether this stage can be omitted from most syntheses using QCA or whether it has a useful role to play in certain situations.
Stage 1: Building the data table
The data table consists of rows that represent studies, while the columns represent conditions (characteristics of the cases) and the outcome/s. The data table for these analyses is shown in Table
1.
Table 1
Data table for breastfeeding interventions that incorporate community engagement
| 0 | 0 | 1 | 1 | 1 | 8.458 | 1.000 |
| 0 | 0 | 1 | 0.333 | 1 | 3.783 | 1.000 |
| 1 | 1 | 0 | 1 | 0 | 1.751 | 0.666 |
Grummer-Strawn (1997) [ 18] | 0 | 0 | 0 | 0 | 0 | 1.927 | 0.666 |
| 0 | 1 | 0 | 0 | 0 | 0.463 | 0.000 |
| 0 | 0 | 1 | 0 | 0 | 5.397 | 1.000 |
| 0 | 0 | 1 | 0 | 0 | 1.729 | 0.333 |
| 1 | 1 | 1 | 0 | 0 | 1.614 | 0.333 |
| 0 | 0 | 0 | 1 | 1 | 6.000 | 1.000 |
| 0 | 0 | 0 | 1 | 1 | 2.786 | 0.666 |
| 0 | 1 | 1 | 1 | 1 | 8.458 | 1.000 |
| 0 | 0 | 1 | 0 | 0 | 2.317 | 0.666 |
Membership in the conditions in this dataset is almost exclusively binary: cases are either members (represented by a ‘1’) or non-members (represented by a ‘0’) of a condition. These are also referred to as ‘crisp’ sets. As defined above, the conditions are empowerment, involvement in intervention design, and lay-led intervention
. Note that there are an additional two conditions, ‘Quality’ and ‘Intensity’, included as columns in Table
1. These will be described later in the paper as they emerged as a part of the iterative QCA process (that is, they were not specified
a priori).
As mentioned above, transposing ‘purposive’ sampling techniques from primary research can be difficult in the context of a systematic review because we cannot necessarily identify positive and negative cases; we can only use the studies that have evaluated the interventions and outcomes of concern. The sample of studies that we have, therefore, is more akin to an unbiased or ‘population’ sample - in that we have all the studies (that we can find) that evaluate a given intervention. If we have little heterogeneity between results, there is little the QCA can do to help us identify sufficient conditions for success.
The outcome in this dataset is an indicator of the effectiveness of the interventions, which were all evaluated in controlled trials (both randomised and non-randomised). For the non-randomised studies, we have no reason to believe that the participants in one condition or another were more likely to breastfeed before the start of the intervention. The original metric used in the meta-analysis to estimate the magnitude of the outcome was an effect size estimate
b that compared the health behaviours of participants in the intervention group to those in the control group at immediate post-test (that is, directly after the intervention finished). In most cases, the measure was the number of mothers breastfeeding at a given time point. Note that all but one of the interventions were effective (in a systematic review, it is not always possible to have clearly differentiated positive and negative cases), so the outcome for the QCA analyses was membership in the set of
highly effective interventions. The log of the effect size estimates (odds ratios, OR) were calibrated for use in the QCA analyses by converting them into a fuzzy set that allows for degrees of membership
c. In this review, we used the following effect size calibration rules:
1.
Full membership in the set of ‘highly effective interventions’: if logged OR > .7.
2.
More in than out of the set: .4 < logged OR ≤ .7.
3.
More out than in the set: 0 < logged OR ≤ .4.
4.
Fully out of the set: logged OR ≤0.
In Table
1, the original raw effect sizes are shown in the column ‘Effect size (raw odds ratio)’ and the calibrated fuzzy set outcome is shown in the column ‘Highly effective intervention fuzzy set’. (Calibration of fuzzy sets is a complex topic, and for further information we recommend Part II of [
8].)
Stage 2: Constructing the truth tables
Once the data have been prepared, the focus of analysis moves from individual studies to the different configurations of conditions that are associated with the outcome of interest. As noted above, conditions are characteristics of the cases. Different combinations of conditions are referred to as
configurations. For
k conditions, there are 2
k
configurations. In the example in Table
2, three possible conditions can be combined in eight (2
3) different configurations. Each configuration is itself a set, or group, to which studies can be members or not members; studies with the same configuration are included in a set, while studies with different configurations will have membership in different sets.
Table 2
Example of possible configurations of three conditions with their set labels
1 | 1 | 1 | Empowerment*Lay-led*Design |
1 | 1 | 0 | Empowerment*Lay-led* ~ Design |
1 | 0 | 1 | Empowerment* ~ Lay-led*Design |
1 | 0 | 0 | Empowerment* ~ Lay-led* ~ Design |
0 | 1 | 0 | ~Empowerment*Lay-led* ~ Design |
0 | 1 | 1 | ~Empowerment*Lay-led*Design |
0 | 0 | 0 | ~Empowerment* ~ Lay-led* ~ Design |
0 | 0 | 1 | ~Empowerment* ~ Lay-led*Design |
The labelling of sets follows certain conventions. An asterisk * is used to combine conditions (equivalent to ‘And’), and a tilde ~ is used to indicate non-membership in a condition. So, for example, a study with the conditions ‘empowerment model evident, with the intervention led by members of the community, but no community involvement in the intervention design’ would be labelled as
Empowerment*Lay-led* ~ Design. The fourth column of Table
2 shows the set labels for the various configurations for those three conditions.
Having constructed a data table as described in ‘Stage 1’ above (that is, with an outcome calibrated to the four fuzzy membership levels and with several binary conditions), we then constructed a truth table. A truth table displays the conditions, configurations, and the number of studies with membership in each configuration set. Unlike the original dataset in Table
1, in which each study is a case, a truth table presents each configuration as a case (as in the example in Table
2).
The truth table summarises how many studies within a set (or configuration) are instances of the outcome. In this example, the outcome of interest is whether the intervention was highly effective, and so the truth table indicates how many of the studies within a configuration are members, or partial members, in the set of highly effective interventions. There are four possible kinds of result for each configuration:
1.
Positive cases: All studies within a set are instances of the outcome (that is, all studies in the configuration are effective).
2.
Negative cases: No studies within a set are instances of the outcome (that is, no studies in the configuration are effective).
3.
Contradictions: Some of the studies are instances of the outcome and some are not (that is, studies in the configuration are mixed in terms of their effectiveness; discussed in ‘Stage 3’ below).
4.
Remainders: There are no studies in the dataset with that particular configuration of conditions and outcome (discussed in ‘Stage 5’ below).
Since the number of possible configurations increases exponentially as the number of conditions increases, it does not take the addition of many conditions for the number of possible configurations to exceed the number of studies in the analysis. The potential problem arising from this is known as ‘limited diversity’ [
27]; that is, the analysis can simply become a description of each individual study, rather than a synthesis where lessons are drawn from across the included studies. The objective is to conduct an analysis that is sufficiently rich, containing the most salient conditions able to explain differences between study outcomes, but where less important conditions are excluded from the analysis. Some primary QCA analyses have used prior analytical strategies including discriminant analysis, factor analysis and cluster analysis to help inform the selection of conditions (for example, [
7,
28]), though for this example, no such techniques were employed.
The truth table for the first model, in which the conditions (empowerment, design, lay-led) were examined for ‘highly effective interventions’, is presented in Table
3. The rows are in descending order of consistency, which is the metric used in QCA to express how far the pattern of all the cases is consistent with sufficiency. Consistency is defined as a metric that answers the question ‘To what extent is the statement ‘configuration A is necessary for the outcome’ consistent? Technically, this can be computed as follows: (the number of cases with a [
1] value on the configuration AND a [
1] outcome value, divided by the total number of cases with a [
1] outcome value)’ [
7]. Ragin discusses appropriate cut-off levels for consistency, arguing that they should be as close to 1 as possible (though the greater the number of studies in a particular configuration, the less likely this becomes), and that it is difficult to justify drawing conclusions when consistency scores are below 0.75 [
29]. For our analyses, we adopted a cut-off for consistency of 0.75 or above.
Table 3
Truth table for model 1: community engagement models as the conditions and ‘highly effective intervention’ as the outcome
0
|
1
|
1
|
1
|
1
|
1.000
|
0 | 0 | 1 | 5 | 0 | 0.666 |
0 | 0 | 0 | 3 | 0 | 0.666 |
1 | 1 | 1 | 1 | 0 | 0.333 |
1 | 1 | 0 | 1 | 0 | 0.333 |
0 | 1 | 0 | 1 | 0 | 0 |
1
|
0
|
0
|
0
| | |
1
|
0
|
1
|
0
| | |
We can see that only one row (that is, one configuration) has sufficient raw consistency (that is, ≥0.75) to be classified as having full membership in the set of ‘highly effective interventions’. However, that configuration consists of only one study.
The truth table for the second model, in which the conditions (empowerment, design, lay-led) were examined for the negated outcome (that is, ‘not highly effective interventions’), is presented in Table
4. Again, only one row (configuration) has sufficient raw consistency to be classified as having full membership in the set of ‘not effective interventions’, but this only represents one study.
Table 4
Truth table for model 2: community engagement models as the conditions and ‘not effective interventions’ as the negated set outcome
0
|
1
|
0
|
1
|
1
|
1
|
1
|
1
|
1
|
1
|
0
|
0.667
|
1
|
1
|
0
|
1
|
0
|
0.667
|
0
|
0
|
0
|
3
|
0
|
0.334
|
0
|
0
|
1
|
5
|
0
|
0.334
|
0
|
1
|
1
|
1
|
0
|
0
|
1
|
0
|
0
|
0
| | |
1
|
0
|
1
|
0
| | |
We mentioned earlier that it is easier to identify sufficient conditions for an outcome than it is to find necessary conditions because logically there may be (many) other ways of arriving at a given outcome, even if those methods are not present in any of the studies in the analysis. Bearing in mind the above mentioned problems relating to sampling - that we can do no more in a systematic review than find the studies that have already been conducted and cannot collect additional data to fill in gaps, as we might in primary research - we think it unlikely that reviewers will want to identify necessary conditions for complex interventions to generate an outcome. This is because our conceptualisation of complexity requires that we view each intervention and context as potentially unique; therefore, a condition that may be necessary in all the studies we can see may not be necessary in all possible situations.
Stage 2B: Checking the quality of the truth table
There are a number of items to check once the truth table has been created in order to ensure that it will perform adequately in the proceeding stages of analysis. Principally, this involves checking that there is a good spread of studies across the different configurations available, and that both positive and negative occurrences of the outcome are well covered. In Tables
3 and
4, we can see that we do not have a reasonable spread in terms of outcomes and data for all but two of the eight possible configurations. Also, in both models, there are two configurations for which there are no cases; these are known as ‘remainders’. Moreover, only one study in each model is ‘consistent’ enough (that is, raw consistency ≥0.75) to proceed to the next stage of analysis.
If the initial check of the truth table reveals areas of concern, for example, a lack of variation among conditions, which might render explanation of the outcome difficult, it is recommended that reviewers return to the conceptual framework that their review is based upon and consider again the dimensions upon which included studies might differ. This, in turn, will prompt a re-examination of the conditions to be used in the synthesis and possibly lead to the incorporation of new, or different, conditions. (See also the suggestions below on resolving contradictory configurations.) Another way of approaching this might be to become more acquainted with the studies themselves, in the expectation that new lines of enquiry will emerge. Whether the former (a more deductive approach) or the latter (inductive) is used, there should be a ‘dialogue’ between the truth table and the studies and concepts it is based on. Some iteration is expected before the final table emerges. (Though discussion of this is outside the scope of this paper, the investigative model adopted by QCA might best be described as ‘abductive’ [
30]).
We considered the conditions tested in the initial analysis to be uninformative other than telling us that there are too few studies that have employed an empowerment approach for us to come to a conclusion regarding the efficacy of this method of engagement. We therefore considered whether any other conditions might meaningfully distinguish between highly effective interventions and those interventions with smaller benefits. Based on our understanding of the studies, we decided that two conditions were likely to have a large impact on the effectiveness of the breastfeeding interventions: the intensity of the intervention and the quality of the intervention.
We returned to the 12 original studies and extracted additional information about intervention intensity and quality. Through an iterative process of interacting with the studies and discussion amongst the team, we developed definitions for these two additional concepts. ‘Intensity’ is based on our understanding of the studies’ theories of change, where the most critical period for supporting breastfeeding is immediately pre- and post-partum. Studies in the set of ‘intense’ interventions would recognise this by having frequent contact in this early period, with less intensive support later on. The second condition, ‘quality’, follows some of the principles of engagement identified elsewhere in our review. ‘High quality’ interventions were defined as those that were customisable to women's needs; had multiple support options; emphasised personal contact (for example, face-to-face as opposed to support via telephone or letters); included counselling (tailored information); were delivered in a location that suited the women; and had trained staff (including practice observation). In this example, these attributes have been combined into a single ‘quality’ intervention characteristic. It would be possible to have a separate condition for each quality attribute, though one might quickly run into the problems of ‘limited diversity’ identified earlier (please see discussion on ‘compound’ conditions).
We then reran the analyses with these conditions in the model. The truth table for this model can be seen in Table
5. We can see that three of the four possible configurations, which represent six of the 12 studies, have sufficiently high raw consistency (≥0.75) to indicate membership in the set of highly effective interventions. Having refined our understanding of what makes a highly effective intervention and having identified some of the characteristics that distinguish them from less effective interventions, we then moved on to the next stage of the analysis.
Table 5
Truth table for model 3: intervention intensity and quality as the conditions, and ‘highly effective interventions’ as the outcome
1
|
0
|
1
|
1
|
1.000
|
1
|
1
|
4
|
1
|
0.923
|
0
|
1
|
1
|
1
|
0.750
|
0
|
0
|
6
|
0
|
0.389
|
Stage 3: Resolving contradictory configurations
Contradictory configurations are sets of studies in which identical configurations of conditions lead to different outcomes. These need to be resolved before the study can proceed as, by definition, they contradict one another in the truth table. They can be identified in crisp sets as consistency values that are non-uniform (that is, values other than 0 or 1). However, checking for contradictions in the context of a fuzzy set outcome is less straightforward than in a crisp set scenario, as studies can be partially in or out of the outcome set. Boolean algorithms help here when data sets are large, and their results can be displayed in the final columns of the truth tables.
In our dataset, we checked for contradictions by referring back to the original data table presented in Table
1. Of the four studies with the potentially contradictory configuration (as indicated by a non-uniform raw consistency) of intensity = 1 and quality = 1, three of the studies were full members of the outcome, while the fourth was ‘more in than out’ (as indicated by the outcome calibration of 0.666). As such, the studies do not contradict each other - they are all highly effective interventions, although the strength of their membership in the outcome set varies slightly. The other potential contradictory configuration of intensity = 0 and quality = 1 only consists of one study, indicating that the raw consistency is non-uniform because of its fuzzy membership in the outcome set, rather than contradiction with another study. We therefore do not have any contradictory configurations.
If contradictory configurations are evident, there are a number of steps that we can take to resolve them: 1) add one or more conditions to the table; 2) remove existing conditions and replace them with others; 3) re-examine the allocation of studies to particular conditions - including outcome - to ensure consistency in interpretation; 4) consider whether variation might be expected given the conceptualisation and operationalization of the outcome; 5) undertake more ‘qualitative’ analysis of the studies to see whether explanatory differences emerge; 6) consider whether the dataset is too heterogeneous; 7) recode contradictory configurations as ‘0’ in the outcome field - presenting them as ‘unclear’; or 8) undertake a ‘vote counting’ procedure, in which the configuration with the most ‘votes’ (studies) is the one on which conclusions are drawn. For further information on these techniques see Rihoux and Ragin (2009) [
7]. The decision we took in the worked example was to accept that the theories of change we had begun the analysis with did not distinguish between successful interventions (possibly because our interventions did not cover the full range of theories adequately); we therefore chose option 2, and replaced these non-distinguishing conditions for others which were able to discriminate between those interventions with highly successful outcomes and those which were less successful.
Stage 4: Boolean minimisation
At this stage of the analysis, the QCA software (for example, fsQCA [
29] or TOSMANA [
31]) utilises Boolean minimisation algorithms to analyse the truth table and identify the most logically simple expression of a Boolean formula. Since the purpose of the exercise is synthesis - to draw conclusions across studies - we would like to find solutions which encompass as many of our studies as possible.
Using the consistency threshold of 0.75 in our truth table for model 3 (Table
5), we are left with three rows to enter our analysis. According to these three rows, membership in the effective intervention set can be written as:
The Boolean minimisation algorithm will reduce the ‘solution’ of the truth table by identifying the fact that rows 1 and 2 differ only in terms of the presence of the ‘quality’ condition, while rows 2 and 3 only differ in terms of the presence of the ‘intensity’ condition. Since the outcome is ‘highly effective’ as long as either intensity or quality are present, regardless of whether both are present or not, the minimisation routine removes it from the solution, as illustrated in Table
6. Thus, the minimised solution can be written as ‘intensity*quality > Outcome’, where * indicates ‘Or’ (in plain English, this solution would read as ‘the presence of high intensity or high quality are sufficient for the outcome to occur’). The solution coverage of 0.714 indicates the proportion of studies with a highly effective intervention that have either of the two configurations, while the solution consistency of 0.833 gives the proportion of studies with either configuration that obtains the outcome of a highly effective intervention.
Table 6
Solution for model 3 with ‘highly effective interventions’ as the outcome
Intensity | .667 | .714 | .833 |
Quality | .619 | | |
Stage 6: Interpretation
Once the simplified solution has been identified, the final stage is to interpret the solution in the light of the studies they are based on, the review’s research questions, and the conceptual framework which guides the review. In our example, we find that we have good evidence for concluding that there are two main routes to a highly effective intervention: first, through an intervention where the intervention is high in intensity and second, when the intervention is high quality. A QCA synthesis may stop at this point, or it may go on to develop theory to explain its findings and to increase generalizable messages.
In the context of our original review, the QCA both challenges our starting assumptions and adds nuance. In terms of our overall conceptualisation of community engagement, this analysis suggests that our broad theories of change cannot explain why some interventions in this sub-set of studies obtained better results than others: it appears to be more important that women receive substantial support in the critical period pre- and post-partum. Thus, while our overall report showed that the theories of change examined to begin with are able to differentiate between interventions at a higher level of abstraction [
11], making finer-grained distinctions between the relative successes of outcomes from similar interventions requires a focus on other intervention characteristics beyond the type of community engagement utilised.