This study presents several innovations that help to advance the use of QCA as an evidence synthesis method. First, the QCA drew on a theory developed from the observations of trialists themselves, from the ‘ground up’ and akin to a grounded theory approach. Previous QCA syntheses of systematic review findings have either necessitated drawing on intervention theories derived from logic models with syntheses of process evaluation studies [
64], or other separate in-depth qualitative evidence syntheses [
19]. The findings here suggest that, in the absence of extant intervention theory or pre-existing synthesis, that working/pragmatic theories can be developed to support QCA synthesis from experiential evidence that is usually overlooked in other synthesis methods, using an ICA framework. Second, this study showed that a theory of how interventions ‘work’, developed through the synthesis of one set of studies using QCA (i.e. the hard mandate studies), can be applied to a conceptually congruent set of separate studies (i.e. the soft mandate and other intervention studies). This form of triangulation can represent a useful adjunct to QCA analyses in systematic reviews that could help to create more robust syntheses in the future. Third, the study also provided a comparison between using fuzzy-set and crisp-set coding schema on the same dataset (hard mandate studies). While similar results were obtained, again providing a further degree of triangulation, the fuzzy-set coding for the hard mandate studies was a more appropriate choice conceptually. This was with respect to both the coding for the outcome, where all the studies had obtained significant reductions in unvaccinated (despite heterogeneity in the original meta-analysis [
9]), as well as the conditions, where in the case of ‘don’t go in cold’ in particular, different levels of previous engagement were apparent among some hard mandate studies in a way which wasn’t as apparent for studies on soft mandates and other intervention modes. Fourth, this is the first example that we are aware of where ‘publication type’ was included in the analysis and was predictive of outcomes. This work thus provides some evidence in support of one issue that’s been long suspected in systematic reviews: that the lack of information in some papers / publications can lead to unreliable review results – and possibly undermine other subgroup analyses [
22]. Finally, this study once again is further demonstration of the potential for further adjunct analysis of evidence that has already been assembled and synthesised in some form, to address new questions and generate new understandings. This study drew on ICA/QCA; other techniques for the reanalysis of existing review evidence have also been suggested elsewhere [
65]. Given the large volume of systematic reviews being published annually, each requiring substantial investment and sometimes generating conflicting results or interpretations, techniques for further probing of the included studies to provide additional nuance or address questions not considered by the original reviewers, may continue to develop as a promising adjunct stream of evidence synthesis.
While the analyses presented here are of importance, both in (i) revealing some of the conditions sufficient to result in successful influenza vaccination campaigns: as well as (ii) emphasising the potential of ICA/QCA in enhancing our understanding of existing review evidence, some limitations should be noted. An important limitation is around the approach itself and its capacity to consistently and correctly reveal complex causal relationships. There exist some critiques around the potential of QCA to produce correct solutions in simulated data sets for which true causal processes are known [
66], although responses provided by others not only highlight flaws in these critiques, but also emphasise that a QCA solution cannot be generated and articulated in the absence of case and substantive knowledge [
67]. While we regard the use of ICA to generate theory to underpin QCA as a useful innovation in the field; we nevertheless recognise that trial reports remain sparse in terms of reporting intervention details [
12], and despite the allowances we made for sparse reporting in letters, ‘missing data’ may be a further caveat on the results. However, a strategy analogous to a ‘‘complete case” approach sometimes used in statistical analysis, where only cases reporting either the presence or absence of a condition are included in the analysis, would not have been appropriate here and would have led to no viable cases being identified. Instead, we have assumed that where a process or intervention component was not reported, that it did not take place; this is a strategy mirrored across the whole of the systematic reviewing literature. The inclusion of a condition for article type (with a letter indicating a short empirical report) reflects that some article types are constrained in the detail that can be published such that some processes that did take place are at risk of going unreported. This is not a marker of study quality per se but of reporting quality and was used in this case to understand and explain why some cases with a low number of active components and processes were successful (contrary to expected theory). The inclusion of the reporting style as a condition does represent a potential limitation as it requires a different interpretation to the other conditions (explanatory not causal). Another potential limitation is the relatively low ratio of cases per conditions, particularly for the hard mandates model, which generated a number of logical remainders which could undermine the soundness of our conclusions [
56]. We based our chosen conditions on our working theory of how the intervention was expected to work, and selected the number of conditions based on guidance proposed on the ideal balance of cases:conditions [
53], which was later investigated more thoroughly [
56]. The results of investigations conducted by Marx and Dusa [
56] suggest that while the number of conditions (4) is relatively high for an intermediate dataset of 11 studies, it remains on the borderline of acceptability. Nevertheless, limited diversity and the relatively high number of unobserved potential configurations is a limitation of this model in particular. [
53]Finally, while we generated an enhanced intermediate solution as proposed by Schneider and Wagemann [
58], and following procedures developed by Duşa [
57], the treatment of logical remainders somewhat contested and unresolved in the literature [
68,
69]. Thus, even though some have suggested a growing consensus in support of the prioritization of intermediate solutions [
70,
71], this could represent a final caveat to these results. However, since QCA requires that the solution is consistent with a programme theory that is identifiable in all relevant cases, it can be seen, in some ways, as having a higher bar for achieving a credible explanation than statistical analysis. In a statistical analysis, deviant cases might increase variance / widen confidence intervals, but are considered ‘explained’ when this happens. In a QCA, a deviant case indicates that a credible solution that properly explains what is going on has not be found, so further analysis is required. As such, given that we identified consistent patterns of association across several independent research studies and that the detail of each case was consistent with our ‘leading from the front’ theory, the credibility of these findings is strengthened.