Causal inference concepts applied to three observational studies in the context of vaccine development: from theory to practice

Since the beginning of this century, an unprecedented amount of scientific research has been conducted on causal inference, the process of assessing causality between exposures and outcomes. Many epidemiological studies are observational in their design, and unlike randomized controlled trials (RCT) where similarity of groups can be experimentally attained, comparability of groups can be difficult or even impossible to demonstrate. This renders causal inference in these cases challenging and conditional on unverifiable assumptions.

To infer causality from association, Sir Austin Bradford Hill synthesized what he called “aspects of association” [1], consisting of 9 distinct criteria that can be used separately or in combination to gather evidence on causal inference. These criteria are known as ‘Hill’s criteria’ and have been extensively used among epidemiologists ever since. More recently, other elaborate frameworks for causal inference have been developed [2‐4], stemming from graph theory and counterfactual theories of causation. The counterfactual framework published by Rubin, 1974 [5], led to the definition of three general conditions needed to draw causal inference; exchangeability, consistency and positivity. Causal Diagrams in the form of Directed Acyclic Graphs (DAGs), summarize the assumed relationships between all variables that are relevant to the causal analysis and can be used to detect confounding and selection bias, and to develop insights on how to adjust for them in data analysis or in the study design. We assessed the applicability and limitations of these approaches in drawing causal inference in observational studies in the context of vaccine research.

Vaccine studies have some particularities that may influence causal analyses. For example, vaccination not only protects the vaccinee but can also reduce the transmission of contagious disease in the unvaccinated population (herd immunity). RCTs measure the effect of the vaccine at the individual subject level to demonstrate the vaccine efficacy for approval by health authorities, whereas epidemiological studies are needed to measure the effect of vaccination (effectiveness and impact) at the population level [6]. We assessed how classical and more recent causal inference methods, namely Hill’s criteria, counterfactual framework, and causal diagrams, are applicable in vaccine research, using examples from epidemiological studies.

We applied Hill’s Criteria, counterfactual reasoning, and causal diagrams to three published studies conducted by GSK: one burden of vaccine target disease (BoD) study, one post-authorization safety study (PASS), and one vaccine effectiveness (VE) study. Based on the published results, for each study Hill’s criteria and counterfactual reasoning were rigorously discussed and assessed by the authors in several meetings until a consensus was reached. No additional analysis was performed during this exercise. Causal diagrams were drawn to depict the exposure, outcome and other factors for each study using standard symbols [7]. The selection and matching factors were symbolized as factors associated with S = 1 [Selection]. Other covariates controlled in the analysis were depicted as associated with both exposure and outcome (Fig. 1).

The applicability of the three major components of causal inference are described below and in Table 1 and in Fig. 1.

We assessed the applicability of Hill’s criteria, counterfactual reasoning to investigate the plausibility of causality in three observational vaccine studies. In addition, we described the relationship between exposure, outcome and other factors using causal diagrams. Based on these assessments, none of the studies allowed us to formally demonstrate causality. Nevertheless, we did conclude that a causal association in the VE study was very likely, based on satisfying 8 out of 9 Hill’s criteria and demonstration of some degree of exchangeability attained due to controlling for known confounders.

Hill’s criteria [33‐37] and causal diagrams [38, 39] have been used previously in the design and interpretation of observational studies, whereas evaluation of the assumptions referred to as identifiability conditions by Hernan and Robins [2] in the counterfactual framework is often done indirectly in the discussion on confounding as part of the study limitations in observational studies. Interestingly counterfactual reasoning introduced the main assumption of exchangeability which was not reflected in Hill’s criteria. The originality of our approach is that we combined all three components and found flaws in all three methods in their applicability to real-life examples of observational studies in the context of vaccine research.

At the time of Hill’s publication 50 years ago, his criteria were not meant to be used be used as a rigid checklist of evidence for causation. However, they have often been used as such. Moreover, their interpretation has changed over time as a result of major advancements in several scientific disciplines, analytical tools and access to big data [40]. As a consequence, some authors have called for revisions to Hill’s criteria [41, 42]. Several criteria are subject to interpretation, for instance the “experiment” criterion, or are difficult to differentiate from each other, such as “plausibility” and “coherence” [43, 44]. In addition, not all Hill’s criteria are applicable or quantifiable for each study type. For example, the “biological gradient” is usually not assessable in vaccine studies because the vaccine dose is fixed. Also, the criterion “experiment: removal of the exposure” is rarely applicable in observational vaccine studies. Some researchers have recommended an evaluation of confounding factors in addition to Hill’s criteria [45]. A good understanding of what factors cause confounding or selection bias can be difficult in complex (possibly longitudinal) designs. Therefore, causal diagrams can be helpful to gain insight via visual depictions of the relationships between factors, along with graphical tools to assess bias. In addition, counterfactual reasoning provides a formal framework for integrating this notion of confounding in the empirical assessment of causality.

With respect to counterfactual criteria, exchangeability is plausibly attained in intention-to-treat analysis of randomized controlled trials, but generally untestable in observational studies. However, some designs can add support to exchangeability in observational studies, such as self-controlled case-series where the subject is his/her own control. Positivity criteria are met within observational study designs where, within each subgroup defined by the adjustment factors, there is a positive probability of observing exposed as well as unexposed subjects. Consistency is usually met when exposure is a well-defined intervention such as vaccination. However, in many observational studies, exposure is not an intervention but a condition, such as body mass index, or as in our example, DM2. Since these exposures have no immediate connection to a well-defined intervention, consistency as defined in counterfactual reasoning is not applicable. The implication is that even if the effect inferred from such study was causal, caution is needed when interpreting such an effect (e.g. the effect of diabetes) as there is no clearly defined strategy that would e.g. cause or prevent diabetes. Moreover, subjects exposed (or unexposed) could be extracted from large databases with coded data, as was done in our BoD study where DM was defined using International Classification of Diseases (ICD) codes. A systematic review of case definitions for diabetes using ICD-9 and ICD-10 codes showed that coding variations and institutional practices for medical record data extraction could significantly alter the performance of different case definitions used in observational studies [46]. In addition to exchangeability, positivity and consistency, several authors recommend other conditions. Rubin’s Stable Unit-Treatment-Value Assumption (SUTVA) includes the assumption of no interference [47]. In the VE study, the validity of this assumption could be in doubt because the unvaccinated subjects can benefit from an indirect effect of vaccination via herd immunity. As a consequence, the estimand of that study was the direct VE and not the total VE [6].

We used causal diagrams to describe the design of studies, including all known confounders that were considered for adjustment. These diagrams visualize the assumptions made; unknown confounders and known but unmeasurable confounders were therefore not depicted. In observational studies, both exposure and outcome may be measured with error. Since no measurement error corrections could be made, measurement error was not incorporated in our causal diagrams.

Several publications have described the basic principles and recommendations for drawing causal diagrams [48]. However, there is still a lack of agreement in how to depict some observational study designs, such as matched case-control designs [49, 50]. Despite some proposals to depict effect modifiers in causal diagrams [51, 52], the fact that effect modification is scale-dependent implies that effect modifiers cannot be successfully incorporated into causal diagrams, which are designed to be model-free. In spite of these limitations, causal diagrams are a useful tool for identifying variables that should be measured and controlled for at study design [48, 53], and for interpreting the results of analysis [39].

In the three selected studies, we assessed the causality between a single exposure and a disease: DM2 and HZ risk, HRV vaccination and hospitalization for rotavirus gastroenteritis, and AS04-HPV-16/18 vaccination and AD. In this context, it is worth pointing out that the etiology of most diseases is multi-factorial. For instance, decreased levels of VZV-specific immunity are known to lead to the reactivation of the latent VZV that subsequently results in the clinical manifestation of HZ [8]. However, there are numerous causes of waning immunity including advancing age, comorbidities, immunosuppressive treatments, etc. Thus, even in the case of a well-defined medical condition and biological cause, a set of complex contributing mechanisms and interactions can occur. In contrast, the PASS study included a composite endpoint (19 different diseases were defined) and the etiology of many ADs is only partially elucidated. Moreover, many or most ADs are a result of multiple factors that can interact or co-exist, including genetic predisposition and potential environmental triggers [43].

Observational studies never account for all possible factors that lead to an outcome; there are always unknown factors or unmeasurable known factors which can be confounders. By comparison, in RCT the unknown or unmeasurable factors are, by design, expected to be consistently distributed in the treated and control groups.

Our analysis was intentionally limited to assessing how causal inference approaches can be applied to observational studies in the context of vaccines. We reached a consensus to include three studies assessing different objectives frequently addressed in vaccine clinical research; (vaccine effectiveness, burden of disease and safety) which we considered to provide acceptable diversity in our examples. The objective was not an exhaustive review of the various causal inference methods or of recent developments in causal inference and analytical methods such as Inverse probability weighting, G-formula, G-estimation, Instrumental variable estimation, and so on [54]. This will be the next step of our research using both real data and simulations.

Our review was limited to three study designs. Nevertheless, our analysis can be used to guide observational study design in the context of causal inference. Assessment of potential causal effects using real data should start by depicting the existing scientific knowledge about a clearly defined research question. Exposure, outcome and confounders need to be explicitly defined and causal diagrams developed to visualize the relationship between these three major elements including the unmeasured confounders. Conditional exchangeability could be reached provided that the confounders are controlled either by study design or by adjustment of the effect estimate during the analysis. Standard methods to assess causality in observational studies (for example, propensity score-based methods, inverse probability of treatment weighting, multivariable regression, etc.) require the assumption of no unmeasured confounding. In our studies, we did not consider possible unmeasured confounders but these can occur in real-world data and therefore need to be addressed. Methods have been developed to control for them [55], but these require alternative assumptions for identification (e.g., that certain measured variables are so-called ‘instruments’). Interestingly, a recent revision of the European Medicines Agency pharmacoepidemiology guidance provides extensive recommendations to address confounding [56]. Finally, standard statistical software (for example SAS or Stata) now include procedures for applying different causal inference methods, which prevent the extrapolation bias to which standard regression methods are prone [57].

In conclusion, the application of Hill’s criteria and counterfactual reasoning in three observational studies showed that definite causality could not be demonstrated. However, these approaches allowed us to determine the level of certainty of causality, in each study and evaluating it from very unlikely to very likely. The causal diagrams appeared to be a complementary tool which can help in designing observational studies and interpreting results of analysis. Even if a causal effect could not be demonstrated using only observational data, these studies have real value in decision making in public health, being complementary to RCTs which are not always feasible or ethical. Moreover, major scientific advances, such as Darwin’s development of the theory of natural selection, were generated entirely from observational data, highlighting the critical contribution of such data to scientific thought.