DEBATE-statistical analysis plans for observational studies

Unlike randomised clinical trials that calculate a sample size to study an intervention effect taking power into consideration, the sample sizes of most observational studies are influenced by other factors (e.g. resources, time restrictions, convenience). Accordingly, most observational studies will have a given sample size and, if sufficiently large, affording enough power. The STROBE guidelines only expect authors to explain how the study size was arrived at [8], which may reduce the incentive to conduct sample size calculations for observational studies.

When there is a given sample size or if a sample size was not specified in the protocol, we suggest providing power considerations for the primary analysis of the observational study to limit random errors. The power considerations necessitate a definition of a minimally important difference or intervention effect in the presence of a given sample size. Any power calculation provides the chance of a type-II error (false negative findings), while a detectable difference may be clinically more informative. For example, it shows the minimal relative risk that can be detected with the specified power and sample size given a type I error probability α.

While causality can never be proven in observational studies, observed associations may fuel hypotheses that later can be tested in randomised clinical trials [18]. Although the vast majority of observational studies test for superiority, there are some that address equivalence and non-inferiority hypotheses [19‐22]. Of course, confounding will always be present in any of these frameworks. Nevertheless, a SAP should describe whether the relevant hypothesis was assessed for superiority, equivalence or non-inferiority.

Interim analyses are typically known to guide randomised clinical trials for early stopping due to benefit, harm or futility of tested interventions. Investigators are ethically obliged to conduct interim analyses to reduce study patients’ exposure to an inferior intervention. While there is usually no intervention component in observational studies which can be halted, there may be incentives to perform interim analyses for early stopping of continued (costly) data collections due to already clear observed associations or futility. Furthermore, observational studies may be subject to repeated testing of accumulating data, which needs adjustment of significance levels to reduce inflated type-I errors (false positive findings), such as those described by O’Brien & Fleming [23]. Such methods should be described in the SAP.

A SAP for a blinded clinical trial should be published prior to unblinding the data or prior to the randomisation of the first participant in case of an open clinical trial. Likewise, a SAP for prospective observational studies should also be published before the first participant is included or at least all access to the database should be restricted. Randomised clinical trials that include blinding have a natural advantage that interventions can be coded during the statistical analyses. Such coding of interventions is usually not in question in observational studies, but it should be possible to mask the statistician by using coding for several covariates (at least dichotomous and categorical). Except for the study monitors, researchers should be unable to read the database before the study is finished or a SAP is written. If all study data were accessible to the researchers, a detailed SAP may still provide transparency on the intended analytic steps and may prevent ‘fishing’ for statistically significant predictors in analyses or other manipulations of the data. Any analysis that was not prespecified in the protocol and/or the SAP can only be explorative in nature, which should be described accordingly (i.e. exploratory or post-hoc analysis).

Multiplicity issues are similar in randomised clinical trials and in observational studies, but rarely addressed in the latter. Most observational studies ignore multiplicity issues by testing in multiple analyses at the same conventional P < 0.05 significance level. This increases the risks of a family wise error rate (FWER), that is the type I error of at least one false positive finding. Several methods have been suggested to adjust for multiplicity, such as those according to Bonferroni or Šidák [24, 25]. Even though International Conference on Harmonization of Good Clinical Practice guidelines recommend full Bonferroni adjustment [26], such an adjustment may be too conservative in correlated outcomes of observational studies [27].

For example, the SICS-I addresses six different primary outcomes spread out across 13 hypotheses [16]. Our outcomes cardiac output, acute kidney injury, and mortality are all affected by a patient’s haemodynamic status, so that most outcomes will probably be positively correlated. Since the Bonferroni adjustment assumes that outcomes are unrelated, we used an adjustment of our significance level that was pragmatic and probably more accurate. For more details we refer to the paper by Jakobsen and colleagues [28].

It is necessary to elucidate the numbers of eligible and included patients of an observational study in a flow diagram, preferably according to the STROBE recommendations [29].

Results of observational studies can be seriously biased by confounding covariates. The randomisation procedure is used in randomised clinical trials to reduce the imbalance in observed and unobserved confounders between the allocated groups, although success can never be guaranteed [30]. The STROBE guidelines advocate to address the rate of confounding; however, it was recently shown that adherence to this statement is suboptimal [31]. A SAP could serve to predefine confounders, and how to address the expected rate of residual confounding by adjustment, or stratification.

Confounding variables are key important to address in observational studies. Usually, datasets of observational studies include large amounts of variables with many inevitably correlated to each other. For example, the SICS-I database contained 19 clinical examination findings which all reflected (a part of) the haemodynamic status of a patient. Next to expected confounding factors, the values of the variables can also be confounded by unmeasured factors such as environmental, genetic, or psychological influences. Therefore, we suggest to provide an a priori list of potentially confounding variables (both ‘measured’ and ‘unmeasured’) so that the reader is better able to assess the degree of residual confounding. Prelisting all potential variables and the approach to model building should be a main concern, if not the most important issue, in the SAP of observational studies.

Analysis methods of clinical trials and observational studies are different, yet both study types are suspicious of selective reporting when no SAP is written [32]. Many decisions are needed during the analysis phase of an observational study and all that can be foreseen should be prespecified. An extensive description of the planned statistical analyses, all covariates, and all considerations need to be prespecified and detailed, which can only be done in a SAP. The usually short statistical analysis section of a manuscript does not allow a detailed explanation, nor can it guarantee the prespecified status of the analysis.

The cost- and time-intensive nature of a randomised clinical trial necessitates a strict protocol in which all sensitivity and subgroup analyses are (usually) specified. In observational studies, these additional analyses are seldomly specified beforehand. A SAP is an opportunity for authors to prove that they had prespecified intentions of their sensitivity and subgroup analyses.

Observational studies are particularly prone to missing data, but often do not address the mechanism of missing values. Complete case analyses in the presence of missing data are associated with bias, when data are not missing completely at random [33, 34]. Tests to identify the patterns and type of missing data, and the statistical methods to handle missing values should be described in a SAP. Examples include multiple imputations for data missing at random or worst-best and best-worst case scenarios for data missing not at random [34, 35].

Randomised clinical trials are costly and therefore often limited in size and length of follow-up, so that rare harms or late harms (e.g. after decades) remain undetected. Observational studies and post-marketing phase IV randomised clinical trials are much more suitable for detection of rare or late harms [35], of which the cardiovascular harms of clarithromycin in patients with stable coronary heart disease or cyclooxygenase-2 (cox-2) inhibitors are good examples [36, 37]. This item only applies to observational studies with a research questions focusing on an intervention effect. Our SICS-I cohort, for example, was not designed to study such associations.