Prediction or causality? A scoping review of their conflation within current observational research

From an epidemiological perspective, clinical studies are often classified as having either a descriptive, etiological or predictive aim. In the current study we focus on etiology and prediction research. In etiological research the typical aim is to uncover the causal effect of a specific exposure (factor) on an outcome. The results generally help us answer ‘what if’ questions about treatment or management and are imperative in furthering our understanding of the mechanisms of disease. The gold standard to do so is traditionally a randomized experiment. However, this is often not feasible and advancements have been made towards undertaking causal inference from observational data [1]. A crucial part of observational causal studies is correction for confounding (variables which influence both the exposure and outcome and muddle the causal relationship). The data itself cannot tell us which variables give confounding, knowledge and assumptions on the underlying causal structure is necessary.

In prediction research, the goal is a model that utilizes multiple factors (‘predictors’) in combination to accurately predict an outcome in individuals to assist in diagnosis or prognosis. This is usually irrespective of whether included predictors are causal or not. The main focus of prediction model studies is the overall predictive or diagnostic performance of the model which should also be assessed in new patients (validation) [2]. A prior step to model development may be to identify novel predictors that have added predictive value when added to known predictors [3]. For a more in depth explanation on etiology and prediction research see the supplemental material. In the case of counterfactual prediction modelling (which answers ‘what if’ questions on prognosis related to interventions) prediction and etiology intentionally collide [4, 5]. Though an important advancement, such studies are rare, require specialist techniques and are not the focus of this article [6].

Both etiological and prediction studies may be performed on the same observational data, but the underlying research question, methods and interpretation of results are usually different [7]. Unfortunately, when reading and reviewing the medical literature, we have found that these aims, methods, results and interpretations are often confused, leaving us with studies that no longer answer a clear etiological or prediction question and may be misinterpreted. In consultations with researchers, we’ve noticed that the distinction between prediction and etiology is often not clearly stipulated and therefore not considered in the research question or proposed statistical approach. The century-old decree that causal inference cannot be made from observational data has resulted in many causal observational studies using vague terminology and an apprehension to tackle causal questions explicitly [8]. The conflation between causal research and prediction research is also common within statistics and data science [9, 10]. Although conflation of etiology and prediction appears anecdotally to be frequent in medical research, the extent of this is unknown. If the frequency and consequences of this conflation are better understood, this may promote change and awareness in medical research practice and scientific education.

Therefore, our research aim is to quantify the frequency of conflation between etiological and prediction goals, analysis approach and interpretation in observational studies from various medical research fields. Furthermore, we aim to discuss common mistakes underlying this conflation, elaborate on the hazards that lie in these mistakes, and provide recommendations on how to recognize and avoid them.

In this scoping review which sampled observational studies from top journals in 6 medical fields, we found that conflation between etiology and prediction occurred in 46 out of 180 included studies (26%) and was more common in prediction studies. The frequency of conflation varied per medical field and journal impact factor, with less conflation in general & internal medicine and epidemiology journals. In causal research, the most frequent type of conflation was selecting adjustment covariates based on their ability to predict the outcome, rather than based on causal reasoning or pathways. In prediction studies, the most frequent form of conflation was a causal interpretation of predictor effects, instead of referring to their added value for risk prediction or overall model performance.

The identified conflation between etiology and prediction could easily lead to incorrect estimations and erroneous conclusions. If confounders are not accounted for in etiological research, effect estimates are most likely incorrect [17]. Specifically, in medicine it is important to determine causal factors correctly, as they might go on to be the target of novel pharmacological studies, be incentive for randomized controlled trials, or be incentive to change clinical patient care. If incorrect causal claims are made based on prediction models, this could have similarly detrimental effects on patients and further research efforts [9]. In our scoping review, 30% of prediction studies interpreted included predictors causally for instance by suggesting modification of a predictor to improve a patient’s prognosis. This is a dangerous misinterpretation; since predictors in no way need to be causally associated with the outcome, these studies cannot conclude that an individual’s prognosis would change if these predictors were to be modified. Imagine a prediction model of mortality, in which medication use is a predictor associated with a higher risk of death. Though medication use is modifiable, this does not mean that individuals should discontinue their medication in order to live a longer life. Though this example is rather obvious, similar mistakes frequently occur. For instance, an included study presenting a dementia risk score, concludes that a high BMI is protective. These conclusions may mislead readers into thinking obesity has health benefits [16]. Other identified conflation types, such as the presentation of performance measures for causal models or the recognition of residual confounding as limitation in prediction studies, probably have fewer directly harmful effects but constitute poor research methodology. Performance measures have a very limited role in etiological research and the concept of confounders or residual confounding is not appropriate in prediction studies that are not designed or aimed at uncovering a causal effect. We believe reducing conflation between etiology and prediction will improve the quality of observational research and lead to better and more efficient science.

Many excellent papers and books have been published on how to conduct both etiological and prediction research on observational data [18‐23]. If these guidelines are followed precisely, conflation between etiology and prediction is unlikely. However, few studies explicitly tackle the differences between etiology and prediction. A study from Zalpuri et al. from 2012 surveyed 435 attendees of an international transfusion conference on the interpretation of a multivariable stepwise logistic regression model. In total, 40% of attendees thought that a stepwise model was a valid method to adjust for confounding and 60% of attendees agreed with a causal interpretation of a stepwise prediction model [24]. An important paper by Shmueli does discuss explaining versus prediction from a statistical viewpoint and concluded that the statistical literature lacks a thorough discussion of differences between prediction and causality [10]. A recent paper by Hernan et al. entitled ‘a second chance to get causal inference right’ contains a plea to data scientists to integrate causal and prediction research questions (and their differences) in their curricula and analysis framework [9]. In previous work we also addressed differences between prediction and etiology and discussed related common pitfalls that arise [7]. The current study has empirically confirmed these pitfalls, quantified how frequent they are, and identified additional types of conflation.

There are various factors that may contribute to conflation between etiology and prediction. We have formulated some general recommendations for researchers, research-institutes, journals and policy-makers (Table 3) to ensure a clear distinction. For researchers, one of the most important recommendations is to clearly define the research question (including whether it is causal, predictive or descriptive). If the research question is unclear this ambiguity frequently continues throughout the rest of the article [25]. Once the aim is clear we suggest consulting appropriate reporting and methodological guidelines as well as methodological experts. For observational causal studies we recommend the STROBE guideline and the accompanying explanation & elaboration paper [18]. For diagnostic and prognostic prediction studies TRIPOD is recommended [19, 20]. Terms such as ‘predictor’ are frequently used in both causal and predictive studies. Though this in itself does not constitute conflation, it may cause uncertainty on the aim of the study. We would suggest reserving the terms ‘risk factor’, ‘causal factor’, ‘exposure’ and ‘confounder’ for causal research and the terms ‘predictor’, ‘prognostic factor’, and ‘prediction’ for prediction studies. The word causation is often avoided in observational studies, and sometimes even forbidden by journals [8]. Though no study can definitively prove causation, lack of clarity regarding the research goals has negative effects on the quality of observational research [8]. Finally, it would be beneficial for journals to include methodological experts in the peer review and editorial process. Our signaling questions can be used by reviewers or in systematic reviews to quantify the risk of conflation between etiology and prediction in observational studies.

Machine learning techniques are methodological advancements that have large potential in prediction research to analyze complex data structures [26]. However, the lack of transparency and black box nature of these methods complicates independent validation, updating and implementation. Therefore, it is imperative that such studies adhere to the same methodological guidelines as regression based prediction studies [26]. The more recent proposed use of machine learning in causal inference is not straightforward and can easily introduce conflation. Similar to data-driven selection methods, machine learning algorithms cannot distinguish mediators from confounders or recognize bias; the researchers’ knowledge and input on the causal structure remains crucial [27].

Though we have argued that variables in a prediction model should not be interpreted causally, many prediction models will benefit from including (previously identified) causal factors as predictors. This may help improve transportability of the model to new settings or different populations and can improve credibility and uptake of the model. Some might even go so far as to say that the ultimate prediction model—though unattainable—would contain all and only causal pathways of a condition [28]. On a less philosophical note, there are research questions for which prediction and etiological methods should be combined. The rather new approach termed counterfactual prediction per definition intertwines prediction and causation. Such studies aim to predict an individual’s prognosis for multiple scenario’s, while only one of these scenario’s is observed in the data (per individual). In particular, development datasets will typically observe outcomes in the context of current care, which includes current treatment and monitoring procedures. Thus, counterfactual prediction is then needed to make risk predictions in the hypothetical scenario where individuals receive a different treatment or care. This allows the comparison of multiple predicted risks for various treatment pathways within one individual.[4‐6, 29] Such research questions require causal inference methods such as inversed probability weighting when constructing a prediction model. It is worth mentioning that such counterfactual prediction studies are currently rare, and indeed were not encountered in our review, but they should not be seen as undesirable conflation.

The current study has a number of limitations. First of all, we included a limited sample of observational studies from 2018 which might not be representative of other years, journals or medical fields. It is worth noting that all recommended reporting guidelines were published before 2018. The median impact factor of our sample was relatively high (9.5) and it is likely that the amount of conflation is higher in journals with lower impact factors, which was a trend we also observed within our sample. Secondly, the data-extraction was performed by a single researcher and a second assessor was only consulted when the first assessor was unsure. The assessment will invariably contain some subjectivity, particularly whether heavily conflated articles are mainly etiological or predictive in aim. Furthermore, we could not assess whether relevant confounders were included or excluded for all included etiology studies, as this requires subject specific expertise. Importantly, our list of signaling questions is not exhaustive and may not pick up on all conflation. Conflation may not always be reflected in used terminology and studies labelled unclear may actually have been conflated. Additionally, conflation between prediction and etiology is only a small part of assessing a study’s methodological quality. A study which does not confuse prediction and etiology may still be sloppy or incorrect and some elements that we termed conflation, others might call poor etiology or prediction practices. Similarly, not all conflation leads to wrong conclusions. Finally, not all observational clinical research can be classified as either etiology or prediction and we appreciate the broad spectrum of forms that may lie between these two or completely outside of these realms.

In conclusion, undesirable conflation between prediction and etiology is common in medical observational studies and may present an obstacle to scientific progress. Researchers and readers should be mindful of the differences between etiology and prediction, to prevent biased estimates, erroneous conclusions and steering research in an inefficient or wrong direction.