Does big data require a methodological change in medical research?

Scientific hypotheses arise through either deduction or induction. Deduction uses available knowledge, and sometimes empirical observations, to derive a new hypothesis by way of strictly deductive logic. A typical example in medicine is the expectation of a particular effect of a particular drug, based upon physiological and pharmacological theory. Induction, by contrast, draws solely upon past experience in that it combines current knowledge, actual observations, and subjective (as opposed to logical) reasoning to create new hypotheses. “We reason inductively when we infer that all Xs are Ys because all past observed Xs were also Ys.” [4] One consequence of this constraint is that universal statements (such as “All Xs are Ys.”) can never be verified by induction alone.

The above notwithstanding, inductive reasoning has played a central role in scientific research in the past, and its significance may well extend to big data. In fact, its conceptual relatedness to inductive reasoning explains why big data analysis is sometimes referred to as ‘hypothesis-neutral’ research [5]. On the other hand, since the use of big data in scientific research is often tantamount to agnostic searches of heterogeneous and unstructured data, the results therefore often lack the self-suggesting nature of the outcome of classical hypothesis generation.

Regardless of its origin, every newly derived hypothesis must stand up to empirical scrutiny. Even though this requirement applies to scientific research in general, it appears particularly appropriate for hypotheses arising from big data analysis because “a research finding is less likely to be true […] when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes” [6]. Undoubtedly, all these characteristics are more or less typical of big data. Moreover, since big data are ‘big’, they are easily misunderstood as automatically providing better results through smaller sampling error. That the gain in precision drawn from larger samples may well be nullified by the introduction of additional population variance and bias is then often not appreciated.

Popper highlighted that, for a hypothesis to count as scientific, it must be falsifiable [7]. His view has since become the formal cornerstone of statistical testing, where researchers aim to reject null hypotheses for the benefit of alternative hypotheses constituting their own scientific conviction. If a hypothesis cannot be falsified in repeated attempts, our belief increases that it might be true. Even then, however, the hypothesis is not proven because a proof would require use of external information. Note that this constraint only applies to universal statements (“All swans are white.”) which are, by far, the more interesting and important ones in medical research. Existence statements (“There are black swans.”) can of course be confirmed empirically.

To test universal hypotheses, researchers must design studies that render falsification possible or, even better, likely (Fig. 1). The chance of falsification depends critically upon the specificity of the study design, the relevance and representativeness of the study subjects, and the quality of the data. It is therefore not surprising that experimental studies are the gold standard of scientific research, especially in disciplines such as physics and chemistry, which are founded upon experiments. In medicine, however, experimental studies are often logistically, legally, or ethically problematic if not impossible [8]. In such instances, well-designed observational studies are the next best option for the evaluation of hypotheses. Both approaches are, however, fundamentally different from the analysis of big data that originate outside the realm of control by the researcher and, hence, lack approved quality. In consequence, the bulk of scientific hypotheses derived through big data analyses require follow-up and validation by classical scientific studies before they can be deemed practically or theoretically relevant (Fig. 1).

“Correlation supersedes causation” [9] is one of the mantras of big data and, indeed, detecting and quantifying correlations may be a worthwhile exercise in various contexts. An online company, for example, would greatly benefit from knowing that customers who buy product X are also inclined to buy product Y. Whether this relationship is causal or not is irrelevant for the company to be economically successful. Similarly, in scientific research, being able to accurately predict an outcome of interest from correlated measurements can be useful, for example, because this may (i) point towards unknown causal mechanisms, (ii) help refining existing models of causality and (iii) provide a ‘reality check’ to the relevance of theories [10]. Moreover, as Mayer-Schönberger and Cukier [11] noted, “correlations may not tell us precisely why something is happening, but they alert us that it is happening.” Not surprisingly then, an area of medicine where correlation matters is the prediction of disease outbreaks. Even though the identification of causal effectors is an important goal of epidemiology, from a public health perspective, indirect but reliable outbreak markers may already be of great practical value.

The above notwithstanding, it is safe to say that the majority view in medical research ranks causality more important than correlation. In the words of Pearl [12], “most studies in the health sciences aim to answer causal rather than associative questions.” If a certain treatment is associated with a certain side effect, it is critical to know whether this relationship is causal because, otherwise, a change of therapy may be in vain. Similar reasoning applies to epidemiology where, despite the uncontested usefulness of prediction, the identification of causal risk factors is an important cornerstone of sensible public health policies. In most medical settings, correlation is therefore only the first step towards a mechanistic understanding of disease development and therapeutic effects.

Causality cannot be inferred from observational data alone but requires additional information or knowledge. From a purist point of view, only “discovering an underlying mechanism proves the truth of the causal hypothesis in question” [13]. It should be emphasized, however, that this extreme position is neither helpful nor necessary. In epidemiology, in particular, causality can often be inferred from observational data with reasonable certainty [13] and the Bradford-Hill criteria, first proposed > 50 years ago, still provide a good guide to transforming correlation into likely causality by way of combining different lines of evidence [14].

Image analysis plays an important role in medical research where computer and magnetic resonance tomography scans, for example, easily qualify as big data, at least in terms of volume and velocity (two of the classical three Vs of big data, the other being variety). However, while processing of single images is an interesting research subject in its own right, particularly with a view to the use of machine learning and artificial intelligence techniques, the medically relevant content of an image often manifests at much lower levels of complexity. In the end, an image may comprise myriads of pixels, but is still one ‘measurement’. For instance, the task of classifying a mammogram as ‘tumour present’ or ‘tumour absent’ may require complex and computationally challenging algorithms, but the resulting diagnosis is essentially binary. Thus, inference making via medical image analysis may well draw upon big data, but most often it does not represent big data analysis sensu stricto.

In other research areas, particularly in clinical medicine, epidemiology and public health, big data are only of low to moderate quality because they are observational, uncontrolled, and only seemingly complete, of mixed origin and were not specifically collected for research purposes. In the context of biomedical research, Leonelli [15] identified three stages of the development of big data, namely de-contextualisation, re-contextualisation, and re-use. The author observed that “big data that is made available through databases for future analysis turns out to represent highly selected phenomena, materials and contributions. […] What is worse, this selection is not the result of scientific choices. […] Rather, it is the serendipitous result of social, political, economic and technical factors, which determines which data get to travel in ways that are non-transparent and hard to reconstruct […].” Note that this type of provenance is fundamentally different from what can be achieved in carefully designed clinical trials or experimental studies.

For big data, the famous idiom “garbage in, garbage out” may easily worsen to “big garbage in, big garbage out”. Data quality is an issue in medical research because of a frequent lack of standardization and structuring. Hence, any sensible use of big data in medical research critically depends upon the extent to which these data are ‘good data’ in the sense that everything possible has been done to sustain their quality. A fourth letter V was therefore proposed as an amendment of the three Vs of big data. The fourth V stands for veracity and for the challenge to ensure adequate care and diligence in big data analyses [16]. It is also worth emphasizing that small is sometimes beautiful in that ‘more data’ does not always imply ‘more information’, and that data reduction can be a sensible way to enhance data quality.

It seems paradoxical at first glance to regard insufficient sample size as a potential problem of big data. However, ‘big data’ often refers to large numbers of attributes but not necessarily to large numbers of independent observations of these attributes. In most commercial applications, such a limitation does not play a role because the data in question are gathered opportunistically from thousands of people anyway, usually as a by-product of internet-based services. In medical research, however, big data may comprise millions of attributes measured in a few patients, typically with the aim to provide them with ‘personalized medicine’. Schork [17] advocated this approach in a Nature commentary entitled “Personalized medicine: Time for one-person trials”. His views reflected the author’s particular professional interest in genomics, where millions of DNA variants are assessed in individual patients. However, it is a wide-spread albeit unfortunate misunderstanding that such ‘in-depth’ investigations of small samples ever lead to generalizable results. Instead, they should be deferred to the realm of case studies. The latter, however, are known to represent a low evidence level only and therefore have confined themselves mostly to hypothesis generation since the advent of evidence-based medicine in the 1990s.