Big Data: transforming drug development and health policy decision making

Incontrovertibly, randomized controlled trials (RCTs) are considered the “gold standard” for generating evidence relating to the comparative risks and benefits of alternative treatment options. Indeed, they hold a prominent place in the hierarchy of evidence, principally because of the role of randomization in ensuring internal validity, ruling out the effects of potential confounding factors (Barton Barton 2000). However, RCTs also have their own limitations. First and foremost, they lack external validity, since the circumstances under which they are executed may not be reflective of real-world experience. Further, for operational or ethical reasons, it may not even be feasible to conduct them. Consequently, there is a growing interest in complementing or buttressing evidence from RCTs using data from other sources, including observational studies, claims and electronic health record (EHR) databases, or other nonstandard and nonconventional information sources.

In the context of the tradition drug development paradigm, the availability of data from observational studies or other sources can seamlessly be integrated in the various phases of clinical research, ranging from supporting proof of concept (PoC) study designs to assessing the safety profile of an approved drug in a post-marketing setting.

In early phase trials, where information needed to characterize the safety and efficacy of a new medicine may often be deficient, data from secondary sources may help to define endpoints, characterize study populations and shed light on relevant aspects of other relevant design parameters. For example, for studies designed in a Bayesian framework, informative priors for a placebo or a well-established control group may be determined from observational data to gain efficiencies (Walley et al. 2015).

This is particularly germane to rare disease research, which is characterized by the small size of the target population, and poorly understood natural history of the disease (Augustine et al. 2013). In many cases the diseases are life-threatening and may not even have any drug development precedent. Data from observational studies can positively help mitigate some of these problems, including development of outcome assessment tools, characterization of disease presentation and disease progression, and better understanding of natural histories.

In the recent literature, much attention has been paid to the role of data from secondary sources in the characterization of the risk and benefit of drugs. Since RCTs used for registration purposes are typically underpowered to reveal safety signals for rare events, there is a lot of focus on evidence generated based on observational data (Finkle et al. 2014). Indeed, the FDA Sentinel Initiative relies on a distributed real world data network (Robb et al. 2012). Using data drawn from queries entered into Internet search engines, evidence of unreported prescription side-effects were detectable before they were found by the FDA’s warning system (White et al. 2013). Moreover, Big Data mining has also been reported to reveal safety signals that conventional pharmacovigilance approaches were unable to detect (Gooden et al. 2013).

While the use of observational data has mostly been limited to population studies, the potential of Big Data in precision medicine is yet to be fully realized. In other industries, much progress has been made to utilize Internet search data and modern predictive analytics techniques to profile customers for the purpose of targeting and enticing them to promote business. The so-called ‘anticipatory shipping’ patent filed by Amazon is a case in point (Kopalle 2014). The idea can certainly be extended to advance personalized medicine. Huge quantities of information may be accessed from personal genome services, which provide direct-to-consumer genetic testing and aggregated customer data for use in medical research (Genetics and Public Policy Center 2009). Wearable devices, which can continuously monitor activity and health state, can provide data in real-time about an individual, which can in turn be used in early diagnosis of diseases and to personalize treatment (Zheng et al. 2013).

In CER, Big Data is poised to play a critical role, providing the opportunity to analyze diverse digitized health data to make decisions about the relative benefits of drugs, and their use in the real-world setting (Gray and Thorpe 2015). Notably, evidence from observational studies may help to fill information gap for which there is inadequate data from RCTs (Berger et al. 2015). Currently the FDA is sponsoring an initiative through the Brookings Institute to examine how “evidence from clinical experience (ECE)” can inform regulatory decision-making. One important issue that needs to be addressed is that the effect sizes in many comparative effectiveness studies are relatively small, hence there is skepticism regarding the robustness of such findings in a regulatory context. Another important issue is whether one can infer causality, rather than just correlation, from observational studies, as will be further expounded below. Further, there needs to be increased focus on a thorough understanding of the operating characteristics of commonly used procedures.

Large providers and insurers are routinely mining their data to examine costs and health outcomes. The insights from these analyses inform their reporting on quality of care indicators as well as their design of care and case management programs. The current focus of much of this activity is to develop better predictive models to assess progression of disease and attendant health care resource use, as well as the response to therapy. Such efforts will provide the engine for a learning health care system (McElwee and Dubois 2015).

With the rising cost of conducting clinical trials, there has been increased focus on data-driven approaches to help improve the quality of data. Non-profit organizations, such as TransCelerate Biopharma and regulatory agencies (FDA 2013; EMA 2013), have focused on a risk-based monitoring framework, which can be informed by real world data on distribution and characteristics of patients, as well as investigators and other site performance metrics. Accordingly, real world data is being used to refine design parameters including definition of inclusion/exclusion criteria, honing inputs for sample size determination, and development of simulated trials to proactively identify and address potential issues that arise in the course of the trial conduct.

Real world data generated at the point of care, including claims data of increasing quality, can be leveraged to make critical decisions regarding access to and pricing of new therapies, by both payers and drug developers. Such data can help characterize diseases and patient populations, evaluate treatment patterns and adherence, gain information on competitors, and target underserved patient groups (Garrison et al. 2007). Further, because of the recognized inadequacy of RCT data used for registration to infer about the experience of patients in real-life setting, there is a growing demand for real world evidence by regulatory authorities, payers and healthcare providers to fully evaluate the cost-benefit of a new treatment (Akhmetov 2015). In the context of the reimbursement of personalized medicine, Big Data will play even a more critical role in addressing some of the challenges faced by payers and drug makers (Faulkner et al. 2012).

In the future, one can imagine that a practitioner will have real-time access to a database that will allow her to rapidly analyze how subsets of patients similar to the one in the examination room (e.g., demographically, principal health problem, general health status, etc.) have been treated and what were their outcomes. This will inform shared decision making by patients and their providers. However, the use of rapid-cycle analytics at the bedside remains controversial today. Nonetheless, in the absence of good insights from the published clinical trials, providers will increasingly seek them from accessible real world data bases (Greenwood et al. 2014).

Historically, a major methodological issue with observational studies in generating evidence has been the handling of bias that arises in the absence of randomization. This has often been tackled using suitable design and/or analytical strategies. Depending on the objective of the study and data sources, various design options are available to mitigate bias, including cohort, case–control and self-controlled case-series approaches. In addition, pragmatic studies, which mimic real-world drug use, while incorporating randomization, are increasingly used to mitigate bias. Analytically, methods such as propensity score and regression analyses have been used to handle observed confounders, while such techniques as instrumental variables are frequently implemented for latent confounders (Berger et al. 2009, 2012). Nonetheless, recent studies have shown that results based on existing approaches are quite dependent on database and methodological choices, and that the accompanying p-values and confidence levels are unreliable, since they are based on incorrect assumptions about underlying null distributions (Madigan et al. 2013). Although modern algorithmic techniques are now brought to bear to mitigate the shortcomings of the conventional approaches, including high dimensionality (Lee et al. 2010), caution is required in the interpretation of the accompanying results in view of their sensitivity to database and methodological choices.

Along with the approaches pertaining to the minimization of bias discussed above, there are other methodological developments that have considerable import in the use of observational data in medical research. Classification and regression trees, including recent enhancements such as random forests, are routinely used in subgroup identification (Foster et al. 2011; Breiman 2001). Regularized regression procedures are now widely used to handle highly dimensional data, particularly in genomics (Li et al. 2007). Alternative data mining tools are available to characterize the safety profiles of drugs (Harpaz et al. 2012). Object oriented data analytic techniques make it possible to tackle data from nom-Euclidean manifolds (Wang and Marron 2007). The so-called topological data analysis (TDA) has been applied to understand patterns in data-driven exploratory settings (Nielson et al. 2015). Traditional methods for detecting outliers are also enhanced to handle high-dimensional cases (Vu and Gopalkrishnan 2010).

These new methodological frontiers have also brought with them challenges of their own, particularly their opacity and the inaccessibility of their operating characteristics to the general audience. Uncritical application of the methods may also lead to unreliable results with untoward consequences for public health (National Research Council 2013).

An additional issue is the challenge of linking data from multiple sources, some of which may be linked and some not. This is not just a technical issue of linkage, but concerns data definitions and how data models are designed. It has been reported that the use of different common data models can impact the results of analyses (Xu et al. 2015).

The generation of actionable knowledge from the vast and complex data generated through high-throughput technologies in genomic and other medical research, as well as data collected via healthcare providers, wearable devices and other electronic media requires the collective expertise of researchers from diverse disciplines. This has in particular presented considerable opportunity for collaborative efforts among statisticians and computer scientists. For example, core capabilities from the two disciplines are leveraged to tackle some of the major Big Data problems, including communication and privacy, in the framework of statistical risk minimization, while taking advantage of the computational capabilities that traditionally exist in the computer science domain (Duchi et al. 2013).

With the advances made in the so-called algorithmic approaches to data analysis, there is always the danger of over interpreting results and ignoring fundamental issues, such as data interdependencies and multiplicity problems. As articulated in a recent article in Science “…quantity of data does not mean that one can ignore foundational issues of measurement and construct validity and reliability and dependencies among data” (Lazer et al. 2014). Indeed, to maximize the value of Big Data in advancing medical science, one should refrain from the uncritical application of readily available algorithmic techniques, and use caution in interpreting results, in the search for patterns and associations in massive data sets. While there is a general consensus about the use of available methods for hypothesis generation, it is prudent to remain skeptical regarding the inference of causation. One of us has argued that rigorous well-designed and well-executed observational study can suggest actionable evidence of causal relationships (Berger et al. 2012).

The growth in volume and complexity of data has required new technological solutions to facilitate the accessibility and linkage of information from the different sources (Hilbert and López 2011). The volume and variety of data necessitates developing highly distributed architectures, introducing increased memory and processing power, and leveraging open-source licensing options (Assuncao et al. 2013). Unlike traditional relational database systems, new platforms, such as Apache Hadoop, are needed to manage unstructured data, as well as data of diverse formats. Cloud solutions with High Performance Computing (HPC) are increasingly relied upon for tasks that traditional computing facilities cannot handle.

Despite the promise of wearable devices to provide real-time data on the health status of individuals, there are still outstanding issues of harmonizing the information gathered from diverse device types. In addition, there is presently no coherent effort to validate the various devices in popular use, or to create a framework to dependably store the data for aggregation purposes.

Despite the considerable advances made in Big data analytics, there are still pertinent methodological issues that limit the potential use of the so-called machine learning tools in evidence-based medicine. In most cases, the operating characteristics of the procedures are not fully explored, and typical applications tend to focus on hypothesis generation, rather than confirmation. Indeed, the introduction of such tools as false discovery rates (FDR) notwithstanding, the issue of multiplicity remains pervasive.

For historical reasons, the development of most of the widely used techniques has evolved in silos, with little or no collaboration among key stakeholders. As was acknowledged in a recent report (National Research Council 2013), there now appears to be a realization that “… massive data analysis is not the province of any one field, but is rather a thoroughly interdisciplinary enterprise. Solutions to massive data problems will require an intimate blending of ideas from computer science and statistics, with essential contributions also needed from applied and pure mathematics, from optimization theory, and from various engineering areas, notably signal processing and information theory.”

An equally important challenge is the ethical issue about ownership of the data. At present there is no clear regulatory or legal framework or guidelines for use of Big Data in advancing medical research (see, e.g., Gray and Thorpe 2015; Williams and Javitt 2006). This requires the collaborative efforts of various stakeholders, including legal scholars, sociologists, and other pertinent professionals. Some of the steps that need to be taken may require enhancing existing measures to regulate the collection and use of health information, with particular emphasis on the challenges posed by the explosion of digitized data outside of the traditionally recognized healthcare sector. A critical component of a viable policy should also be the recognition of the role to be played by patients in the decision making regarding the use of personal information to advance medical science. Most importantly, a responsible public policy should be one that does foster innovation, while protecting privacy and the confidentiality of personal data.

From drug licensing perspectives, there is no definitive standard about the acceptability of data from non-RCT sources to support approval of a new medicine for use in humans. Hitherto, use of observational data has mainly been limited to the assessment of safety signals, and, in frequently, in post hoc exploration of drug utilization and other cost-benefit evaluations. In certain situations, including rare disease research, it may be essential to rely on data available at the time of drug approval (see, e.g., Stuart et al. 2001, for a research direction pertaining to the generalizability of results from RCTs). While the recent passage of the “21^st Century Cures Act” in US congress may eventually pave the way for use of data from other sources to support approval of new drugs (H.R.6 - 21st Century Cures Act 2015), more effort should be exerted to formulate a clear policy about the value and locus of such data in the evidence generation continuum for drug development and licensing. As mentioned earlier, there is an FDA sponsored initiative underway examining this issue.