The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation

BCIs^g are policies, activities, services or products designed to induce or support people to act differently from how they would have acted otherwise. They involve attempting to change either characteristics of members of the target population (in terms of their knowledge, skills, beliefs, feelings or habits), or their social or physical environment, or both. In the large majority of cases, the goal is to achieve change that is sustained over an extended period of time (e.g., reducing excessive alcohol consumption or smoking prevalence in the general population, or fostering new prescribing patterns among clinicians). Research findings have the potential to provide invaluable knowledge to help with developing or selecting BCIs but this evidence needs to be synthesised and interpreted. We need a cumulative, contemporaneous and accessible knowledge base^g of behaviour change findings to continue to build the science of human behaviour change.

Systematic reviews and meta-analyses provide a means of gathering and synthesising this evidence but the scientific literature on behaviour change is vast and accumulating exponentially. Considering the person-hours required for any given review, there are neither the human nor financial resources to achieve this manually at the scale required. Insufficient human resources to undertake evidence reviews and syntheses also means that these are often out of date by the time of completion [3]. The median time for primary study results to be incorporated into a systematic review has been found to range from 2.5 to 6.5 years [4] and only a minority of reviews are updated within 2 years of publication [5]. A further limitation of the current method is that there is often insufficient power in the evidence gathered to enable moderator analyses, especially for under-researched populations and geographical areas.

In addition, the diversity in the literature presents considerable challenges when it comes to making generalisations in terms of intervention effectiveness. Target behaviours^g vary widely in their characteristics, from cessation of unwanted behaviours such as tobacco smoking to increases in desired ones such as implementing evidence-based practice. The types of interventions evaluated are also subject to wide variation from policies such as raising excise duty on unhealthy products to digital mobile applications for promoting medication adherence. Populations^g also vary, with some studies involving what are intended to be general population samples and others based on participants with special characteristics, such as mental health problems. Settings^g vary across dimensions from physical locality to culture. With such diversity in the evidence base, there is a need for a coherent conceptual framework to allow evidence from different studies to be integrated and compared.

Addressing heterogeneity in the research literature is made more challenging by inconsistent and incomplete reporting of interventions and study methods and findings. The situation has been improved by the publication of a number of guidelines [6], but intervention evaluations still vary widely in quality and format, and are reported inconsistently and incompletely using terminology with limited standardisation [7].

Methods of evidence synthesis such as meta-analysis and meta-regression have substantially improved the ability to draw generalisable conclusions from intervention evaluations, but they are mostly limited to making inferences about simple effects for interventions that have been evaluated, or first-order interactions with moderator variables. More advanced statistical techniques are beginning to be developed [8], and will need to be built on. There is a need to be able to draw inferences that take account of complex interactions between intervention characteristics, populations and settings. Moreover, even with the numbers of studies retrievable by current methods, the populations and settings to which one may wish to generalise are so varied that making inferences from studies to real-world applications is problematic.

Important challenges facing evidence synthesis and interpretation, and approaches to addressing those challenges are shown in Table 2.

The vision for the Human Behaviour-Change Project [9] is to build a Knowledge System that accesses the growing number of BCI evaluation reports^g, automatically annotates these reports to identify key features^g, and synthesises and interprets the findings to answer variants of the big question: ‘What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what settings and why?’. The project includes the development of a user interface^g to allow intervention designers, policymakers, researchers, the general public and other computer systems to access, interrogate and update the knowledge base.

A multi-disciplinary team, spanning behavioural, computer and information scientists and system architects, supported by substantial engagement from scientists and users, will develop and evaluate the first iteration of the HBCP establishing proof of principle, with an initial focus on smoking cessation. This domain was selected due to its large and relatively well-defined evidence base and outcome measures that are relatively robust and important for public health.

The process of knowledge accumulation requires a common conceptual framework within which information can be represented. Data structures that organise knowledge in a structure that specifies entities^g and their relationships are called ‘ontologies’^g [10, 11].

In information science an ‘ontology’ is defined as a data structure consisting of a set of 1) unique identifiers representing types of ‘entity’^g (primarily objects^g, attributes^g, processes^g, or collections of these), 2) labels and definitions corresponding to these identifiers, and 3) specified relationships between the entities. The labels and definitions of entities and relationships in a given ontology^g make up a ‘controlled vocabulary’ which provides a basis for the interoperability of databases using the ontology [10, 11].

Ontologies have transformed a number of areas of science. Most notably the Gene Ontology has unified the field of biology which previously was highly fragmented [12]. Ontology development requires considerable expertise and to that end the OBO Foundry [13] was established to provide a resource for ontology developers and a set of guiding principles from which to work.

As yet, no widely-used ontology has been developed for behavioural science, although ones have been developed for public health [14] and mental entities such as emotions [15], mental disorders and mental functioning [16]. An ontology for understanding human behaviour change needs to represent both causal relationships (e.g., that a given type of intervention affects a given behaviour in a specified context) as well as semantic relationships (e.g., that a given type of intervention is a subclass of a broader type of intervention) [10, 11].

The HBCP will develop a BCI ontology (BCIO^g) that will define important entities described in BCI evaluation reports. Fig. 1 shows upper-level entities that need to be captured in the BCIO and some of their relationships. The labels for these may change in the course of development of the BCIO but this provides an indication of what information needs to be captured. Note that Fig. 1 is not the formal ontology but is shown to illustrate key parts that need to be included.

The BCIO includes entities that are important in answering questions about BCI effectiveness as follows:

The entities in the BCI scenario interact in specific ways, as showed by the arrows in Fig. 2. The content and delivery of an intervention influences the target behaviour through one or more mechanisms of action. The context moderates the influence of 1) the intervention on the mechanism of action and 2) the mechanism of action on the behaviour. Exposure moderates the influence of the intervention on the mechanism of action and is itself influenced by the intervention and context.

Thus if a GP prescribes nicotine replacement therapy (intervention) to smokers interested in stopping (population), as part of a routine consultation in a GP surgery in the UK (context), and 60% of smokers obtain the medication and start the treatment, and 50% take it as prescribed (exposure), this may reduce cigarette cravings (mechanism of action) and so lead to at least 6 months of abstinence (outcome behaviour) in 15% (outcome value) of cases [18].

If one were to conduct a study to assess the effect of GPs prescribing nicotine replacement therapy, this scenario would be compared with a BCI scenario such as GP advice without the offer of a prescription. The comparison would have a number of features relating to study design (e.g., RCT), sample recruitment and selection, sample size, baseline and outcome measures etc. The comparison of outcomes between the two scenarios would constitute the ‘effect’ of the prescription intervention relative to advice without a prescription, expressed in terms of an odds ratio or risk ratio with a corresponding confidence interval. The observed effect would therefore be a function of the features of the intervention and comparator BCI scenarios together with the study methods (Fig. 1).

Artificial intelligence (AI^g) and machine learning (ML^g) applications have been developed to generate and interrogate large, accumulating knowledge bases using ontological approaches. In the HBCP, building computer programs to extract and process knowledge from text documents at a level that is usable by experts in the domain, requires several elements that can generally be equated with intelligence, such as advanced reading ability and significant domain understanding. In this respect, a computer program performing this task can be thought of as artificially intelligent.

Other approaches to artificial intelligence, such as logic-based reasoning have been successful in domains such as robotics and sensor-based systems. Here axioms or rules describe the behaviour of the world allowing a computer program to decide how to respond to inputs. Since the HBCP is concerned with learning patterns from text it is expected that statistical learning, rather than other approaches such as logic-based learning, will be most appropriate.

Artificial intelligence and machine learning have been used successfully, for example, in banking customer service [19], and in areas of medicine [20‐22]. IBM’s ‘Watson Oncology’ uses AI and ML to extract information from research publications to help clinicians identify appropriate treatment options. Algorithms^g are used for entity recognition, information extraction, semantic query expansion in information retrieval, pattern detection, sentiment analysis, and reasoning [23‐26].

In the HBCP, computer scientists will develop automated processes to annotate BCI evaluation reports in terms of key features defined according to the BCIO. These will populate a database^g structured according to the BCIO. Automated annotation^g will require developing and training ‘natural language processing’ (NLP^g) algorithms and other systems for extracting features from tables and graphs. ML together with reasoning algorithms^g will then be used to synthesise and interpret the findings to answer questions and make predictions about what would be expected in as yet unstudied scenarios^g.

Evidence from studies of human-computer interaction^g (HCI) will inform the development of the user interface through which people will use the system. Different groups of users will have different requirements and concerns, which will be addressed in the way that information is presented, and the functionalities available for interacting with it. Understanding user interaction in this project is particularly important, given the ‘black box’ nature of the knowledge base that people will be querying. Addressing concerns relating to the Knowledge System’s trustworthiness, and how the reliability of its predictions can be evidenced, are likely to be particularly important.

Six sets of activities will be undertaken, much of the work being conducted in parallel: 1) forming and engaging with stakeholder groups; 2) developing the BCIO; 3) annotating BCI evaluations according to the BCIO using manual and automated processes and building the BCI database^g; 4) developing and applying ML and reasoning algorithms to draw inferences in response to queries; 5) developing an interface for users and other applications to query the system and provide feedback that can be used to update the BCI Knowledge System as a whole; and 6) evaluating the BCI Knowledge System and its components.

Details of the methodological approach being taken to BCI Ontology development, manual annotation of BCI evaluation reports and the development of automated annotation algorithms, machine learning and reasoning algorithms are presented in Additional file 1. Methods of working will be made accessible in Open Science Framework [27] as they are updated. Outputs and processes of the HBCP will be made available to potential collaborators who are interested in applying these or conducting complementary projects. We will engage a wide variety of stakeholders in a number of groups to enable engagement across countries, cultures, academic disciplines and behavioural domains. A summary of engagement methods are outlined in Additional file 2.

An interface will be developed to facilitate querying and updating the knowledge base, and the BCI Knowledge System as a whole. It will consist of a machine interface and a user interface.

The machine interface will provide the primary means by which BCI reports are added to the database. It will provide a facility by which programs that search and screen reports can feed those that are relevant into the database, ready for annotation. It will also include an application programming interface (API) to allow for other programs to formulate queries and receive responses in machine readable form. The aim is to make the BCI Knowledge System as interoperable as possible with other software that is being, and will be, developed.

The user interface will be a website that will build on the wide range of external perspectives that have fed into the BCIO development and ML components of this work and engagement with a wide range of stakeholders. It will handle three types of scenario:

Users of the interface will be able to generate queries about BCI scenarios. They will enter fixed or constrained parameters (e.g., the behavioural outcome, the mode of delivery, the target population, the setting, or a range of effect sizes) and interrogate the BCI knowledge base for predicted values of BCIO entities that are left open. Examples of queries are shown in Table 3.

Because users will vary in their levels of expertise in the topic of the query, the user interface will provide a facility to guide them through the generation of the query so that they arrive at the most useful results. For example, users may start the query at too general a level of abstraction for the Knowledge System to be able to generate meaningful results, or they may not be aware of the importance of particular moderators or intervention components when generating the query. The user interface should be able to draw attention to these issues and prompt users to generate queries that get the most out of the data available.

Users will also be able to use the interface to generate a curated and annotated bibliography of research reports relevant to their query. This may be particularly useful for systematic reviewers who may want to take advantage of the precision with which the system will permit searches to be carried out, but may want to undertake data extraction and synthesis by hand or using a different program.

The HBCP involves evaluation of BCI Knowledge System as a whole as well as its parts. There will be an ongoing process of evaluation and development throughout the project, but at a certain point it will be necessary to assess to what extent the project has met its objectives, and to provide information to guide future decisions. In accordance with the HBCP research questions, the HBCP will undertake the following assessment: