Monitoring adverse social and medical events in public health trials: assessing predictors and interpretation against a proposed model of adverse event reporting

Adverse event (AE) reporting is an integral part of safety monitoring for clinical trials. However, the processes for collecting, recording, analysing, and reporting AEs can be considered to be more complex and less developed than those processes used when evaluating efficacy in a trial [1]. Safety monitoring in clinical trials has been standardised using AE and serious adverse event (SAE) reporting protocols; for example, The Medicines for Human Use (Clinical Trials) Regulations 2004, which focus on medical events of varying severity. Such AEs may or may not be associated with the intervention. In comparison to clinical trials of medicinal products, public health or social care trials will often evaluate complex interventions in populations with adverse social circumstances; for example, in deprived populations. Such interventions may still have unexpected and unwelcome effects. Monitoring unintended or unexpected outcomes in such trials and participant well-being in general will involve outcomes which are social and psychological in nature in addition to medical. Systems for monitoring these events are underdeveloped and inconsistent in public health, social care, and psychotherapy trials; for example, Duggan et al. (2014) found that the recording of AEs in a trial of psychological intervention was either not attempted/reported, or used definitions not entirely suitable to the intervention or condition being studied [2]. While some authors have attempted to expand on the Good Clinical Practice (GCP) definition of AEs and SAEs to incorporate other types of events [3‐6], none of these have included social events.

The Building Blocks randomised controlled trial evaluated the effectiveness and cost-effectiveness of the Family Nurse Partnership (FNP) home-visiting programme in England [7, 8]. Field and office-based researchers were responsible for the reporting of AEs on a site level to the trial team.

Monitoring events in the trial performed two functions. The first was to detect any undesirable consequence of the intervention. FNP is a supportive and voluntary home-visiting intervention which was not expected to produce harm, but with up to 64 home visits to women in often vulnerable circumstances, the intensive and structured approach could have been unwelcome to some families. The second purpose was to monitor in general the well-being of research participants in both trial arms. This included attempting to ensure that research processes did not add to participants’ distress if they were experiencing adverse social circumstances, and to facilitate optimal trial processes.

The value of monitoring AEs in trials is in detecting harmful effects attributable to an intervention. However, this signal may be obscured by other non-relevant factors that introduce unhelpful ‘noise’. For example, some studies have found reporting rates of AEs to vary by country [3], by reporter (e.g. clinician vs. participant) [9], and by reporting site. Reporting of AEs by health professionals may depend upon their awareness of the event, their judgement about the event, and their willingness to document the event [10]. Variation in reporting AEs driven by under-developed monitoring systems or inconsistent training reduces the potential to adequately monitor unintended effects of both public health and other interventions.

In summary, systems for AE monitoring in interventional studies in public health and social care are under-developed and variation in reports may be due to factors other than the intervention itself. In this paper our first aim is to describe our approach to reporting and categorising Medical and other AE reports in a large public health trial. Our second aim is to assess variability in safety reporting, and explore factors associated with the nature (i.e. the type of event reported), level (i.e. the level of seriousness) and quality of reporting (for example, any differences between study sites) in our study sample.

The Building Blocks trial evaluated the effectiveness of the FNP programme. The intervention consisted of up to 64 home visits from a specially trained Family Nurse during pregnancy and in the 2 years after birth, with the aim of improving outcomes for the health, well-being and social circumstances of young, first-time mothers and their children. The intervention covered core content areas of personal and environmental health, life course development, maternal rôle, family and friends and access to health and social services, including promoting healthy behaviours. The control group did not receive the intervention and instead received usual services, this included the Healthy Child Programme (universally offered screening, education, immunisation, and support from birth to the child’s second birthday) delivered by specialist community public health nurses, and maternity care appropriate to clinical need. Following birth the control group continued to receive postnatal midwifery care and care from existing child health services available locally, including an allocated health visitor. Details of the intervention and control conditions, as well as the full Building Blocks trial methods, can be found in the trial protocol and results papers [7, 8]. Trial outcome data was collected during face-to-face interviews by local researchers and through telephone interviews by staff located in Cardiff who were also responsible for the reporting of AEs to the trial team. From the outset, while a primary focus for safety monitoring was on Medical AEs, other concerns could have been noted by both field and office-based researchers. The collection of AEs was also intended to monitor the general well-being of research participants in both trial arms. For example, we intended to collect information to allow the trial team to have prior knowledge if they were contacting participants at difficult times (e.g. if either a mother or child was undergoing formal safeguarding procedures). Similarly, during the 24-month follow-up interview, scoring positively for items indicating serious abuse on a domestic abuse scale also triggered the completion of an AE form [11]. Detection of domestic abuse via this scale triggered the family’s health visitor being informed, and if ongoing and a new disclosure, resulted in a mandatory referral to social services.

Participants: participants in the Building Blocks trial were 1618 women aged 19 years or under at recruitment and expecting their first child. Young maternal age was used as a programme proxy for a range of poor longer-term outcomes for both child and mother and is also associated with socioeconomic deprivation. It was expected that many trial participants would face challenging individual personal and social circumstances. Baseline characteristics of the participants were collected through a home-based interview prior to randomisation.

Setting: 18 sites in England each comprising partnerships between primary healthcare organisations and local authorities for the purposes of delivering the FNP programme.

Adverse event reporting: AEs were reported during the approximately 2.5-year follow-up period by field and office-based researchers. Field researchers were usually trained midwives or nurses. They collected trial information on outcomes from medical notes as well as in face-to-face interviews (at baseline and final 24 months’ follow-up). They also had a remit to maintain contact with participants for the purposes of data collection. The office-based researchers collected self-report data via telephone interview at late pregnancy, 6, 12 and 18 months following birth. In both telephone and face-to-face interviews AE reports were triggered by participant responses to other open-ended questions or were reported directly from a participant un-prompted. AEs could also be reported by any other health professional associated with the trial including Family Nurses (intervention group only) and general practitioners (GPs). To report AEs a form was completed and sent to the trial team via secure fax, or emailed to the Data Manager. The Building Blocks trial Manager or the Chief Investigator and one clinical member of the research team jointly assessed each form to ascertain the nature, seriousness, causality and expectedness of the AE. Following receipt of the initial form, the trial team could request follow-up data from the reporting site or researcher. Some pregnancy-related events, such as hospitalisation due to child birth, and termination of pregnancy for foetal anomaly, were expected in the context of the trial and were, therefore, not expected to be reported as AEs.

Training: prior to the start of recruitment, field and office-based researchers were trained to collect AEs using a standardised reporting form and following GCP guidance. Instructions were included in the data collection forms (e.g. for the telephone interviews), that reminded interviewers to enquire about participant well-being at the start of the interview (as an open question). Any issues related to well-being at this stage would have been reported as AEs if appropriate to do so. After variations in AE reporting rates were observed during the course of the trial follow-up, additional face-to-face training was provided to all field researchers.

Most AEs reported to the Building Blocks trial were classified as being Medical SAEs or AEs of a physical nature. However, our finding that over 19% of events were Social AEs supports the idea that the GCP definition of AEs and SAEs cannot capture all events related to well-being and social circumstances that might be important for a public health or social care trial.

Reporting of AEs in trials requires a number of inter-related processes to occur (Fig. 2). First, there has to be a reportable event; therefore, an ‘event’ needs to be defined. Pre-existing factors related to the individual may affect this; for example, ongoing or intermittent ill-health which may or may not be related to the individual’s trial eligibility. Factors arising during the course of the trial will also affect this, perhaps most notably, but not solely, exposure to the intervention. Second, events need to be recognised as reportable, either by the individual participant or by a relevant professional. The pivotal factors at this stage are how observable the event is, and its severity. Third, a decision needs to be made to formally report. This may involve decision-making by the participant as well as a professional and key to this will be an assessment of relevance (i.e. is the event of sufficient importance?). This is of course a judgement that can be dependent on many factors; e.g. value placed on the particular event, and is it within the scope of interest of the trial? Most of this is pre-defined. Finally, a mechanism needs to facilitate capture of the event. As we have seen in our trial, mechanisms for capture include direct reporting (e.g. to field or office-based researchers using standardised forms), identification through review of routine records, or identification via screening questions.

How well a trial system can capture with precision all relevant events will depend on adequate progression through each of the stages described above. Clinical trials of investigational medicinal products, which may be most concerned with reporting serious Medical AEs, may fare better in adequately progressing through these required processes than trials of complex interventions where unexpected and undesirable impacts may be less tangible and arise within a broader social context. Defining undesirable impacts may be more complex in public health or social care trials, and the use of Public and Patient Involvement to assist with definitions may be particularly useful in some cases.

We hypothesised that lower baseline psycho-social status or poorer health status may increase the likelihood of both Medical and Social AEs. This relates to the first step of our model (i.e. pre-existing factors). Participants with existing conditions are more likely to continue or repeat experience of that condition. We also hypothesised that poorer psycho-social status would be a better predictor of Social AEs rather than of Medical AEs or SAEs, and this hypothesis was also supported. Our third hypothesis was that participants with at least one reported AE or SAE, regardless of whether Medical or Social event, are more likely to be in the intervention arm. This relates to the Recognition and Decision-making steps in our model. Women in receipt of intervention were regularly in contact with a health professional who in turn also promoted her access to supportive services. The personal relationship between a participant and her Family Nurse would have meant an increased number of opportunities for observing events, and would also have increased the likelihood of the women disclosing a concern, which they may not have otherwise presented to another health professional or researcher. We found trial arm to be a predictor of Medical S/AE, but not of Social AEs, thus providing partial support for our hypothesis. It is possible that expected social concerns may have simply been addressed within the routine remit of the Family Nurse’s work, rather than being documented or reportable as a trial AE. Our final hypothesis was that site-level differences would affect reporting of both Medical S/AEs and Social AEs. While we have been unable to fully explore this facet of the process in our analysis, factors that may vary by site cumulatively impact upon successive stages of event processing and are discussed more fully below. These factors could include local capacity, experience of field researchers and adequacy of training cascaded to local professional and research staff. Table 7 summarises our hypotheses in relation to our results.

How trial teams can optimise the capture of AEs is represented to the right of our model (Fig. 2). These include well-established practices such as having a clearly defined set of criteria for reportable events, awareness-raising amongst key stakeholders and provision of accessible reporting forms. While for clinical trials of medical interventions, the scope of reportable events is well established, this will need to be expanded for trials of complex public health and social care interventions. Adherence to these processes will need to be supported through training, performance monitoring and feedback mechanisms which could involve one-to-one review of reported events and/or systematic assessment of sets of reported events. These combined processes are most likely to impact upon the Recognition, Decision-making and Reporting stages of the model.

Taking our trial as an example, process optimisation would involve training field and office-based researchers to ensure that AEs were collected in a standardised manner. It is important to collect AE data in a standardised manner to enable researchers to pool evidence from large trials [3], and standardisation also allows researchers to compare efficacy outcomes with AEs reported. There were some variations reported in the ways that AEs were collected in the Building Blocks trial, and this may have had a bearing on the proportion of AEs collected from each site. While advice was provided at the outset about what was reportable as an AE (i.e. a clear definition of what an event is), we revised this advice based on early experiences in the trial. Researchers were responsible for asking local health professional teams; for example, Family Nurses, to alert them to any AEs concerning Building Blocks participants. Stickers were also placed inside participant hospital notes alerting hospital staff to contact the researcher with details of any AEs. Having accessible reporting forms and other guidance defining what is reportable is key. Even though field and office-based researchers were trained in the collection of AEs, verbal reports alluded to some variation in the way AEs that were collected in practice. Some researchers reviewed hospital notes for AEs when collecting data for the birth data collection phase of the trial. While this was valuable for identifying some otherwise unreported events, clearer direction at the outset to target this activity would have reduced some apparent unhelpful variation by site. Scoring positively for items indicating serious abuse on a domestic abuse scale also triggered the completion of an AE form, and formally triangulating between data sources to identify AEs where possible might be another way to improve the collection of AEs. It should also be noted that some events have a subjective element; for example, events related to mental health are probably more subjective than those relating to physical health, and the recognition of an event may be affected by the subjectivity of that event. Other researchers have written about the importance of systematic collection of events in medical trials to produce reliable data [14] and to prevent biased reporting [15]. The subjectivity of medical events may be a reason for the slightly lower agreement during classification when compared to physical events which include more objectively observable physical descriptions symptoms/signs/diagnoses, and for AEs (rather than SAEs) the rôle of subjective decision-making may be greater as the apparent importance is less severe. In this study, although reporting systems for SAEs were systematic (i.e. using a common reporting form) and reporting came via multiple routes, there was not a wholly systematic process for their identification. Ensuring that data was collected in a more systematic way could have been done in a number of ways in the current trial. For example, we could have asked researchers to all either periodically search notes for AEs (this was done by a proportion of researchers) or to do this at the end of the trial. While domestic violence was systematically screened for at the end of the study period and where applicable reported as an SAE, other items specifically designed to collect AE data could have been included in the various data collection stages. Tools such as MedDRA have been used for safety monitoring in drug trials; something similar could be used, with supplementary items designed to capture social events. These amendments, however, would have increased costs and participant burden and doing so would have to be balanced against the risk of missing such harms.

Improvements could also have been made in the training given to field and office-based researchers to ensure that AE forms were completed in a standardised manner. The quality of an individual case safety report is dependent on the accuracy and completeness of the information gleaned about the case [16], and the same can be said in the case of AE reporting in the Building Blocks trial. The need for training on the completion of a form should be balanced with ensuring that the forms are self-explanatory as many health professionals completing the forms will be doing so without receiving any formal training. For example, as well as field and office-based researchers, other health professionals and even participants may provide information on AEs in the trial. Guidance on determining expectedness of events was provided during the training; however, some events reported as being ‘unexpected’ were subsequently reclassified due to the context of the Building Blocks trial.

Horigian et al. [17] listed five principles for defining AEs in behavioural research and our own study can be viewed in light of these. Firstly, that they should be grounded in previous research, and secondly, queries on AEs should include domains plausibly affected by the interventions being tested. The current study also defined AEs in light of research, but were more open in what were accepted as AE reports. This may have caused some problems with too much interpretation by Research Nurses and too much variation in reporting by site; this issue was responded to with more training. Perhaps a framework of possible AEs should be put in place a priori which then allows for unanticipated AEs to be observed and reported. Compared to some psycho-therapeutic settings, home-visiting is a more complex intervention, may impact on a broader range of outcomes and not solely for the participant (for example, there could be an impact on a partner, parent, etc.). Even though a logic model and previous literature can inform in advance what AEs may be likely, some flexibility within an overarching framework is helpful. Thirdly, monitoring should attempt to assess relatedness between interventions and AEs, we agree with this but it should be kept in mind that relatedness is perhaps even harder to establish when an intervention is delivered over such a long period of time (2.5 years) and where the intervention (in this case FNP) is also seeking to engage the client with a range of other services, social and family support, this simply adds to the complexity of causation. We agree with both the fourth principle that systematic monitoring is essential for identifying unexpected events, and the fifth, that effective monitoring is a shared responsibility. In summary, the current piece of work provides support for Horigian’s model in a different setting (community-based public heath within families of young children). As they comment on the need for the utility of the principles on other settings in their paper, we provide some evidence of that generalisability. A robust theory to identify broad AE domains, in addition to more specific AEs, is essential to capture unexpected AEs, and that training is even more essential to ensure that. Our study provides an example of where we aimed to capture AEs, specifically Medical or Social, although the approach of Horigian et al (2010) would probably actually address both. The approach to monitoring AEs in social and public health is still limited and variable; our study perhaps identifies the need to better train staff to monitor this in more complex intervention settings rather than with clinical patients.

Active systematic safety monitoring in public health and social care trials which additionally focus on Social AEs is rarely reported. In public health and social care trials, it is likely that there will be adverse experiences that are not medical but may reflect social circumstances. A system of safety monitoring should be considered which would include both Medical and Social AEs. We recognise that this may result in a valid decision not to actively monitor AEs based, for example, on likely frequency and severity. Collecting social events needs to be tailored to the circumstances of the trial and to reflect how the information is likely to be used. This could include assessing any unexpected adverse consequences of the intervention, more general safeguarding of participant well-being during a trial, identifying matters that need to be considered in running the trial (e.g. to avoid contacting participants in distress) and also exploring more broadly the mechanism and broader impacts of an intervention (Fig. 2). How information about AEs will be used should be clearly stated by researchers and guide decision-making about how best to resource and support high-quality data capture.