INTRODUCTION
The Health Protection Agency (HPA) provides an integrated surveillance approach to health protection, using a range of tools including syndromic surveillance [1]. Within the HPA national syndromic surveillance is undertaken by the Real-time Syndromic Surveillance Team (ReSST).
Syndromic Surveillance is the real-time (or near real-time) collation, interpretation and dissemination of routine electronic data to allow the early identification of potential public health threats and their impact, enabling effective public health action. The surveillance is based not on the laboratory-confirmed diagnosis of a disease but on the presentation of signs and symptoms or proxy measures available through routine data sources that can constitute a syndrome/provisional diagnosis [Reference Triple2].
Syndromic surveillance was first developed in response to the deliberate release of anthrax in the USA following the 9/11 terrorist attacks [Reference Kman and Bachmann3]. The particular strength of this form of surveillance was seen as the ability to detect unusual signals.
In order to rapidly detect potential infectious disease threats during the Olympic and Paralympic Games the HPA set up a suite of robust and multi-source syndromic surveillance systems. These included enhancements of already established systems as well as new systems created for the Games [Reference Severi4].
Prior to the Games, the HPA undertook an assessment of the ability of their syndromic systems to detect various public health incidents, e.g. a Cryptosporidium outbreak, pandemic influenza, etc. The aim of the project was to quantify what could and could not be detected, thereby improving understanding of the strengths and weaknesses of syndromic surveillance and so giving policy makers more confidence in conclusions drawn from the surveillance. This paper describes this work to show how scenario planning can be used to evaluate the effectiveness of syndromic surveillance.
METHODS
Overview
A number of scenarios were identified, reflecting potential major incidents that the HPA would be required to provide rapid information about if they occurred during the Games. ReSST used these scenarios to test their syndromic systems and quantify the detection abilities of their statistical surveillance methods.
The ReSST coordinates four main surveillance systems; HPA/NHS Direct (a national telephone health advice line run by NHS Direct), HPA/QSurveillance [in-hours general practitioner (GP) consultations], GP Out-of-hours/unscheduled care (GPOOHSS, out-of-hours GP consultations) [Reference Harcourt5] and the Emergency Departments Syndromic Surveillance System (EDSSS, emergency department attendances), the latter two having been developed as part of the enhanced Games surveillance programme [Reference Elliot6, 7].
Estimating baseline activity – what is normal?
It was initially necessary to describe what is expected under normal circumstances during the Games period. For the established systems this was quite straightforward, the expected activity can be modelled across the relevant syndromic indicators by looking at previous years' summer activity. For the newer systems, where extra providers were continually being recruited, data from previous summers did not reflect the volume of data we expected to see during the Games period. It was necessary to scale up the historical data to take account of the increased coverage as more data providers joined the syndromic systems; for instance during the summer of 2011 information was only being received from two emergency departments, equivalent to 6·5% of London emergency departments by volume, whereas by the Games period coverage was expected to include 51% of London activity.
During the Games there were expected to be population changes (both influx and efflux) that might have an impact on healthcare usage, these included an influx of day visitors and overnight visitors and the possibility of local residents choosing to ‘avoid’ the Games by taking holidays during this period.
For the influx for example, when assessing the HPA/NHS Direct surveillance system a range of estimates for the number of extra visitors expected in addition to the usual summer London population was used to model the baseline data.
For the efflux, population changes could have an impact on London residents, for example, an upper estimate of a 10% decrease during the Games of the Newham Primary Care Trust resident population was used. [Healthcare in England is commissioned by 152 primary-care trusts (PCTs), typically consisting of 200 000 residents.] If this occurred then there might be a reduction in the consultation rate at doctors' surgeries, where the denominator is the registered population which does not vary due to holidays.
Games scenarios
[It is important to note that the scenarios were not real events but were constructed purely to test the syndromic systems. They were not the result of any threat analysis undertaken by the HPA or other bodies.]
A number of different scenarios were considered and identified as the most important:
-
• Contamination of a local water supply by Cryptosporidium oocysts.
-
• A localized food poisoning incident involving scombrotoxin.
-
• An outbreak of a new variant of influenza, arriving with Games overseas visitors.
-
• An intentional release of botulism into the food chain at a Games venue.
-
• An intentional release of anthrax via aerosol dispersion.
The Cryptosporidium scenario was based on a historical event [Reference Cooper8], using the observed epidemic curve to estimate the number of cases each day and translating the location to a London PCT where the impact on the Games would be the greatest. For the other scenarios there were no directly comparable historical examples that could be used although historic information was used to help model some of the scenarios.
In the influenza scenario, which involved modelling an infectious disease, a SEIR (susceptible, exposed, infected, resistant) model was used to estimate how the outbreak would develop over time and therefore how many new cases would be expected each day in the early stages. Table 1 shows how transmission was modelled to spread between groups, for instance each infectious overseas visitor is expected to result on average in 0·310 new cases in other overseas visitors and 0·115 in UK residents outside London who are not visiting the Games.
Table 1. Parameters used in SEIR influenza model
Table 2 shows which of the syndromic indicators we would expect cases to be predominately coded to, including higher and lower estimates.
Table 2. Percentage coded to syndromic indicator with upper and lower estimates
ILI, Influenza-like illness; ARI, Acute respiratory infection.
Modelling healthcare presentation
For each scenario, the proportion of patients choosing to use different methods of healthcare (e.g. telephone advice, local doctors, hospitals) were estimated in order to calculate the number of extra consultations captured in the syndromic systems (Table 3). Research is available to support the healthcare use patterns for some pathogens via the second study of infectious intestinal disease in the community (IID2 study) [Reference Tam9] and self-reporting via influenza surveys (W. Edmunds, personal communication). Where research on specific pathogens was not available estimates were based on similar pathogens or the total volume accessing healthcare by each method. Where estimates were used, an upper and lower estimate was also included to reflect the plausible range of values and a sensitivity analysis performed to identify the impact on the analysis of different estimate values.
Table 3. Percentage presenting to different healthcare providers with upper and lower estimates
ED, Emergency department.
The methodology varied slightly between scenarios but the estimates usually took the form of a ‘presentation pyramid’ (Fig. 1), with the combined effect of estimates for: development of symptoms, healthcare usage, system coverage and coding, giving the number of people expected to be coded to a syndrome within each system.
Fig. 1. Presentation pyramid for people exposed to pathogen.
Overseas visitors, domestic visitors and London residents were considered separately as they would be likely to access care in different locations; domestic visitors may return home before developing symptoms and using local care, international visitors are more likely to use walk-in centres to access general practice services.
With the exception of the HPA/NHS Direct surveillance scheme in England and Wales, population coverage by the syndromic systems is only partial. Coverage varies across the country but is usually well known and it is clear in which parts of the country incidents are more likely to be detected because of better coverage.
With each scenario specialist epidemiological colleagues were consulted as to the most likely symptoms that people will present with, and estimates made as to the proportion of these that will get coded to the syndromic indicators. There is considerable uncertainty in some of these estimates, particularly where there are no historical precedents. The symptoms with which patients present can have a considerable effect on what can be detected because some indicators have a much lower level of background activity than others.
The central estimates used for coverage and proportion coded are presented in Table 4.
Table 4. Central estimates used to calculate numbers presenting
ILI, Influenza-like illness; GP, general practitioner; ED, emergency department.
Simulating outbreaks
The baseline expected activity for the Games period (July–September) was combined with the extra activity predicted for each scenario to test whether or not the syndromic systems could detect the changes and how quickly extra activity would be identified.
The simulated number of diarrhoea calls under the Cryptosporidium scenario, combining the modelled baseline, extra cases due to expected population changes during the Olympics and outbreak calls, are presented as an example in Figure 2. Here the upper confidence interval forms the ‘alarm threshold’ and it can be seen that the outbreak would only be detected in this example at its peak.
Fig. 2 [colour online]. Modelled diarrhoea calls to NHS Direct (London).
For the HPA/NHS Direct scheme, a simulation approach was used; the background data were combined with the scenario data plus random background noise, to reflect the historical variation outside epidemics, and the proportion of true alarms was counted, along with failed detections and false alarms. This approach was applicable to HPA/NHS Direct because the data had been well modelled using 8 years of historical data and the statistical methodology had been validated over many years of use; hence it was possible to accurately estimate the random variation in terms of a series of over-dispersed Poisson distributions for each indicator and Strategic Health Authority (SHA) area. For the newer systems there was no tried and tested parametric model for the random variation so simulations were not appropriate.
For the other systems the general approach was to add the extra activity predicted by the scenarios to the base data ‘once for each date in the Games period’ and then calculate the proportion of dates which resulted in the extra activity being detected. These combined datasets enable an estimate to be given for the minimum size of incident that can be detected with a probability of at least 50% (Table 5) and the time until probability of detection reaches 50% (Table 6). This approach has the advantages of using the actual random variation in the baseline data which came from previous years and identifying how changes due to day of week effects and bank holidays might affect detection.
Table 5. Minimum size of incidents detectable by syndromic surveillance during the Olympics
HPA/NHS Direct, A national telephone health advice line run by NHS Direct; HPA/QSurveillance, in-hours general practitioner (GP) consultations; GPOOHSS, GP out-of-hours/unscheduled care; EDSSS, Emergency Departments Syndromic Surveillance System.
Table 6. Expected number of days between incident and detection
HPA/NHS Direct, A national telephone health advice line run by NHS Direct; HPA/QSurveillance, in-hours general practitioner (GP) consultations; EDSSS, Emergency Departments Syndromic Surveillance System; GPOOHSS, GP out-of-hours/unscheduled care.
Table 7. Sensitivity analysis example – hypothetical Cryptosporidium outbreak on 1 July 2012
Bold text shows assumptions and results used as central estimates.
Using simulated and modelled datasets it was also possible to vary the scale of the incidents in order to ascertain how big an incident needed to be before it was detectable and how soon incidents of different sizes would be observed.
RESULTS
Sensitivity analysis
The sensitivity analysis identified which assumptions and estimates had the most impact on detection rates given the different levels of uncertainty involved in the estimates. Estimates needed for the analysis can be divided into the following broad categories:
-
(1) Estimates of the number of people falling ill and becoming symptomatic and modelling assumptions about reproduction rates for infectious diseases.
-
(2) Estimates of the proportion of symptomatic people who access care via telephone helplines, GPs, or emergency departments and assumptions about differences between weekend and work-day proportions.
-
(3) Estimates of population changes during the Games period, travel and spectator demographics and assumptions about visitor healthcare use.
-
(4) Estimates of local coverage by syndromic systems.
-
(5) Estimates of proportion coded to syndromic indicators.
In most scenarios the biggest impact, reflecting the greatest uncertainty, was linked to estimates of the proportion accessing various types of healthcare. This was true even in the Cryptosporidium scenario where more robust information was available of estimates for the proportion of people likely to call HPA/NHS Direct or visit their GP. Where healthcare use was not the main factor affecting detection rates, the main issue was local coverage and this was reflected in sub-scenarios used to quantify these differences. For instance the GP system, HPA/QSurveillance, provides data regularly from all the GP surgeries within one London PCT, while for a neighbouring PCT the scheme only covers around 13% of surgeries.
Probability of detection
The probability of detecting scombrotoxin poisoning via HPA/NHS Direct is presented in Figure 3; under this scenario patients may present with either diarrhoea or vomiting, this graph shows that incidents are more likely to be detected using the diarrhoea indicator.
Fig. 3 [colour online]. Probability of detecting diarrhoea/vomiting cases, NHS Direct, 2010–2011.
The minimum size of incident expected to be detected with at least 50% probability is shown in Table 5. (The influenza scenario is not included in this table because in all cases considered the numbers would grow exponentially and be detected at some point.)
Under the influenza scenario, which assumed a similar reproduction rate to the H1N1 outbreak in 2009, the number of cases would grow exponentially and always be detected by the syndromic systems at some point, even if it was introduced by just a handful of seed cases.
The different strengths of different systems can be seen in Table 5:
-
• Fewer cases are needed to trigger an alarm in the emergency departments for incidents involving the most severe symptoms.
-
• The size of incidents that can be detected vary widely across systems and between scenarios. Differences between scenarios depend on which indicators patients' consultations are coded to, how geographically contained the incident is and over how many days patients are likely to present with symptoms.
-
• Some systems, for instance HPA/QSurveillance, are better able to detect local events because larger numbers enable local as well as regional surveillance, but coverage varies across the UK.
An example of a sensitivity analysis, in this case for the HPA/NHS Direct surveillance system using the Cryptosporidium scenario, with a hypothetical onset date of 1 July 2012 is presented in Table 7. The greatest range of detection probabilities occurs under the assumption about the ‘reporting ratio’, which is based on evidence for the proportion accessing healthcare in the IID2 study [Reference Tam9].
Timeliness of detection
Timeliness is measured by how many days would elapse after an incident occurs before there is a better than 50% chance of detecting the incident (Table 6).
The speed of detection depends mainly on the organism involved; with cases of food poisoning at one event, people are likely to all become ill within 1 day, while with infectious diseases like influenza there will be a gradual increase in cases over time and symptoms will take a few days to develop. A scenario involving people being ill over several weeks will be harder to detect than a similar sized incident occurring on just one day.
DISCUSSION
This study showed that the syndromic surveillance systems could detect the key incidents of public health concern identified in the scenarios, and provided estimates for the scale of incident that could be detected and the speed of detection. The probability of detection by syndromic surveillance alone remains low when the total number of people symptomatic is small. With scenarios involving patients with very severe symptoms the new emergency department system is the most sensitive detection system, provided the incident occurs near a sentinel site. Which system provides the timeliest detection varies depending on the scenario (Table 6).
Demographic changes due to the 2012 Olympic and Paralympic Games were found to have a negligible impact on detection rates, although the increased travel would make detection harder; incidents are easier to spot when concentrated spatially and temporally. An incident at a Games venue would be harder to detect if those affected came from many different places and returned home before developing symptoms.
There were, fortunately, no major health issues affecting the games and the demographic changes did not impact on the ability of syndromic systems to monitor public health.
An intrinsic limitation of syndromic surveillance is that although an incident may lead to a noticeable rise in a syndromic indicator, it is very unlikely that the cause or pathogen would be identified by syndromic systems alone, but rather that the rise will lead to further investigation.
The accuracy of simulations depends on having good historical baseline data. Where systems are introduced prior to a mass gathering it is preferable to have good coverage for at least a year prior to the event.
The ability of syndromic surveillance to identify the scale of public health issues in the UK would be greatly enhanced by better understanding of healthcare use for common infections/conditions. With better estimates for the proportion of symptomatic patients who access the different types of healthcare it would be possible to extrapolate from increased activity to incidence in the community. Ideally data collected for syndromic surveillance at mass gatherings should include information on whether patients have attended the event(s).
By better quantifying the detection abilities of syndromic surveillance in the UK, public health practitioners will have better information when planning for emergencies and have more confidence in interpreting syndromic data alongside other intelligence. During the Olympics, the Health Protection Agency was able to provide reassurance that no major health incidents had occurred and that none of the local incidents recorded had become major incidents, and was able to quantify what was meant by a major incident. This is an approach that can be used in any country for their syndromic surveillance and should be a prerequisite when providing reports that seek to reassure that no major incident has occurred both during mass gatherings and in routine surveillance.
ACKNOWLEDGEMENTS
For the provision of routine surveillance data we thank the following: NHS Direct for the call data; and the University of Nottingham, EMIS and the EMIS practices for the QSurveillance data extraction; Ascribe Ltd and L2S2 Ltd and staff within the respective NHS Trusts for emergency department data; and out-of-hours/unscheduled care providers and Advanced Health & Care. We also thank the following for their expert advice and input into the scenarios; Sue Odams, Gionvanni Leonardi, Kathie Grant and Bob Adak. We thank the following for help with technical systems advice and statistical modelling: Sue Smith, Helen Hughes, Sally Harcourt, Paul Loveridge, and Joseph Egan.
DECLARATION OF INTEREST
None.
