Surveillance of antenatal influenza vaccination: validity of current systems and recommendations for improvement

To minimise the incidence of influenza infection in pregnant women and infants aged less than 6 months, the World Health Organization has recommended pregnant women be given the highest priority in seasonal influenza vaccination programs [1]. Due to the health benefits to both mother and infant, influenza vaccination has been recommended in Australia since 2009 [2]. Despite strong recommendations promoting vaccination, there is currently no comprehensive, population-based surveillance system for antenatal influenza vaccination nationally or in any Australian jurisdiction. Similar to surveillance systems in the United States [3] and other countries, estimates of influenza vaccine uptake in Australia typically rely on self-reported data collected by telephone survey. However, these surveys are not conducted routinely and often rely on a small sample of pregnant women. The most recent national estimate of antenatal influenza vaccine uptake in Australia (12.7 %) is based on results from a computer-assisted telephone survey conducted in 2009 of just over 10,000 Australian adults, 182 of whom were pregnant [4]. More recent publications have reported uptake within a single jurisdiction or health service, and are not based on routinely collected surveillance data [5‐7].

In the absence of a surveillance system for monitoring antenatal vaccinations, WA Health has conducted an annual survey of pregnant women who delivered during influenza season to estimate the proportion of women who were vaccinated since 2012 [5]. While these surveys have been useful in tracking vaccine uptake in pregnant women and have demonstrated improvement from 10 % in 2009 to 36 % in 2013 [7], implementing annual surveys is resource-intensive and more efficient vaccine surveillance methods may be available for this population.

Other potential data sources for monitoring influenza vaccine uptake during pregnancy are summarised in Table 1. In 2012, WA Health established the Western Australian Influenza Vaccination Database (WAAIVD), a state-wide database of government-funded antenatal influenza vaccinations reported to WA Health, as provided by vaccination providers. Antenatal clinics, general practitioners, and hospitals submit reports which include the full name, date of birth, trimester of pregnancy, batch number and brand of influenza vaccine administered. Reports are generally submitted 1–2 days post-vaccination. Between March 2012 and September 2014, 11,427 doses of trivalent influenza vaccine were reported to WAAIVD. In addition to this database, some public and private maternity hospitals maintain electronic databases into which influenza vaccination status is entered after delivery and before hospital discharge. The accuracy and completeness of antenatal influenza vaccination status recorded in these systems is currently unknown.

This study investigates the sensitivity, specificity and positive and negative predictive values of antenatal influenza vaccination status as recorded in the WAAIVD and maternity hospital databases using self-reported vaccinated status from an annual survey as the “gold standard.”

Western Australia has a population of 2.5 million people, representing 11 % of the total Australian population [8]. Approximately 30,000 babies are delivered each year in Western Australia. Hospital A is a large public antenatal hospital in the Perth metropolitan area which provides maternity care for 10 % of the metropolitan population as well as high risk pregnancies across the state. Health service B is a private health service located in the north metropolitan area, which manages three hospitals providing private maternity care for 20 % of the metropolitan area.

In November and December 2013, 831 women who were pregnant during the 2013 influenza vaccination season and had given birth to a live baby between 07 April 2013 and 06 October 2013 were randomly selected from Western Australia’s statutory Midwives Notifications System. The Western Australia Department of Health conducted a 10-min computer-assisted telephone interview with selected women, which asked whether the woman had been vaccinated during her last pregnancy and reasons why or why she was not vaccinated [7]. Vaccinated women were asked for permission to verify their self-reported vaccination status with their immunisation provider; 563 women provided details for verification of their vaccination status with their immunisation provider: 211 delivered in health service B, 201 delivered in hospital A, and 151 delivered outside these services (Fig. 1). Because the WAAIVD is a state-wide database, the WAAIVD was screened for the vaccination status of all 563 women who provided details, based on full name and date of birth. Hospital A and health service B were provided the names and dates of birth of women who delivered within their respective health system and were asked to screen their electronic medical databases to confirm the women’s vaccination details. This project was reviewed and approved by the Western Australia Department of Health Human Research Ethics Committee.

We calculated the sensitivity, specificity, positive predictive values (PPVs), and negative predictive values (NPVs) for the WAAIVD and the electronic databases of participating maternity hospitals. A number of studies have shown that self-reported vaccination status is an accurate measure of vaccination status [9‐11]. As a result, we decided to use self-reported antenatal influenza vaccination status as the “gold standard.” We calculated 95 % confidence intervals (CIs) using Wilson Score intervals.

Information on the woman’s age, antenatal care provider, chronic medical conditions, and postcode were available based on survey responses. The woman’s postcode was used to determine socioeconomic status, based on calculated socioeconomic indexes for areas (SEIFA) score, which is an indicator for socioeconomic conditions by postcode. SEIFA is comprised of several indices, the main index being the index of relative disadvantage which is derived from low income, low educational attainment, high unemployment and jobs in unskilled occupations [12]. Scores range from 700–1200. Estimates were stratified by quintile based on the distribution of scores in the state, with the lowest quintile indicating the woman was in the most disadvantaged socioeconomic group and the highest quintile indicating the woman was in the least disadvantaged socioeconomic group. Sensitivity and specificity were further stratified by subgroups of women for the WAAIVD, as there was sufficient sample size. Stratified sensitivity and specificity values were compared using z-tests at α = .05.

The demographic characteristics of participating women are shown in Table 2. More than half of women were 30–39 years of age (57.4 %), were in the highest two socioeconomic quintiles (58.8 %), and had an undergraduate university degree or Training and Further Education (TAFE) qualification (52.2 %); 15.8 % had a chronic medical condition. Two in five women (44.1 %) received their antenatal care at a public antenatal hospital clinic; 37.3 % received care from a private obstetrician, 17.1 % from a general practitioner, and 1.4 % from another antenatal care provider.

A total of 37.9 % of women reported they had been vaccinated against influenza during their most recent pregnancy; 80.6 % of these could be verified with the immunisation provider. The majority of vaccinations were administered by a general practitioner (59.6 %); 20.3 % were administered by a public hospital antenatal clinic, and 16.8 % were administered by another immunisation provider, e.g. private hospital, community health clinic, and workplace immunisation clinic; 3.2 % of women did not specify who provided the vaccine. Most women reported being vaccinated in either their second (56.5 %) or third (31.7 %) trimester.

To our knowledge, this is the first formal evaluation of data sources for estimating antenatal influenza vaccinations using self-report as the “gold standard.” We found that systems which rely on provider-reported vaccination events or electronic medical records poorly capture influenza vaccinations administered to pregnant women, and the validity of these systems varies widely. A state-wide antenatal influenza vaccination database which relies on passive reporting from immunisation providers accurately recorded 46 % of influenza vaccinations administered to pregnant women. Electronic medical records within a public maternity hospital recorded 67 % of antenatal vaccinations, and public health service records recorded 29 %. These results indicate there is significant under-reporting of vaccinations administered to pregnant women in all systems.

The low sensitivity and negative predictive values identified in our study are perhaps not surprising, considering these systems rely on passive reporting from either immunisation providers or hospital staff. Although surveillance systems which rely on passive reporting have methodological advantages, such as low cost and relatively simple implementation, they often have low sensitivity and may not be representative. In the case of maternal vaccinations, it is apparent that immunisations are not comprehensively reported and entered, as demonstrated by our study. Previous research has demonstrated that incentives can be used to improve the timeliness and accuracy of recording vaccination events, and this may be one method for improving the validity of these systems [13, 14]. However, this introduces additional resource requirements for sustaining this surveillance activity. Education and targeted intervention may also improve recording; for example, in our setting, only one-third of vaccinations administered by a general practitioner were recorded in the state vaccination database. Targeted education could help improve provider reporting from these sites.

Vaccinations administered outside of traditional providers, such as places of employment, were also infrequently recorded in these systems, likely because these providers are unlikely to report vaccination events to the state government and are less often included in hospital records, as observed by our study. Previous investigations have shown that approximately 30 % of working-age adults receive influenza vaccines in non-traditional healthcare settings [15], and nine percent of vaccinated women in our study reported having been vaccinated at their workplace. Considering 13 % of immunisations provided outside of general practice or public hospital clinics were recorded by the WAAIVD and 29 % were recorded in the maternity hospital database, this is likely a large factor in the under-reporting of vaccinations to these systems and an area for improvement.

In the absence of a comprehensive system for monitoring adult vaccinations, such as an adult vaccination register, influenza vaccinations are difficult to routinely monitor in all adults. However, unique to antenatal vaccinations, there are two logical time points available for recording influenza vaccinations during pregnancy: once at vaccine administration, as the WAAIVD is structured, and again at delivery, as recorded by hospital databases. Data collection at the time of delivery can be logistically convenient as there are statutory requirements for the collection of information related to the pregnancy and birth [16]. Under the Health Act 1911, midwives are required to notify the Department of Health of the outcomes of all birth events within 28 h of birth by completing a Notification of Case Attended form [16]. Data collected in these forms are used to establish the Midwives Notification System, a state-based perinatal data collection [17]. The inclusion of influenza vaccination status in state or national perinatal data collections would establish an annual, electronic source of estimating influenza vaccine uptake in pregnant women. Other countries such as Canada choose to monitor influenza vaccination in this way. For example, the Nova Scotia Atlee Perinatal Database, a population perinatal database collects data on maternal health, details on the delivery, and information related to influenza vaccination [18]. Data are collected after hospital discharge and are based on standardised clinical forms and hospital records. These databases have proven useful for evaluating uptake and the effectiveness of maternal immunisation programs and could be used for similar purposes in Australia.

Globally, there are three types of systems which have been used to estimate influenza vaccine uptake during pregnancy: 1) population surveys; 2) healthcare utilisation, insurance claims, and pharmaceutical dispensary data; and 3) vaccination registries. Population surveys are a valid and reliable method for estimating uptake [5‐7] and have been previously used for surveillance of maternal influenza immunisations in Australia [4]. However, they are time-consuming, resource intensive, and can be difficult to implement annually. Health service utilisation and health insurance claims databases have also been used for surveillance of maternal influenza immunisation. The UK General Practice Research Database, a primary care database containing de-identified health records from 8.4 % of the UK population, has been used to determine the proportion of women immunised against influenza [19, 20]. In the United States, patient-specific insurance claims data, such as Kaiser Permanente health plan membership data [21, 22] and LifeLink^TM Health Plan Claims Database [23] have been used to evaluate uptake of influenza vaccine during pregnancy. In France, a database of prescription use in the general population has been used to evaluate risks associated with drugs administered during pregnancy [24, 25]. While these databases tend to draw from a large population of unique members and have produced estimates similar to those of population surveys [18, 20], such databases are not necessarily designed to provide accurate vaccination uptake estimates. National databases or registries, such as the national database in Denmark which was established during the H1N1 pandemic to monitor H1N1 vaccination [26], are an ideal source of vaccination uptake data. However, the Denmark registry was restricted to pandemic vaccinations, and would need to be maintained annually for it to be useful in monitoring seasonal influenza vaccine uptake in pregnant women.

Electronic vaccination registries of the whole population, such as those of Denmark and a number of other countries in Europe [27, 28], could be used to collect seasonal influenza vaccination information needed to evaluate vaccination programs, in pregnant women and other target groups. Australia’s electronic immunisation registry is currently restricted to children under the age of seven years, and the case for a whole-of-life immunisation registry in Australia has been argued in the past [29, 30]. Expansion of this registry to adults would allow for monitoring of influenza vaccination during pregnancy as well as other target groups. Previous research has shown that electronic vaccination registries estimate vaccination status similar to self-reported vaccination status for women and high risk groups [31, 32], and electronic information is likely the most efficient source of routine vaccination information. Recent establishment of antenatal pertussis vaccination programs in the United Kingdom and the United States of America, and their imminent introduction in Australian jurisdictions underscore the importance of national surveillance systems for adult, including antenatal, vaccination uptake.

There are several limitations to consider when interpreting our data. First, we used self-reported vaccination status as the “gold standard” in this evaluation. While self-report has historically been proven to be a good measure of vaccination status, it could be argued that this is an imperfect measure. In our study, the majority of self-reported vaccinations could be verified by the immunisation provider (80.6 %); however, influenza vaccines administered in the workplace or by non-traditional providers could not be verified. Published literature support the validity of self-reported vaccination status in adults and indicate that false negative self-reports are extremely rare. The sensitivity of self-report has previously been shown to range from 90–100 % [9‐11], and self-reported vaccination status is commonly used to estimate influenza vaccine coverage [33, 34]. However, given 20 % of self-reported records could not be verified, it is possible that some self-reported vaccinations were inaccurately classified as ‘true.’ Second, hospital A routinely offers influenza vaccination to its patients and reports these to WA Health. Hospital A is responsible for the majority of deliveries in public hospitals in Western Australia. It is likely that a large proportion of women who reported a public hospital clinic as their antenatal care provider were immunised at hospital A and were more likely to be recorded. Third, Hospital A and health service B provide maternity services within the metropolitan area of the state, and so the sample of women in this evaluation may not be representative of pregnant women across Western Australia. Finally, it is possible that a name and date of birth search in both databases was insufficient to identify women, which may lead to some discrepancies in the data.

Monitoring influenza vaccine coverage is an integral component of national and state-based evaluation of immunisation programs, particularly for influenza, which is administered annually to a range of target groups [29, 33, 35]. Considering influenza vaccination is an important intervention for preventing disease in pregnant women and is a component of standard care for antenatal patients, surveillance of antenatal influenza vaccination could be improved. In addition to identifying contributing factors to the poor sensitivity of the systems evaluated in this study, additional systems for recording antenatal influenza vaccinations, such as recording vaccination status in perinatal data collections, are available which could be used to monitor this public health intervention. The gaps identified in this evaluation likely apply to other populations where monitoring vaccine uptake annually can be difficult. Exploration of alternative vaccination registers which include both children and adults, and record indication/s for vaccination, such as pregnancy and immunosuppression could potentially replace the fragmented immunisation registers currently available for specific age groups or vaccines. Such information is critical for providing data for monitoring coverage and evaluating disease prevention programs [29, 35].