Defining & assessing the quality, usability, and utilization of immunization data

High quality public health data are needed for many reasons, including decision-making and planning at all levels of the health system, monitoring program performance, and justifying financial support; this is equally true for data related to vaccinations [1, 2]. The primary way that the performance of national immunization programs is monitored and evaluated on a global level is by tracking national estimates of vaccine coverage (the proportion of a target population that received a given vaccine) over time [3]. A key source of information for generating these estimates is routinely collected data on numbers of vaccines administered (called aggregate or “administrative” data) [3].

Unfortunately, such data from low- and middle income countries are often deemed to be of poor quality [3‐6]. Because of the need for high quality data, many organizations have invested considerably in improving data quality, including Gavi, The Vaccine Alliance (Gavi); the World Health Organization (WHO); UNICEF; and the U.S. Centers for Disease Control and Prevention (CDC). Unfortunately, there is no gold standard measurement of data quality against which any other measurement can be compared.

Some published methods for assessing the quality of routinely collected (administrative) data at national and subnational levels include the immunization Data Quality Audit (DQA), the immunization Data Quality Self-Assessment (DQS), and WHO’s Service Availability and Readiness Assessment (SARA) [7‐9], however each has important limitations, as described later. In order to identify a more comprehensive range of issues that might affect data quality, CDC, WHO, and country-level partners have developed a holistic approach that looks broadly at the immunization information system itself, including triangulating data from multiple sources and looking at other relevant components (e.g., workforce) that affect the overall quality of vaccination coverage data [10].

On a global level, national vaccine coverage estimates derived from administrative data quality are frequently evaluated by comparing estimates of vaccination coverage from administrative data with coverage estimates from population-based surveys. This approach may reveal large differences in coverage estimates; a recent comparison of country-level coverage estimates derived from administrative data with those derived from representative surveys found that the administrative data were 26–30% higher than estimates from surveys for the same year [11]. While coverage estimates derived from surveys are considered the most reliable, data sources used to determine vaccination status of children (e.g., parental recall, review of home-based records) are themselves subject to inaccuracy [12, 13]. Although discordance between administrative data and survey-based estimates might indicate underlying issues with data quality for at least one of these sources, data quality issues can occur with both, so good agreement between sources does not necessarily guarantee high quality data. Furthermore the accuracy of administrative vaccine coverage relies on both accurate administrative data and accurate estimates of the target population, introducing an additional source of potential error [14].

To adjust for potentially poor quality data, WHO and UNICEF have developed the WHO/UNICEF Estimates of National Immunization Coverage (WUENIC) approach that yields a “best guess” estimate of national vaccination coverage by reviewing information from official country reports, surveys, temporal trends, and expert judgement [15]. The WUENIC approach provides relevant contextual information (such as known vaccine stockouts) and a subjective grade of confidence in the quality of the data sources used. An alternative approach proposed by the Institute of Health Metrics and Evaluation (IHME) uses a model-driven quantitative approach and provides statistically derived 95% uncertainty intervals [16]. Ultimately, though, assessment of administrative data quality represents a comparison of differences between potentially flawed measures with little way to judge the magnitude or importance of those flaws.

Challenges also exist in simply defining “data quality” and consensus around a single definition is lacking [1]. Certain terms (such as accuracy, reliability, precision, and validity) are used interchangeably in discussions of data quality even though they actually mean different things (see Table 1) [1].

Applying inconsistent and vague definitions has made measuring the underlying quality of administrative data subject to incorrect or misleading interpretation. For example, the prevailing methods for formally assessing vaccination data quality, the DQA and the DQS, focus heavily on evaluating the consistency of data between sources (especially between facility-based paper records and aggregated monthly reports) or between various reporting levels (such as between facility, district, and central levels) [7, 8]. However, the point of primary data collection – by the frontline healthcare worker at the time of vaccination – is mostly unexamined, yet potentially represents a substantial source of error that, once recorded, is not correctable at higher levels of the system. The DQA calls for observation of vaccination sessions, but only calls for observing five healthcare worker-patient interactions per facility if visits coincide with vaccination sessions [7]. The DQS does not address primary data collection at all, but instead defines “accuracy” as consistency of data between reporting levels [8].

We believe that basic concepts underlying “data quality” need to be untangled in order to have more operationally relevant discussions about what each means; how they interact with each other; the relative importance of each; and appropriate methods with which to measure, interpret, and improve data quality.

We therefore offer a framework that describes immunization data in terms of three key characteristics (data quality, data usability, and data utilization) intended to keep the underlying concepts separate and specific; to keep the definition of each concept as understandable and measurable as possible (or to explicitly identify those that are inherently difficult to measure); and to provide specific indicators and methods that can be used for assessment.

Although we have focused on issues related to vaccines, the underlying issues likely hold across many public health programs. At the center of the push for high-quality data is a search for trueness in available data and the estimates derived from them. For vaccine data, in settings where a high degree of attention is paid and value ascribed to careful record keeping, where there is a good system of accountability, and where there are accurate estimates of the target population, routinely collected immunization data likely can be trusted. However, in many low- and middle-income countries, this is often not the case.

To improve data quality overall, concern about data must begin at the point where they are first collected (Fig. 2). Those responsible for primary data collection (typically frontline health care workers) are often overworked, under-motivated, or see data recording duties as unwanted, time-consuming additions to providing care to patients [21]. Forms needed to collect data properly are frequently outdated, poorly designed, or missing. Because there is often 1) little or no feedback on the data they do report, 2) limited time and ability to analyze data locally, or 3) a lack of understanding of how to use the data to help them do their jobs, they often do not see the value of the data they are charged with collecting [21]. Finally, there are rarely consequences for poor data recording practices or rewards for good record keeping; in some cases, there may actually be incentives to falsify reported data [3, 8, 16]. If there is not adequate attention paid to the quality and consistency of initial data collection, no technologic solution or amount of effort invested after the fact will improve those data.

Nonetheless, good decisions can be made with bad, or even no, data, while bad decisions can be made despite good data being available. Therefore, what degree of quality (especially if narrowly defined as “trueness”) is actually required to make informed or “good” decisions, to plan, or to monitor program performance? Should the international community be focusing on improving data quality in order to meet its own information needs while mostly ignoring whether or not data are actually being used at country and local levels to monitor and improve service delivery?

We believe that while data do need to have a reasonable level of quality, a focus on quality alone (or even quality and usability, as defined here) is shortsighted and misses the value and importance of data utilization. Any real, lasting improvements in data quality will likely not be achieved through external pressure from the international community, but rather through increasing the internal demand for data, especially at the level of primary data collection. Improving the quality and usability of routinely collected immunization data will likely depend on a feedback loop being established where increasing use of data (especially at or near the point of primary collection) results in an increased demand for better data that, in turn, encourages more use [22]. Such a feedback loop would require far more attention be paid to increasing data use skills of staff throughout the health system, establishing standard operating procedures for primary data collection, simplifying reporting, introducing accountability structures and procedures, and implementing automated data support and visualization systems where possible. With such an effort, we believe that not only would the quality and usability of immunization-related data be greatly improved, but more importantly, so too would the utilization of those data, leading to better decision-making for immunization program management and improved health outcomes.

Being deliberate about the choice of indicators; being clear on their definitions, limitations, and methods of measurement; and describing how those indicators work together to give a more comprehensive and practical understanding of immunization data quality, usability, and use, should yield more informed, and therefore better, programmatic decision-making.