Changes in on-time vaccination following the introduction of an electronic immunization registry, Tanzania 2016-2018: interrupted time-series analysis

The Better Immunization Data (BID) Initiative partnered with the Ministry of Health (MOH) in Tanzania to address key challenges in immunization data collection, quality, and use beginning in 2013 [23]. The government identified areas of concern at the outset of the project were poor data quality, inaccurate denominator data, defaulter tracing, poor data visibility, complex data systems, and inadequate data management and use [24]. Digital technologies can help overcome these types of health system challenges, particular through interventions built for clients and health care providers [1].

The BID Initiative implemented an intervention package to address the challenges identified, including: establishment of user-advisory groups (UAGs); development of tablet-based EIR software, the establishment of the Tanzania Immunization Registry (TImR), with online and offline functionality that enabled automated, simplified reports; development of logistics management information systems; provision of targeted supportive supervision for HCWs; establishment of peer support networks (via WhatsApp groups); and support for a data-use culture [24]. By 2018, the package was deployed in the regions of Arusha, Dodoma, Kilimanjaro, and Tanga of Tanzania (Fig. 1). Project staff used a phased roll-out approach to introduce the intervention package to each district within these regions, starting with Arusha as the pilot site which implemented a different EIR before migrating to TImR. Completing paper-based forms and reports remained a requirement by the MOH throughout the project; therefore, all facilities completed dual data entry from the time of EIR introduction and onwards. The EIR allowed HCWs to register children, record vaccinations administered, quickly identify vaccinations due, and generate aggregate facility-level reports that fed into the health management information system. The intention of the EIR was to replicate and eventually replace the use of paper-based data collection tools used in immunization clinics, and in March 2018 facilities in the Tanga region of Tanzania started transitioning to entirely digital reporting as the MOH in this region was comfortable removing the paper-based reporting requirement. Additionally, beginning in April 2018 SMS vaccination reminders were sent automatically through the EIR’s server to the caregivers of children with delayed vaccination visits.

Upon introduction of the EIRs, facilities underwent a data back-entry process where all records included in the facility’s paper-based vaccination register were input into the EIR in batches. HCWs were instructed to register each child seen for immunization services and to enter all vaccines the child had previously received, if they were not already recorded, to maintain a complete vaccination record for every child in the EIR. When children were seen for services at the facility, they were assigned unique patient identifiers which were printed on barcode labels and provided to caregivers, so records could be quickly retrieved upon a child’s subsequent visit at any facility using the EIR. Vaccination information was entered into the EIR for registered children returning to the facility either at the point-of-care (POC) or retrospectively on the same day after the immunization clinic ended. Within the EIR records there is no indication of whether a record was back entered.

Our causal linkage diagram is specific to our study, including key assumptions, and describes our theory of change for the EIR (Fig. 2), however a separate theory was developed for the BID intervention package [25]. Briefly, we believed that by introducing the BID intervention package, HCWs should have improved access to and use of data to identify vaccines due and follow-up with defaulters, therefore vaccination timeliness should improve. This theory makes the following assumptions: a) every child seen at a facility is entered into the EIR; b) all required information is entered into the system; and c) HCWs actively complete follow-up with defaulters, including the use of automated SMS-reminders.

We used secondary data from the EIR as implemented in over 1000 health facilities to test the following hypothesis: the introduction of the EIR in Tanzania lead to improvement in vaccination timeliness after 3,6, and > 6 months post-introduction. We used an interrupted time-series (ITS) analysis study design to measure changes over time. We chose this design for several reasons. First, we could not use any experimental designs that randomized intervention sites on practical grounds, as the MOH and BID Initiative dictated where and when the intervention would be introduced, and our analysis began after implementation. Second, we had limited resources to collect new data, so we had to leverage existing routine health information system data and the EIR data. These constraints made a quasi-experimental study design like ITS a good choice as it is logistically easier to conduct than a randomized trial and minimizes threats to ecological validity [26].

We set up our primary analysis based on our hypothesized theory of change, but quickly found substantial levels of missingness in the data. Missingness could be either due to a child not receiving a vaccination or lack of recording. As such, we conducted a sensitivity analysis to attempt to address this limitation using only registered children with a subsequent vaccination dose recorded in the EIR as this likely meant their record had less missing information.

All data available from the time the EIR was first used at each facility were pulled from the central EIR repository. Duplicate records, records with erroneous dates for vaccination or date of birth such as those with implausible or invalid dates, and records of children born before 2010 were all excluded. Only data from the regions of Arusha, Kilimanjaro, and Tanga were included since these regions had more than 6 months of experience with the EIR at the time of analysis; these data included EIR records from January 2015-September 2018. Facilities with fewer than 6 months of data were excluded because they did not have enough data to be assessed at our 6-month endpoint, also we assumed that these facilities were still adjusting to the EIR and not consistently using the system to capture vaccinations. Observations were excluded if the date of vaccination (real or planned) occurred after the data pull. Children > 2 years at the time of the data pull were excluded based on the assumption that their records may have been incomplete, since they were unlikely to visit a facility to receive vaccinations following introduction of the EIR. To assess time trends relative to EIR introduction across all implementation waves, time was centered relative to the date of EIR introduction for each health facility despite receiving the EIR at different times. Vaccinations administered before EIR introduction were captured from the retrospectively entered data; the original data were recorded in vaccination registers held at the facilities. Because we could not disentangle back-entered data from records entered in real-time, we relied on EIR-introduction date as a proxy.

We summarized health facility characteristics by the number of children registered in the EIR, time since EIR introduction, primary power source, facility ownership (private or public), and facility type using data from RHIS, including the EIR. Counts and proportions were used to summarize facility characteristics within each region. Facility types were categorized based on government definitions; dispensaries were the lowest level of health service provision, health centers provide a wider range of services, and district or regional hospitals offer inpatient, outpatient, and specialized services [27]. Vaccination characteristics were summarized by vaccine type, region, and pre- or post-EIR time period. We calculated the number of eligible children per vaccine type based on a child’s age at the time of the EIR data pull in September 2018 and recommended vaccination schedule for Tanzania [28]. On-time vaccinations were summarized by individual-level data transformed into counts and proportions with the denominator as all eligible children. The number of days after the scheduled date among eligible children and number of vaccinations administered per facility per month were summarized using means and standard deviations. Vaccination history missingness was summarized by the number of children with no vaccinations recorded, no vaccinations beyond birth doses, those with a second or third dose recorded, but missing an earlier dose in a vaccine series, and those receiving one, but not all vaccines due at a single visit.

To estimate the changes in vaccination timeliness, we fit a hierarchical binomial model with a logit link at the level of the health facility, where the outcome was proportion of on-time visits per facility per month. Recall that we hypothesized that the introduction of the EIR would lead to changes in vaccination timeliness over time. We chose to assess the time periods of 0-3, 4-6, and > 6 months post EIR-introduction based on programmatic expectations for when EIR-users would become comfortable using a new technology at the POC and when we expected to see routine use of the system. To test this hypothesis we structured our model to estimate the immediate impact on the level of vaccination timeliness at each time period post EIR-introduction compared to pre EIR (level change) coded as a categorical variable, the slope of the change in successive months for each time period (slope change) was coded as sequentially numbered months during the time period and 0 before or after, and the secular trend in timeliness (time) was centered on the date of EIR-introduction as time 0 and coded sequentially throughout the entire study period. Random intercepts were included at the district and regional levels to account for clustering of observations.

The p_ft term represents our outcome of interest denoting the proportion of on-time vaccinations for each facility f (running from 1 to 937 facilities) at month t (running from − 16 to 27, with month centered on the date of EIR introduction); β₀ estimates the mean level of the outcome; β_1 − 4 are variables measuring slope changes for each time period t and facility f, β₁(Time) estimates the baseline monthly secular trend in on-time vaccinations before EIR-introduction, β₂ (TimeAfterIntro3mos) estimates the slope change during months 0-3 post EIR-introduction, β₃ (TimeAfterIntro6mos) estimates the slope change for months 4-6 following EIR-introduction, and β₄ (TimeAfterIntrogt6mos) estimates the slope change for > 6 months post-introduction; β_5 − 7 are intercept variables measuring immediate level changes in the proportion of on-time vaccinations post EIR-introduction for each time period t and each facility f compared to time pre EIR, segmenting time for 0-3 months (3mosAfterIntro), 4-6 months (6mosAfterIntro), and > 6 months (gt6mosAfterIntro) following EIR-introduction. Random effects separately estimated the outcome for each district and region and were assumed to be normally distributed on the log-odds scale. No other covariates were included in the model for simplicity of interpretation.

Due to the amount of missing data found during our primary analysis, we conducted a sensitivity analysis to understand if timeliness differed among children with multiple vaccinations recorded. To conduct this analysis, we estimated timeliness of DTP1 among children with a documented dose of DTP2 in the EIR, which could have been retrospectively or prospectively entered. For the sensitivity analysis, we used the same model, but estimated the proportion of on-time visits only for DTP1 per facility among only children that had received DTP2.

All analyses were conducted in RStudio (version 1.1). The glmer function in the R lme4 package was used to model our binomial outcome for the four vaccines of interest, weighted by the number of eligible children. Significance was determined at a two-sided alpha value of 0.05.

This study was determined to be non-human subjects research by PATH, as it is part of routine program evaluation. SD, RB, HL, and JS led the design, implementation, and interpretation of findings for this study and were not involved in the BID Initiative design or implementation.

In Tanzania, 1006 facilities within 20 districts and 3 regions had initiated using the EIR and were included in this study, totaling 251,815 children with vaccinations recorded in the EIR (Table 1). At the time of data extraction from the EIR database, facilities had used the EIR from 9 to 27 months. The most used power source amongst facilities was grid power (37%). Most facilities were public, government-owned (75%) and were dispensaries (82%).

There were 246,940 children eligible for DPT1, 243,871 for DTP2, 236,691 for DTP3, and 170,279 for MCV1 vaccines (Table 2). The DTP1 vaccine was most frequently on-time (57.9%) with a mean of 13 days after the scheduled date, while MCV1 vaccine was least likely to be on-time (15.8%), with a mean of 25 days off-schedule. For all vaccine types, vaccinations were more frequently on-time or early before introduction of the EIR. Differences between pre- and post-EIR time periods were observed for all vaccines, with DTP3 and MCV1 vaccines having the largest differences. For all vaccine types, on average more vaccinations were recorded post-EIR introduction per month, with an average of 11 doses administered per month per facility pre-EIR compared to 16 doses post-EIR for DTP1. Fewer doses of MCV1 were administered pre-EIR, 6 versus 15 post-EIR. There were no differences in the direction of the trends observed across regions. Variations by vaccine were observed, with the proportion of vaccinations on-time generally decreasing over time, but a slightly positive trend was observed in Arusha for DTP1 (Fig. 3). Amongst all children under 2 years at the time of the data pull, 479 (0.2%) had no recorded vaccinations, 22,321 (8.9%) received no doses after their birth doses, 8778 (3.5%) had DTP2 or DTP3 recorded, but not the previous dose (these were dropped from the analysis), and 5734 (2.3%) had a recorded MCV1 vaccine, but no DTP3 recorded. There were 10,797 (4.3%) children receiving at least one, but not all of the vaccines recommended at their 6-week visit, 23,757 (9.4%) at the 10-week visit, and 7648 (3.0%) at the 14-week visit. (Fig. 4).

The binomial regression models estimated significant differences in proportion of on-time vaccinations before and after rollout of the EIR, based on level changes (Table 3 and Fig. 5). Compared to the pre-EIR time period, the likelihood of an on-time DTP1 vaccination decreased by 5% (OR:0.95, 95% CI: 0.90-0.99) in the first 3 months following EIR-introduction, but increased 14% (OR:1.15 95% CI: 1.07-1.22) and 25% (OR:1.26, 95% CI: 1.15-1.35) 4-6 months and > 6 months post-EIR, respectively. For DTP2, the likelihood of on-time vaccinations was lower post-EIR, compared to pre-EIR, although the reduction decreased over time with 32% (OR:0.68, 95% CI: 0.62-0.75) reduced likelihood 0-3 months post-EIR, 22% (OR:0.78, 95% CI: 0.68-0.88) 4-6 months post-EIR, and 20% (OR:0.80, 95% CI: 0.68-0.93) post-EIR > 6 months after. A similar trend in level changes was observed for DTP3, with a consistent lower likelihood of on-time vaccination post-EIR introduction, but with improvements seen over time. For MCV1 vaccinations, compared to the pre-EIR time period, the likelihood of on-time vaccination was 55% lower (OR:0.45, 95% CI: 0.36-0.53) 0-3 months post-introduction, 62% lower (OR:0.38, 95% CI: 0.23-0.54) 4-6 months, and 57% lower (OR:0.43, 95% CI: 0.25-0.61) > 6 months post-introduction.

Statistically significant changes in the slope of the trend were estimated for > 6 months post-EIR introduction, compared to pre-EIR, for all vaccine types. For DTP1 vaccinations, the likelihood of on-time vaccinations increased by 5% per month (OR:1.05, 95% CI: 1.05-1.06), for DTP2 the likelihood increased by 2% (OR:1.02, 95% CI: 1.01-1.03), for DTP3 the likelihood increased by 2% (OR:1.01, 95% CI: 1.00-1.03), and for MCV1 the likelihood decreased by 6% per month (OR:0.94, 95% CI: 0.92-0.97).

Changes over time were observed showing a decrease and then roughly a plateau in timeliness following the introduction of the EIR (Fig. 3). We estimated significant differences in the likelihood of on-time DTP1 vaccination post-EIR introduction compared to pre-EIR, with a 19% increased likelihood of being on-time 4-6 months post-EIR (OR: 1.19, 95% CI: 1.11-1.27) and 34% increase > 6 months post-EIR (OR: 1.34, 95% CI: 1.23-1.45) (Table 4 and Fig. 6). Significant increases in the likelihood of on-time vaccinations were estimated for 0-3 months and > 6 months following EIR-introduction, with 2% increase per month (OR: 1.02, 95% CI: 1.00-1.04) and 6% increase per month (OR: 1.06, 95% CI: 1.05-1.07), respectively.

We observed decreases in the proportion of on-time vaccinations following EIR-introduction. However, we emphasize caution interpreting these findings as additional information is needed to understand if the changes observed reflect true estimates of timeliness or if they reflect “noise” due to incompleteness in EIR vaccination records and biases from the data capture process. From our sensitivity analysis, we observed that among those children receiving DTP2, there were improvements in DTP1 timeliness following EIR-introduction, indicating that our belief about incomplete EIR records may very well be valid and the decreases in timeliness observed from the primary analysis are not accurate.

Upon a crude comparison of our estimates to the most recent Demographic and Health Survey (DHS) conducted in Tanzania for 2015-2016, we found further evidence for inconsistent EIR data entry if we assume true immunization coverage did not vary much between 2015 and 2018. The DHS defines timeliness as children receiving recommended vaccinations before age 12 months [29]. We compared the survey estimates to EIR estimates using the same timeliness definition, and observed that prior to EIR-introduction national timeliness estimates were comparable across data sources, for instance DTP1 timeliness was 96.5% in the EIR and estimated to be 96.6% nationally from DHS (See Additional file 1). However, we found that estimates for DTP2, DTP3, and MCV1 decreased following EIR-introduction, suggesting inconsistent data entry post-EIR. We should note that DHSs use information recorded from vaccination cards and parental recall, these data have been found to be unreliable when compared to medical records, and therefore the DHS should not be considered the gold standard for our comparison [30].

Our research group previously noted, “completeness and quality of data input into a system dictates the accuracy of the estimates generated by the system, contingent on the system’s design, user compliance, and system maintenance”, and that “calculating accurate estimates of performance measures using EIR data will likely remain elusive until the challenges” have been addressed [31]. Implementation challenges that may have affected the completeness and/or accuracy of the data in our study included: inconsistent use of the EIR over time, the official requirement of completing dual data entry with the paper record remaining the official record potentially causing HCWs to ensure paper records were more complete than EIR records, inconsistent use of unique patient identifiers causing individuals to have multiple IDs, or poor data entry practices leading to inaccuracies due to workflow or training issues [32, 33]. Inconsistent use of the system is further complicated by facilities using different methods for entering data retrospectively and documenting outreach sessions, as well as staffing changes. During early to mid-2017, the Tanzanian government began restricting public employees who could not prove they had completed their secondary education via a paper certificate. This resulted in the loss of approximately 10,000 employees, including HCWs, who were fired from their positions if they could not present the certificate [34]. The drop in the workforce likely affected the capacity for HCWs to consistently use the EIR during our study period. Additionally, there were potentially server-side issues that could have prevented all data from being made available due to the server timing out or being overloaded. Also, the EIR’s validation rules may not have been functioning correctly, since we found many records with implausible vaccination dates based on the date of birth. Studies based in other countries have found similar challenges, particularly a high rate of under-reporting which causes underestimation of vaccination coverage, low IT literacy needed for adoption of DHI, and poor integration of the DHI within the existing health system [7, 35‐37].

Considering our study team found no alternative reason for true vaccination timeliness to decrease following EIR-introduction after consulting with implementers and MOH staff, and that our results did not align with survey data, the most likely explanation is that our primary analysis results suffer from presumed threats to validity. Rather than timeliness decreasing at a population-level, it seems more likely that the results reflect the EIR implementation challenges described above. However, some facilities may have captured more accurate information in the EIR than previously captured by paper-based tools, and post-EIR, we may have observed true vaccination timeliness previously uncaptured in surveys and other assessment methods, which is useful when reviewing the descriptive results, but is less useful for our time-series analysis. It will be important to reanalyze the data again in a couple of years to understand if the trends have changed due to improvements in EIR-use. Additionally, these threats to validity also further violate the exchangeability assumption needed to assess impact using an interrupted time-series analysis, where we cannot assume that children’s records entered retrospectively versus prospectively are comparable. There may be multiple potential explanations for the trends seen in our data that would require primary data collection to confirm.

We have outlined the potential implementation challenges that may have impacted our analysis; however, it is worth revisiting our theoretical framework to understand where gaps in adoption and use of the EIR may have occurred and subsequently affected vaccination timeliness. EIRs are implemented within complex health systems and require HCW activities to accommodate new workflows that incorporate the tool so their effectiveness relies on how well they are designed, developed, implemented, and used [21]. Consistent entry of data into an EIR may be dependent on HCW competency in using DHI tools, a facility’s internet and electricity connectivity, dual data entry, and HCW motivation, all of which could impact the completeness of EIR records.

Simply having HCWs utilize an electronic tool will not on its own increase timeliness of vaccinations; our causal linkage diagram shows that HCWs would have to use the information in the EIR to encourage caregivers to bring children on-time for their next scheduled immunization appointment and follow-up with defaulters. A realist review found similar findings, with only moderate to low-certainty evidence of EIRs improving data use amongst district- and facility-level staff [21, 38]. Although in Tanzania HCWs were trained to follow-up using EIR information, it is unclear how consistent this was performed. We did not estimate the effect of the SMS-reminders component of the intervention due to the limited follow-up time, but this is a future area of research as it could have affected timeliness. Finally, the length of time to observe behavior change is unknown as it likely varies by caregiver, HCW, and facility. Our assumptions may have underestimated the amount of time needed for HCWs’ behaviors to change.

Furthermore, these challenges beg the question: if these systems make no impact on timeliness, should we invest in them? Improvements have been seen in other settings, an EIR and SMS-reminder system was successfully deployed in Vietnam, where improvement in vaccination timeliness was observed 2 years following system introduction [6]. Additionally, in high-income settings, improvements in coverage and timeliness have been observed due to electronic systems [5]. However, it is worth first asking whether health-related outcomes are the best measures to quantify the impact of these digital systems. Considering the large footprint required for deployment, that often involves cross-team collaborations and implicates staff at each health system level, these systems have additional effects and impacts that are not captured by patient health outcome metrics but may still improve healthcare provision. Digital systems can provide a secure location for record storage, increase patient trust in the healthcare system, improve data quality and accessibility, and can reduce the burden of data management activities, freeing up time for staff to focus on patient care [39]. As the health benefits of DHI may take 3-13 years to be observed, the importance of these more proximal process outcomes should be acknowledged and these metrics used in DHI evaluations [40].

Our study took advantage of the opportunity to use individual-level RHIS data to conduct a quasi-experimental analysis. We were able to showcase the utility and power of these data for answering an implementation science research question by developing appropriate performance metrics within the Tanzanian context that considered changes over time, clinician practices at the facility and district levels, and the cohort of children we expected to see most affected by the EIR’s introduction. The study design process required that the research and implementation teams worked closely together to create a model that would accurately capture EIR introduction and use among facilities and provide interpretable findings.

We encountered numerous challenges using these data to answer our research question, mostly due to EIR-implementation complexities. Upon review of the challenges, we considered some of them to be natural to the process of implementing a new DHI. However, using these data “as-is” for our analysis was difficult due to poor data quality, necessitating a need for a clear understanding of the implementation setting and challenges to continuous data entry and use. There are several important contextual factors which were not accounted for in our models. Because time was centered on EIR introduction date, we were unable to account for the timing of other events or secular trends, such as the public employee dismissal, that could have potentially impacted vaccination timeliness, including changes made to the intervention package. Additionally, without further verification through other data sources and observations, it is difficult to know the level of completeness of the EIR data or when to consider the data to have “normalized”. We also recognized that our study could be affected by system impacts not accounted for in the analysis and other unmeasured confounders. A key limitation of this study is that we were unable to assess immunization coverage as an outcome since we lacked data on the full denominator population of children eligible for vaccination in the community. Future analyses should assess vaccination drop-out, comparing the number of children receiving the first to the third dose, as this is a better measure of timeliness because it accounts for the entire vaccine series and allows changes in timeliness to be measured at the individual-level, rather than facility-level.