Data collection systems for active safety surveillance of vaccines during pregnancy in low- and middle-income countries: developing and piloting an assessment tool (VPASS)

Figure 1 summarizes the methodology and processes performed throughout the landscape analysis to identify sites to be evaluated. During previous steps of this landscape analysis, we performed a scoping review and an expert consultation to select perinatal collection systems in LMICs. The systems selection criteria during those steps are summarized in Fig. 1, and the findings are described elsewhere [8]. Only three systems that could perform MNCH data collection at the individual level were prioritized during those steps.

This manuscript describes the finding's last steps of the landscape analysis including System Level interviews and Site Level interviews.

We conducted a rapid review of the literature to identify frameworks on population health or post-marketing drug surveillance that could serve as the basis for developing a tool to evaluate country sites that could support active monitoring of vaccine safety during pregnancy. The inclusion criteria included any guide, publication, or review that describes tools that assess health data collection systems for population or post-marketing drug active surveillance. In addition, we prioritized those frameworks developed or published by international and known public health research organizations. Records detailing tools that focus only on one-hospital medical record system assessment or health information systems mainly used in high-income countries were not considered. We also excluded frameworks focused on specific disease surveillance assessments (for instance, HIV/AIDS, malaria, or tuberculosis). Topics related to passive safety surveillance were not considered since it is not an object of our study.

We ran a search strategy in PubMed/Medline on September 15th, 2020, that contains the following terms: (Framework[tiab] OR assessment tool[tiab]) AND (Health information systems[tiab] OR safety surveillance[tiab]). We applied the "Humans" filter and searched for studies published in the last twenty years. In order not to lose any record of interest, the research group also carried out a gray literature search through Google Scholar using the following words: "framework", "health information system", and "assessment tool".

A total of 156 records were identified through the literature search strategy. After evaluating by title and abstract, four frameworks that met the eligibility criteria were identified and prioritized: The Center for Disease Control (CDC) Updated Guidelines for Evaluating Public Health Surveillance Systems [15]; Guide to Active Vaccine Safety Surveillance: Report of The Council for International Organizations of Medical Sciences (CIOMS) Working Group on vaccine safety [16]; MEASURE Evaluation: Strengthening Health Information Systems in LMICs [17] and World Health Organization (WHO) COVID-19 vaccines: Safety surveillance manual [18]. The last two frameworks were identified through the gray literature search. After a detailed examination, the research team prioritized specific subjects described in these guides consistent with the objectives of this tool. That was because the guidelines identified also included domains outside our study's interest. For example, the CDC [15] and MEASURE Evaluation [17] guidelines assess not only vaccine surveillance domains but also other surveillance components in population health. If some subject was mentioned and described in more than one of these four sources, the research team agreed to take as reference the one that best suited the purposes of this research.

The research group then formulated different domains to group them. The sum of these domains constituted the entire assessment tool. A first draft of the assessment tool was then prepared and evaluated by a group of subject matter experts who constituted our project’s Scientific and Technical Advisory Group (STAG). After implementing their feedback, a definitive validated version was obtained and is described in detail in the results section. The tool was named “VPASS Assessment Tool” (New Vaccines during Pregnancy—Sites Supporting Active Safety Surveillance Assessment Tool).

We carried out a qualitative study using semi-structured interviews to pilot the VPASS Assessment Tool. Stakeholders responsible for managing and/or using the data collection systems on-site were identified and interviewed via online audio conferencing. The data collection took place from April to August 2021.

Key informants were identified using purposive sampling through the recommendations of the research team and members of the STAG. Central-level systems informants—key staff in charge of system development, management, and coordination—were selected based on their knowledge and experience managing one of the three data collection systems.

Site-level informants were experts, managers, or directors of the system at the country site. The decision to include two sites per system was taken deliberately based on the importance of piloting the tool in more than one site per system that were feasible to reach and meet the following criteria: experience collecting MNCH data, availability of MNCH outcomes background rates, and success in previous data collection, management, or analysis implementations.

The selection of the sites was agreed between the central-level informants and the research team.

Kampala (Uganda) and Kigali (Rwanda) were the sites selected for DHIS2, while Belagavi (India) and Lusaka (Zambia) were those for GNMNHR. Due to the decentralized nature of the INDEPTH, a central-level interview could not be performed. Thus, STAG members were requested to suggest sites following the same criteria as for the other systems. As per their recommendation the informants for INDEPTH were one in Nanoro (Burkina Faso), and the other in Manhica (Mozambique).

Semi-structured questionnaires were designed to obtain key information using the different domains of the assessment tool as a guide. We used the same set of open-ended questions in each of the domains. We adapted the questions according to the information we already had before conducting the interviews in an attempt to validate the information we already had or to go deeper into data we did not have. We did not follow a specific order of questions but adapted them according to each interviewee's narration.

Interviews were conducted virtually using the Zoom cloud-based video communications app [19]. Interviews were led by a researcher with experience in qualitative interviews (MB) and researchers with experience in MNCH and surveillance (FRC and AM). In addition, a research assistant took the role of note-taker (EG). Verbal consent was obtained before each interview. Interviews were conducted in English and lasted between 35–60 min.

Meetings were audio-recorded, and each interview was then transcribed verbatim in preparation for the analysis. Transcripts were coded using the software for qualitative analysis Atlas TI version 8 [20]. A thematic analysis was conducted for each relevant domain from the assessment tool. After analyzing all the data, the research team sent the results to the interviewees via email for their review.

The assessment tool included general system domains such as data management policies, internal data quality monitoring, and system’s purpose, and specific site-level domains such as the scale of implementation, data capture, and experiences in MNCH data collection. Table 1 describes the assessment tool domains.

The research team first interviewed central-level referents from DHIS2 at the University of Oslo (n = 2) and GNMNHR at RTI International (n = 1). Then, local key informants from Kampala, Uganda (n = 1) and Kigali, Rwanda (n = 1) were interviewed for DHIS2; and from Belagavi, ​​India (n = 1) and Lusaka, Zambia (n = 1) for GNMNHR. For INDEPTH, local key informants from Nanoro, Burkina Faso (n = 1) and Manhica, Mozambique (n = 1) were interviewed.

Table 2 summarizes the findings for each site. The domains "data quality and data completeness" and "timeliness" were not addressed in this study since they should be answered objectively by evaluating information produced by the systems on-site and not through interviews. Also, the main findings, challenges, and possible important strengths that have been highlighted by the informants during the interviews are described narratively.

DHIS2 is currently used at the national level in the Rwanda and Uganda sites. However, most data collected by DHIS2 Uganda is aggregate data, though there is some individual surveillance data for notifiable diseases, maternal and perinatal audits, and to some extent, tuberculosis. In particular, the only maternal and perinatal health data collected by DHIS2 in Uganda is aggregate data of maternal and perinatal deaths. In Rwanda, individual data collection was introduced at the national level in 2014 with the DHIS2 Tracker, which is an events-based system. Individual data collection for maternal and neonatal variables began in 2017: previously aggregate information was collected on the number of births and the number of newborns. The DHIS2 Tracker is also used for the national immunization registry that follows the mothers and children up until the child receives their last recommended vaccination.

Concerning the local context, the informant from Uganda highlighted the challenges related to data collection on the site. While this information is being collected on paper, it is not being entered into the platform on time. Sites may not have access to technology resources like Internet connectivity or the availability of skilled analysts.

“I think the biggest challenge we have is related to real-time data capture. I say this because vaccination is happening in almost every district and its wide. But we are not able to capture data in real-time. So, we are to enter it into the paper and then turn it into data entry, so we are not able to scale it to all the facilities”

"You need to have people on the ground to be able to enter data, it requires technological support in terms Internet connectivity… So then also the skill set and skill levels, can all these health workers be trained to use the new technology” (Uganda, DHIS2).

In the case of Rwanda, the informant commented that one of the main challenges they encountered was the resistance to the switch from paper to electronic data collection. However, Rwanda has made progress in this area. Indeed, the informant talked about the benefits of using DHIS2 during the COVID-19 pandemic for individual data collection and data analysis.

The Rwanda informant highlighted the wide acceptance of DHIS2 at the national level, both in the private and public sectors:

"Since the DHIS2 implementation at the national level in 2012, we are using it at all levels, including private facilities. So, all public facilities are using it. Private facilities are using it and the other sites that are linked to the Ministry of Health. The ministry really encourages all partners and stakeholders to support the HMIS system, which is DHIS2" (Rwanda, DHI2)

The central-level interviewee for the GNMNHR shared that the GNMNHR's activities are entirely research-based, and the registry does not provide country surveillance nor is the registry linked to medical records. At each site, one institution oversees that data collection, and its data managers and field supervisors provide quality assurance and registry administrators. The GNMNHR is representative of its catchment areas, as the field workers enumerate all households in the catchment areas for which they may conduct annual surveys. This type of data collection allows for pregnancies and deliveries to be captured even if the pregnant person does not attend a health facility. Data may also be collected from the local health centers.

The system-level researchers work to review the data and identify outliers or variables that are not being collected uniformly across the sites. Finally, the stakeholder shared that all the GNMNHR sites have recently transitioned from paper-based data collection to electronic data collection using REDCap and previously collected data is being transferred to the new electronic platform.

The Zambian stakeholder shared that data is collected from ten different health centers in the country, and every individual who attends one of these clinics during their pregnancy is enrolled into the registry. Even though institutional deliveries are increasing in Zambia, the GNMNHR engages with the community they work in to promote enrollment into their registry to ensure its representativeness of the catchment area.

"It's not a national registry. It's within Lusaka, but within that vast Lusaka, it covers quite a number of 10 health centers, which is quite representative of the country. Most of the information collected can be extrapolated to a predictable extent." (Zambia, GNMNHR)

The Indian stakeholder emphasized the representativeness of the GNMNH registry. Overall, enrollment into the registry in Belagavi is above 95%, and the registry does not exclude any individuals unless they do not consent. The MNH registry in Belagavi, India conducts an annual household survey to identify people who are likely to conceive that year. The auxiliary social health advocates (ASHAs) are the field workers who conduct the initial survey and the follow-ups with the identified people. When a person is confirmed to be pregnant, an ASHA will advise them to visit the closest health facility to be enrolled in the registry. This system allows for very early pregnancy identification. Fifty percent of the pregnancies in the MNH registry are registered before 8 weeks, and as much as 95% of the pregnancies are registered before 12 weeks.

“If we just look into the process, it will start from the annual household survey. Our auxiliary social health advocates visit houses to see to it that any pregnancy should not get missed. So, we will register all the pregnant women. Additionally, we follow them up not only to delivery, but 42 days post-delivery” (Belagavi, GNMNHR)

Interviews about the INDEPTH network were only obtained from site-level stakeholders. Stakeholders from Burkina Faso and Mozambique emphasized the decentralized nature of the network and the independence of their own country's systems. In both countries, data collection is conducted by field workers. This type of field data collection is necessary as the percentage of people who deliver at health institutions or attend antenatal care is low. Also, the network is not linked with health facilities' information.

In Burkina Faso, field workers are present in every village the center covers, and once a pregnancy is reported it is confirmed by a supervisor. In Mozambique, pregnancies are registered both by field workers and community informants.

"As you may know, not all of the population or women will attend the health facility for the national health system; we may miss some of the population. So actually, HDSS, uh, let's see, a coverage gap and what we are collecting is actually births and pregnancies." (Burkina Faso, INDEPTH)

"So, for some group of neighborhoods, we identified a key informant. So we visit them in order to collect the information or events that happened between our rounds. They collect information about their neighbors or their households or people who live in the neighborhood,". (Mozambique, INDEPTH)

The INDEPTH in Mozambique can collect data on mothers and neonates in several ways, including the bi-annual household visits conducted by the field workers, the village informants, and their free call center. Community key informants also aid in data collection, and staff from the center visit with the informants bi-weekly to collect additional information that may have been missed. The Mozambican stakeholder mentioned that challenges in using INDEPTH were difficulties adapting the system to meet the specific health needs, as when the Center began to introduce longitudinal data and enumerate households. Finally, data collection is mainly paper based for INDEPTH in Burkina Faso and Mozambique, with both sites citing resources as a barrier to implementing electronic data collection.