Background
The Sustainable Development Goals call to end preventable deaths of newborns and children under five, with all countries aiming to reduce neonatal mortality to 12 or fewer deaths per 1,000 live births and under-five mortality to 25 or fewer deaths per 1,000 births by 2030 [
1]. In 2020, more than 5 million children died before age five, and 47% of those deaths occurred during the neonatal period—the first 28 days of life—even without the increase in mortality attributable to COVID-19 [
1]. Sub-Saharan Africa has the highest neonatal mortality rate in the world at 27 deaths per 1,000 live births, contributing 43% of the global share of neonatal deaths [
1]. In Mozambique, a country with limited resources and a high disease burden, neonatal mortality remains a significant public health problem, with a rate of 28 deaths per 1,000 live births in 2020 [
1].
Research suggests that increasing the coverage and quality of preconception, antenatal, intrapartum, and postnatal evidence-based interventions globally by 2025 could avert 71% of neonatal deaths, saving approximately two million lives per year at a low-cost [
2]. Furthermore, available interventions can reduce the major cause of neonatal mortality—preterm, intrapartum, and infections-related deaths —by 58%, 79%, and 84%, respectively [
2]. Examples of these interventions include maternal immunization, screening and management of infections, preventive treatments for malaria, emergency obstetric care (EmOC), and immediate procedures for neonatal care [
2,
3].
Countries with high neonatal mortality, including Mozambique, have clinical practice guidelines to implement these interventions along the continuum of care in health facilities and at the community level. Despite these guidelines, implementation is inconsistent, with only 47% of recommended care provided by providers [
4,
5]. Additionally, the coverage and quality of evidence-based interventions are uneven in low-resource countries, primarily due to poor service readiness (lack of financial, material, and human resources), lack of provider training, weak provider awareness of current clinical guidelines [
4,
6,
7], lack of accountability of provider performance, and poor leadership and management capacity [
8,
9].
The utilization of maternal and child health services in Mozambique is high. In 2015, an estimated 93% of pregnant women attended a first antenatal care (ANC) visit, 73% gave birth in a health facility, and 76% of children ages 12–23 months received the third pentavalent vaccination [
10]. Compliance with clinical guidelines can be improved by implementing structural changes and strengthening leadership and management across different health system levels [
11,
12], facilitating improvements in health service utilization and neonatal outcomes.
Audit and feedback (A&F) is an evidence-based implementation strategy used in healthcare settings to systematically evaluate individual professional practice or performance based on targets or standards and improve health professionals’ compliance with guidelines [
13]. However, more evidence is needed on how to use A&F most effectively [
14]. Most studies on A&F use a randomized controlled trial design and have been conducted in high-income settings and demonstrate small to moderate effectiveness [
13,
15,
16]. Uncertainty remains about the potential impact of A&F in low-income settings where the disease burden is higher, health systems are weaker, and baseline implementation of clinical practice guidelines has considerable space for improvement. Further evidence generation is needed from these settings, where more significant effects can be expected.
The Integrated District Evidence to Action (IDEAs) program is a multi-component A&F implementation strategy that aims to improve the coverage and quality of a bundle of existing evidence-based interventions targeting major causes of neonatal mortality. Funded by the Doris Duke Charitable Foundation and the National Institutes of Health, IDEAs was implemented between October 2016 and December 2020 in two provinces across 12 districts in central Mozambique. The goal was to determine the effectiveness of the IDEAs intervention strategy, led by district management teams, to serve as a foundation for national scale-up.
The Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework is one of the most frequently applied implementation science frameworks [
16‐
18]. RE-AIM was developed to guide research in complex real-world settings and has great potential to provide detailed, nuanced information on whether and how quality improvement interventions succeed [
19,
20]. Reach describes the absolute number, proportion, and representativeness of individuals willing to participate in the intervention. Effectiveness describes the impact on selected outcomes. Adoption captures the absolute number, proportion, and representativeness of settings and intervention agents willing to initiate a program. Implementation represents the intervention agents’ fidelity to the components of an intervention’s protocol, including consistency of delivery as intended, time spent, and associated costs. Maintenance refers to the extent to which a program or policy becomes institutionalized or part of routine organizational practices and policies [
18,
19]. Guided by the RE-AIM framework, we report on the IDEAs A&F implementation process and outcomes in this article. We hope our findings will help inform and improve the IDEAs program’s future replication, adaptation, or scale-up.
Discussion
The IDEAs strategy was designed to support achieving the Sustainable Development Goal target 3.2, focusing on reducing newborn and child mortality. The strategy utilized MCH managers to lead an A&F process aiming to improve the implementation of evidence-based interventions targeting causes of neonatal mortality by identifying and solving facility-level gaps related to the delivery of services according to MOH guidelines. The number of districts and facilities exposed and adopting IDEAs was close to the proposed targets, indicating that the program was successfully introduced and implemented in most primary healthcare facilities across 12 Manica and Sofala provinces districts.
Most facilities completed the intended nine cycles of A&F meetings. However, a handful of facilities did not complete all cycles due to a variety of reasons, such as competing activities during scheduled A&F meetings that prevented MCH managers from participating (e.g., health campaigns, MOH visits, or other supervision visits), as well as unavailability of MCH managers due to vacations, maternity leave, and sickness.
Notably, the participation of MCH managers was high across all levels (provincial, district, and facility) in all 107 cycles, and most participants were from the facility level. This distribution in participation was important because changes needed to improve guideline compliance focus on the service delivery (facility) level. The presence of district and provincial managers was essential for reinforcing the accountability of healthcare providers and harmonizing operational recommendations to implement action plan activities, as described in a separate qualitative study conducted at an earlier stage of the strategy [
25].
Positive intervention fidelity was also verified in the duration and frequency of meetings for most districts. However, Barue and Mossurize districts presented the shortest average intervals between meetings (5.3 and 5.5 months, respectively), which could potentially reduce opportunities to implement action plan activities.
The selection of facilities based on performance to receive supervision visits and financial support also had high fidelity, as most facilities had the opportunity to receive supportive supervision.
Identifying and specifying problems and micro-interventions improved over time as part of the action-planning process. A higher number of problems were identified in the first two cycles, and poorly specified micro-interventions were proposed during this period, posing challenges in monitoring their implementation. As a result, around the fourth cycle, the IDEAs team recommended that district teams limit the number of problems per facility to five to improve problem prioritization, specification, and implementation of micro-interventions.
Many identified problems were repeated across different facilities within the same district. This repetition could be attributed to the fact that health facilities faced similar challenges, or it could be due to the group setting of the meetings, which may have influenced participants to think similarly when developing action plans.
The consistent identification of the same problems across multiple meeting cycles suggests that these issues persisted or that the definition of the problem was too broad. For example, the “weak diagnosis or management of EmOC” issue was prioritized in all nine A&F cycles when data from all districts were aggregated. However, different facilities may have observed varying degrees of weaknesses in different components of EmOC. While some issues might have been partially addressed, these levels of detail were not reflected in the listed problems. A better specification of the problems and examination of specific indicators are necessary to better understand progress in addressing these issues. Conversely, identifying a problem in just one cycle may indicate that the issue was resolved or that other problems were deemed a higher priority in subsequent cycles.
We also analyzed whether alignment existed between the identified problems, proposed micro-interventions, and the existing literature or formal guidelines and found successful alignment. For instance, the main problem identified in A&F cycles was weak diagnosis and management of obstetric complications, a well-documented weakness in Mozambique [
26,
27] as in many other low-resource settings [
28,
29]. Specific actions written in action plans to address this issue included reinforcing the use of partograms to monitor labor, recording the entire case history of patients, conducting complete physical examinations of all pregnant and postpartum women upon entry into the maternity ward and before discharge, and providing on-the-job training in EmOC. These actions are all recommended in national norms for childbirth, newborn care, and obstetric complications in Mozambique [
30].
Poor fidelity was observed in conducting semiannual SRAs before A&F meetings and in reporting micro-interventions implemented successfully. Readiness assessment delays resulted from challenges in elaborating the assessment protocol, delays in IRB approval, and failure to rapidly synthesize results to feed back into the A&F meeting after conducting the readiness assessment. Given the delays and high cost, this component was not supported throughout the study. We believe that the impact of not having SRA before each A&F meeting was minimal because MCH managers were knowledgeable about the availability of resources based on other sources, such as supervision visits and routine facility reports.
Reporting the extent to which micro-interventions were completed was sub-optimal, especially at the beginning of the program, due to challenges in monitoring the large number of poorly specified micro-interventions. Additionally, comparing this indicator between the two implementing provinces found better documentation of micro-intervention implementation in Sofala than in Manica province, suggesting that MCH managers from these provinces had different levels of experience in evaluating action plan implementation. One possible explanation for the differences could be that Sofala had prior experience with a similar implementation strategy piloted before the expansion to include Manica province [
31].
Quality improvement strategies for maternal and child health have been studied in other developing countries [
32,
33]. For A&F in particular, the bulk of evidence available reports on the effectiveness, mostly from high-income settings where the magnitude of the effect varied between a -17–49% improvement in professional practice and null effects for improvements in distal health outcomes [
16,
34]. While we agree that it is essential to assess the effectiveness of new strategies, we want to highlight in this manuscript the need for evaluating and reporting implementation processes of A&F in practice in a resource-constrained setting. By gaining this knowledge, we can develop data-driven and contextual strategies to enhance service delivery and quality of care. For instance, in the case of the IDEAs strategy, we identified priority problems, including weaknesses in clinical practice and challenges in implementing micro-interventions, indicating that other well-known bottlenecks affecting health services need to be studied and targeted, including health workers shortages, clinical skills gaps in the management of care, absence or stock-outs of essential commodities and supplies, weak leadership, and inadequate resource allocation [
35] and that other strategies, such as training of nurses in EmOC and direct funding of health facilities instead of districts, should be considered to add to the A&F strategy.
This evaluation has several limitations. First, it is essential to note that no statistical inference is made as the evaluation focuses solely on describing the implementation process and outcomes; the strategy’s effectiveness in improving availability and quality of services will be reported in a separate manuscript. Second, population reach was evaluated using indirect indicators of pregnant women using antenatal services. Third, our approach to categorizing problems and micro-interventions based on predetermined categories could potentially obscure important details about these components. Lastly, this study took place in only two provinces of central Mozambique, and therefore, it may not be appropriate to generalize the results to other locations. Despite these limitations, the study also has notable strengths. We applied the RE-AIM framework to plan, evaluate, and report the implementation process and impact of the IDEAs strategy. RE-AIM is a well-recognized and widely used framework in implementation science, and its use provides clarity on how strategies are being designed and evaluated, allowing meaningful comparisons between similar studies.
Additionally, the detailed description of a 4.5-year implementation process and its outcomes enabled highlighting the perspectives of MCH managers related to priority challenges and strategies to overcome them, as well as to detect weaknesses in the components of the IDEAs strategy in responding to those challenges. This information is valuable in recommending refinements or adaptations to improve implementation fidelity and in discussing the strategy’s potential impact on improving health outcomes. Moreover, the cyclical nature of the IDEAs strategy offered an opportunity to build the capacity of MCH managers in Manica and Sofala in using data for learning and decision-making and in proposing tailored evidence-based interventions towards improvements of readiness and quality of care, contributing to the global effort of improving health service delivery to reduce neonatal mortality.
Based on the lessons in implementing IDEAs, we recommend refinements of the strategy components to improve fidelity in implementation, namely, remove the semiannual SRA component or reduce its frequency and only assess a sub-sample of SRA items that are related to indicators examined in the A&F meetings. This approach would improve practicality and reduce the costs of conducting SRAs. In the A&F meetings, each facility should elaborate its action plan separately from other facilities to avoid potential replication in identified problems and solutions, as the definition of problems and micro-interventions need to be specific to the individual health facility context. Additionally, it is crucial to explore the challenges in implementing micro-interventions, specifically the availability of resources and clinical competencies. In the IDEAs case, understanding the impact of funding to the districts to support health facility action plan implementation and reasons for weakness in clinical practice are essential to improve the design of the strategy or identify other strategies that can be added to A&F.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.