Evaluating implementation of the World Health Organization’s Strategic Approach to strengthening sexual and reproductive health policies and programs to address unintended pregnancy and unsafe abortion

There is a gap in research on understanding the processes and mechanisms by which complex interventions are implemented in practice [9‐11]. Many SRH programs report on program impacts without describing how, when, where, and under what conditions programs were developed and implemented [12]. As a result, it is difficult to replicate programs that are successful in achieving intended outcomes. For unsuccessful programs, it is unclear whether the program is ineffective, whether it was limited by contextual factors, or whether it was not implemented well [10]. Process evaluations can help to unpack the “black box” of implementation by providing insight into how program components interact with factors in the implementation context that help or hinder implementation processes [11, 12].

We conducted a process evaluation guided by the following objectives:

The primary document review phase (conducted between April and December 2015) was designed to collect and analyze the strategic assessment reports for all 15 countries. We then identified and analyzed secondary documents (e.g., country background papers; historical documents; national standards, policies, and guidelines for abortion care; donor reports; and information educational materials).

A librarian conducted a literature search of databases (e.g., MEDLINE and EMBASE) and grey literature in July 2015 to identify English-language articles (published between 1946 and 2015) on the SA and reproductive health issues in the countries of interest. Search terms consisted of keywords, such as “World Health Organization”, “strategic approach”, “abortion”, “birth control”, “contraception”, “fertility control”, “family planning”, “unwanted pregnancy”, and “unplanned pregnancy”, (Additional file 2). Two reviewers independently screened the titles and abstracts of records to select articles for full-text review. Articles were included for full-text review if they explicitly mentioned the WHO’s SA recommendations and/or described SA initiatives being implemented at the national or local level in any of the 15 countries of interest. Articles were excluded if they did not directly mention the WHO’s SA or described initiatives outside of the countries of interest. Inter-rater reliability was calculated to assess the degree to which the two reviewers consistently agreed on which articles were eligible for full-text review. The calculated Kappa coefficient, k = 0.82, indicated substantial inter-rater agreement [13]; discrepancies were reconciled through discussion. Full-text versions of the selected articles were then retrieved and reviewed by two independent reviewers to determine eligibility.

Interviews were conducted between September 2015 and January 2016 to explore perceptions of two stakeholder groups: (1) frontline implementers who used the SA for policy and program strengthening in their countries and (2) technical staff who had a role in development, operations, or technical support of the SA activities in these countries (Fig. 2).

Stakeholders were identified and contacted by the WHO and introduced to the research staff. The research staff asked country stakeholders to provide additional documents and names of potential stakeholders who could be recruited for interviews through purposive and snowball sampling. Interviews were 45–60 min in length and conducted in English by experienced interviewers via telephone or in person using semi-structured interview guides (Additional file 4). Interview guide development was informed by research on factors that affect the implementation process [15]. Interviews were audio recorded and transcribed verbatim and de-identified prior to analysis.

A multi-source (i.e., document and literature review and interviews) qualitative method triangulation technique [21] was applied to analyze the data. Data collected from all sources were combined through a matrix where data were mapped according to country, SEM, and stage of the SA. This method allowed analysts to consider data from the document and literature review and interviews collectively to better understand how the SA was implemented in each country.

For the primary document review, we analyzed 15 strategic assessment reports. For the secondary document review, we analyzed 61 documents from country stakeholders and 69 documents from the literature search. The literature search (Fig. 3) yielded 169 records after duplicates were removed. After screening for title and abstract (Stage 1) and conducting full-text screening of articles (Stage 2), 69 documents were included in the final analysis. Additional file 7 provides the complete list of citations.

The 69 documents selected from the literature search described a wide range of SRH policies, programs, and practices. Of the 44 peer-reviewed articles, 13 articles examined abortion methods and practices, 11 articles evaluated programs to improve access to and quality of SRH services, eight articles reported the incidence of abortion and/or abortion-related deaths, four articles were related to reproductive rights and policy reform, three articles described social factors and attitudes towards contraceptives and abortions, three articles examined specific strategies and tools to improve the quality of SRH services, and two articles offered guidance on providing high-quality SRH services. The 25 grey literature documents included progress reports, highlights reports, as well as biennial and annual technical reports from WHO’s Department of Reproductive Health and Research.

We completed 19 interviews with a response rate of 83%: 11 with technical staff and eight with frontline implementers.

Figure 4 presents the timeframe during which the SA was implemented in each country. All 15 countries completed Stage 1 activities. All except one country (Guinea) initiated Stage 2 activities based on Stage 1 outputs/recommendations. Seven countries (i.e., Ghana, Moldova, Mongolia, Romania, Russian Federation, Ukraine, and Vietnam) were perceived to have simultaneously engaged in Stage 2 activities (i.e., piloting certain innovations) and Stage 3 activities (i.e., scaling up other innovations from Stage 2). Interview participants could not confirm the status of activities (i.e., whether these activities were ongoing or completed) in the countries that had initiated Stage 3. Examples of common country-specific activities by stage are provided in Table 1.

A number of factors were perceived to have influenced the implementation of the SA overall and progression through the SA stages.

Interview participants’ descriptions of how the SA was implemented in their settings and perceptions of the key strengths of the SA directly aligned with the essential features or guiding principles of the approach itself: country-driven, participatory, and staged process using a systems framework.

This evaluation identified key success factors for implementing the SA and opportunities to leverage these factors to enhance ongoing and future implementation efforts in varied settings. It also identified country stakeholders’ key role in developing, piloting, and scaling up policies and programs to address SRH issues in their respective countries. First, countries that made the most progress towards implementing the SA were successful in engaging and maintaining country stakeholder support beyond Stage 1 activities. Opportunities to sustain engagement of country stakeholders beyond Stage 1 include sharing resources on SRH guidelines and interventions, inviting them to share their insights on how to plan for sustainability as part of Stage 1 activities and building capacity on how to scale up interventions, by sharing tools such as EXPANDNET capability building resources [22]. Second, clarifying that Stage 1 of the SA is an opportunity to better understand unintended pregnancy and unsafe abortion rather than to promote abortion services enhanced country stakeholders’ buy-in to launch the SA. Third, in some countries, following up Stage 1 assessment activities (i.e., qualitative interviews and focus group discussions) with quantitative data collection offered a complementary method for providing evidence to strengthen stakeholder support for reforming policies and programs. Fourth, establishing a central coordinating body (e.g., the strategic assessment team) to coordinate SA activities was perceived to facilitate SA implementation. When establishing a central coordinating body, countries can consider assessing leadership buy-in [23] or using coordinated leadership and organizational development strategies [24] to improve leadership and organizational supports for implementing and sustaining SA interventions. Countries can also employ a contingency plan for personnel turnover through an ongoing process of training and feedback to establish a stable team [25]. Coordination efforts can be further strengthened by using Communication for Development (C4D) strategies [26] to comprehensively target each level of the SEM for achieving the behavioral and social change needed to implement the SA. Fifth, leveraging Stage 1 outputs to obtain support and funding for Stage 2 activities facilitated implementation. For example, coordinating and integrating efforts of local NGOs and research organizations interested in reproductive health issues helped maximize in-country technical capacity to implement and evaluate Stage 2 interventions. Securing technical assistance from the WHO and other international organizations early on helped countries avoid losing the momentum or enthusiasm generated during Stage 1. Sixth, countries that monitored and reported on SA interventions were able to use these data to show progress and secure additional funding. Incorporating a formal evaluation plan (including process and outcome metrics) and using SRH reporting tools, such as Programme Reporting Standards for SRH programmes [27], can help countries ensure consistent monitoring of process and outcomes.

There are limitations to evaluating the process of implementing the SA. First, interviews were conducted with a relatively small group of individuals. The SA is a multi-year process, and in some cases, retrospectively evaluating the process meant that the stakeholders who were initially involved with SA activities were no longer involved with ongoing efforts. This may have limited the number of country stakeholders available to participate in the evaluation. Second, the study design and methods were influenced by considerations of feasibility in terms of available resources, number of country sites, and availability of information. For example, face-to-face engagement with country stakeholders may have encouraged more responsiveness than email contact and telephone interviews; however, the former was not feasible due to resource limitations. Language and communication barriers in cases where English was not a first language may have also affected stakeholder involvement with the evaluation. Third, there was considerable variation in the quantity of data available for each country, which may affect the generalizability of findings. In addition, information from documents was not always linked directly to the SA. Most country documents had limited descriptions of Stage 2 and 3 activities, and these were often not explicitly linked to Stage 1 recommendations. Fourth, the document review phase included English-language articles only, potentially limiting the review of related documents in other languages. The document review process also required interpretation by the study team, which may have led to an information bias, misclassification, or errors in linking activities to output variables. We note, however, that the data calibration exercises were done with good agreement, and the document review phase was followed by interviews, which enabled the study team to supplement findings by collecting richer insights on factors related to implementation process and context [28]. Finally, because bias may have been introduced due to self-reporting, findings may not be applicable to other implementers and their experiences. The study team designed interview questions to encourage participants’ retrospective reflections on the implementation process, and data from multiple informants was compared and synthesized, which likely helped reduce bias from memory. Conclusions were based on the consistency of data from multiple sources and triangulation of data aimed at increasing the rigour of the data collection and analysis process.

This study focused on how the SA was applied in 15 countries to address issues of unintended pregnancy and unsafe abortion. Although we could not directly assess the quality with which each country applied the SA, the alignment between the perceived strengths of the SA and the key elements of the SA (e.g., country-driven, participatory, and systematic) suggest that the SA has been implemented as intended in most countries. The study identified gaps in the implementation process that should be addressed as well as opportunities that could be leveraged by others who are considering using the SA to address SRH issues in their countries.

Future studies may consider conducting an outcome or impact evaluation using quantitative outcomes data along with process data to identify changes in health systems and outcomes as a result of applying the SA.