Applying the results based management framework to the CERCA multi-component project in adolescent sexual and reproductive health: a retrospective analysis

Adolescent pregnancy can negatively affect the health and well-being of adolescents. Complex problems like adolescent pregnancy require multi-faceted prevention interventions that are carefully planned, implemented and evaluated in order to satisfy funders and community members. The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project (2010-2014) set out to lower adolescent pregnancy in Bolivia, Ecuador, and Nicaragua through multiple ways: increasing communication; increasing the number of adolescents who can get sexual and reproductive information and services, and increasing the number of adolescents using birth control. When the project ended, CERCA had not met all of its goals. In this paper, we studied the CERCA project using a specific project management framework called Results Based Management (RBM) to learn more about the project’s strengths and weaknesses. RBM is a system that can enhance effectiveness and verify that the project is going in the right direction using feedback loops at each level. Using the RBM framework, we found important lessons in the 3 main elements of RBM, which are (1) planning, (2) monitoring and (3) evaluation.

(1) We learned that what was proposed in the project differed from what was carried out. (2) We also learned that the way the project’s progress was monitored did not meaningfully allow for an understanding of what was being done. (3) We also learned that measuring the impact left some unanswered questions. This analysis using RBM sheds new light on lessons learned from the CERCA project and may be useful in future adolescent SRH initiatives.

Multi-component, community-integrated health intervention packages are necessary to address complex social problems such as adolescent pregnancy [9]. However, these intervention packages are often themselves complex and require intensive planning, sound implementation, and rigorous evaluation to ensure effectiveness, community acceptance, and return on investment by non-governmental organizations (NGO) and government agencies [16, 27]. Multi-country programmes have the opportunity to create broad, population-based change, but without the use of evidence-based approaches to create intervention packages and appropriate management of interventions, their impact may fall short ([11, 18, 22, 25].

In Latin America, the need for health intervention packages is particularly pressing in the area of adolescent sexual and reproductive health (SRH) as adolescent birth rates are among highest in the world [31]. The adolescent birth rate has remained persistently elevated, declining only from 77 to 72 per 1000 women of 15-19 years of age between 1990 and 2005, while access to contraceptive methods and comprehensive sexuality education remains restricted among adolescents [31]. Bolivia, Ecuador and Nicaragua face particularly high adolescent birth rates and associated poor health and psychosocial outcomes. In Bolivia, the adolescent birth rate is 89.1 per 1000 women (2005), Ecuador 99.6 (2002) and Nicaragua 92 (2010), thus are settings warranting special consideration for complex interventions [31].

A recent initiative titled Community-Embedded Reproductive health Care for Adolescents (CERCA) Project (2010-2014) aimed to reduce adolescent pregnancy rates and improve SRH of adolescents through a comprehensive strategy to modify health behaviors related to communication, SRH information-seeking, SRH care-seeking, and promotion of safe sexual relationships. To achieve these changes, CERCA implemented a strategy involving multiple stakeholders such as adolescents, parents, teachers, health personnel, health and education authorities, community and local authorities in three settings: Cochabamba (Bolivia), Cuenca (Ecuador) and Managua (Nicaragua).

CERCA was implemented from March 2010 and concluded in March 2014. The key institutions leading the implementation were The International Center of Reproductive Health-Ghent University in Belgium as the general project coordinator) and South Group in Bolivia, University of Cuenca in Ecuador, CIES and ICAS in Nicaragua, Kaunas University in Lithuania, and the University of Amsterdam in The Netherlands as the country implementers [4, 8]. The quantitative evaluation of CERCA, using a pre- and post-test controlled design, did not demonstrate effects after twenty months of intervention in the following key areas: reinforced behaviour, knowledge, attitudes, skills, and self-reflection. This lack of measured success was primarily attributed to challenges in data collection described in the CERCA report [16] amid others. A qualitative post hoc impact evaluation, supported by the World Health Organization, was carried out from June 2014 until March 2015 and aimed to expand upon the quantitative evaluation of the CERCA project (Crescendo [5]). This qualitative analysis revealed that a concerted application of a multi-faceted strategy reduced taboo in a conservative setting, sensitized the community and authorities, enabled the environment for adolescent interventions, and that subsequent, multi-country projects warrant a more thorough preparatory phase [13].

In the past, many health projects have been implemented without a clear idea of whether they did, in fact, achieve the expected results due to incomplete monitoring or inadequate evaluation phase [32]. However, more recently, a number of bilateral agencies and United Nations’ programs have adopted the Results Based Management (RBM) framework to demonstrate aid effectiveness and accountability from project planning to completion [1, 26]. We aimed to understand whether an RBM framework might have helped CERCA produce evidence of tangible and positive results by allowing country implementers to more effectively address challenges in design, implementation, monitoring, and evaluation.

The objectives of this paper are to apply the RBM framework retrospectively to the CERCA project in order to reflect on lessons learned and provide recommendations for future similar multi-country projects with complex, community-integrated interventions. This will be conducted through the application and analysis of three main components of RBM framework: 1) Planning, 2) Monitoring and 3) Impact evaluation. Findings may offer project managers, implementers, evaluators, and funding agencies valuable information regarding adolescent SRH intervention packages and improve effectiveness in management, accountability, and creation of evidence in subsequent SRH projects.

RBM is a system designed to support efficiency and effectiveness at different stages of the project cycle starting from planning, designing the intervention package, monitoring, and supporting impact evaluation. RBM uses feedback loops at each of these levels. In complex interventions, this framework helps ensure that the project implements a package of interventions that is necessary and sufficient to achieve the expected result [29, 30]. Its goal is to enhance effectiveness and provide avenues to verify if the project is progressing appropriately through three phases: planning, monitoring and evaluation. Planning relies on a Theory of Change (ToC) framework which is the basis for the intervention package (IP). The ToC explains how a project will achieve a series of outcomes, by outlining causal relations based on assumptions to reach behavior change [15]. It involves choosing strategic interventions that are necessary and sufficient to reach the expected outcome [29, 30]. Monitoring tracks the process of change and how to ascertain success, tracking the implementation progress against the expected results of ToC [21]. Quantitative and qualitative evidence is collected for defined indicators to help determine the expected results from the intervention. Impact evaluation compares the control and intervention groups to assess the degree to which the impact can be attributable to the project after controlling for all external variables. Impact evaluation allows for decisions about whether project activities should be continued, replicated, scaled-up and improves the overall understanding about why certain changes have happened ([12, 21]. RBM is versatile enough to accommodate other research strategies within its qualitative arm such as community-based participatory research (CBPR). CBPR strategies foster empowerment in the community with the creation of knowledge as well as engagement, capacity building and quality assurance [10].

RBM can be used both retrospectively and prospectively to sharpen the strategic focus of the project. Organizations that use it for their project management are the Swiss Agency for Development and Cooperation and the United Nations Population Fund [24, 28]. In order to perform a retrospective analysis, we reviewed and analyzed a series of CERCA documents and other data sources (detailed in Appendix 1) structured by applying the RBM framework in order to see in what way the CERCA project performance might have reaped improved results. This included CERCA methodology design, final CERCA report 2014 (Crescendo [5]), internet report [23], post-hoc evaluation process report 2015 [16], international published papers [3, 4, 6‐8, 13, 14, 17] and the A-form for results. Records and discussion notes from two meetings (recorded without transcript) were also reviewed: 3-day meeting in Cuenca-Ecuador (February, 2014), and 3-day meeting in Ghent-Belgium with country project leaders, researchers from Ghent and the study advisor from WHO-HRP (December, 2014).

The CERCA programme relied upon three methodological frameworks: action research, community based participatory research and intervention mapping but did not adopt any project management framework. When the project ended, CERCA had not met all of its goals. Using the RBM framework, we expand the CERCA insights to demonstrate the importance of appropriate timeframes and cause-effect analyses while planning activities that can originate both from the community and from review of effective, evidenced-based interventions. Testing the activities for feasibility and acceptability in a pilot can enhance a project’s chances of success. Designing a clear monitoring plan for each activity, reporting failures during the process, and having constant feedback meetings on the advancement process against the ToC is paramount. Evaluation should assess cost-effectiveness of every intervention, success and failure factors to produce outputs and outcomes, and a breakdown analysis to measure the progress at the end of the project so that future initiatives can benefit of this information. This analysis conveys in a summary of lessons learned (Table 4).

Multi-component, community-integrated health intervention packages are necessary to address complex social problems such as adolescent pregnancy [9]. This analysis applied an RBM framework retrospectively in order to learn from a multi-country project to improve adolescent sexual and reproductive health. The analysis using the RBM framework highlights important findings of the project. In Planning, the proposed Theory of Change (ToC) included a range of stakeholders who did not consistently plan activities based on evidence, situation analysis, timeframe or a sequential path of change. Intervention packages changed according to different circumstances and did not always take into account guidelines or quality of delivery, leading to different results across sites which were difficult to compare. Some planned activities did not take into account the possibility to be continued, reproduced or scaled-up. We identified missed opportunities to select evidence-based interventions with appropriate linkages to a ToC that would ensure the intended outcomes were reached. In Monitoring, we also highlighted gaps due to time, resources, and variability across sites. The project oversight was mainly based on administrative and financial requirements rather than monitoring fidelity or challenges to carry out the intervention model regarding content, behavior change, quality of delivery and community perception. In Impact Evaluation, the project did not plan a comprehensive evaluation, did not dedicate sufficient time and budget. Limited data was available due to inadequate record keeping; the original evaluation did not identify success and failure factors related to the outcomes, and the cost-effectiveness analysis was not inclusive of all activities and costs (e.g. analysis did not account for training time and costs, challenges to monitoring their activities and their impact on every adolescent from the intervention group). Examining the project within an RBM framework generates valuable insight for future adolescent health initiatives.

Research is growing in the area of project accountability and assessing validity of evidence from complex interventions Our analysis with RBM is an effective accountability mechanism that controls at all levels and can strengthen management of large-scale projects that include cross-sectoral strategies. The use of RBM from the start to the end of the project encourages managers to make decisions with the results in mind and to plan all aspects of the project in advance with detail, essential for complex interventions [29]. Over the years, many NGOs and other institutions implementing health projects in the field of SRH have focused more on their respective donors’ needs and not on how the project contributes to a broader social change [20]. With the implementation of RBM in international institutions, donors are encouraged to support implementation as well as data collection, evaluation and sharing of information gathered. In the case of CERCA, project leaders had the opportunity to evaluate various factors of success and failure under the supervision of an advisor from the WHO-HRP who oriented these talks and helped understand the lessons learned (meeting in Ghent, 2014); but not all projects have this opportunity as most do not even have budget and time planned for this.

The RBM framework analysis has some limitations. Firstly, this is a retrospective assessment and CERCA did not use it in any phase. Secondly, our analysis relied upon the availability and quality of records kept during planning, implementation and evaluation (Appendix 1). Third, the authors of this paper were the project managers or project implementers in every country; therefore, this may introduce bias into the analysis. Finally, this analysis was conducted retrospectively after the qualitative evaluation and may be influenced by these findings.

This analysis demonstrates the delicate balance between planning comparable multi-country interventions and allowing each country to develop its own approach per their specific needs. A thoughtful project design incorporates a situation analysis of the unique needs of each site together with scientific evidence on what interventions have been shown to work and how they might work in each context. More research is needed in this area to evaluate the feasibility and effectiveness of complex intervention packages.

This analysis showed that multi-country projects with cross-sectoral strategies are complex, entail risks in execution and require rigorous project management. Large-scale, multi-country programmes have the opportunity to create broad, population-based change, but without the use of evidence-based approaches and appropriate management of interventions, their impact may fall short. RBM can be an effective accountability mechanism to ensure a systematic approach and appropriate use of resources at different levels within a multi-country setting. RBM can offer country implementers and managers an avenue to apply systematic management at all levels, improve accountability, promote the development of evidence base and strengthen commitment to adolescent health in future initiatives.