The findings from the study are set out against the three PbR mechanisms outlined in the research framework: incentive effects, systems effects, and modality effects.
Incentive effects
During the design stage of WISH, Department for International Development (DFID) advisors felt that the ‘quantity’ focus of coverage targets required offsetting to ensure that disadvantaged populations were reached and that other goals, such as sustainability, be included. DFID
3 viewed its main remit to be poverty reduction,
4 and therefore held that WISH should reach the poorest in each country. It was also clear that young people (defined as those under 20 years of age) face particular barriers to accessing SRH/FP services, especially the potential of stigma. PbR was therefore explicitly linked to the goal of improving reach to the poorest and to adolescents.
The tender document set out that PbR would be applicable and provided a list of countries where the programme could operate, inviting applicants to specify where they would operate.
5 The successful consortia came in with the full list of countries in the tender, and with a maximum proportion of fees (100%) linked to the PbR KPIs. The scoring structure in the Invitation to Tender (ITT) incentivised the applicants to include the maximum number of countries possible and to place a large proportion of fees with the PbR mechanism and hence at financial risk.
For both lots, consortia partners used their own historic data to model what could be confidently expected from the programme in terms of delivery against the PbR KPIs, where that data was available. There was not necessarily up-to-date national data on the poverty status of many of the countries in both lots. For Lot 2 (led by IPPF), the lack of baseline data on FP service users’ poverty status was particularly challenging. Both consortia felt that it was not clear how reliable, comparable, and representative the poverty KPI would be. Another issue was trying to identify a single indicator, particularly for adolescents, that could be relevant to different demographic contexts: for example, a very ambitious target in the South Asia context would likely be unambitious in Sahelian countries (due to demographics).
The WISH programme was hit by the COVID-19 pandemic 17 months into implementation. The national lockdowns, closures of facilities, commodity procurement and supply challenges, as well as the need to provide services safely with new social distancing procedures, led to two adaptations: (1) a COVID waiver in 2020, and (2) a ‘post-COVID’ waiver in 2021. The former led to the suspension of PbR for six months, while the latter gave partners the opportunity to make a case for waiving some PbR risk based on how COVID-19 made the achievement of the PbR KPIs less feasible. In both cases, waivers had the greatest effect in addressing risk relating to the poverty PbR KPI (i.e., eliminating the financial risks from the relative underperformance on this KPI).
Even in the COVID-19 context, the use of PbR KPIs clearly incentivised the programme to make a significant effort to reach adolescents and people living in poverty. Several strategies had been put in place to reach these specific communities in the three case study countries. To reach more adolescents, strategies included training of service delivery staff and mobilisation activities aiming to locate services closer and more conveniently for the adolescent clientele, while also providing services that were appropriate to younger people (e.g., extending opening hours to offer services on evenings and weekends). Other strategies included direct outreach in universities and industrial parks, adolescent group sessions, engagement of boys and men at community level and through mass media, peer-to-peer approaches, and the use of an outreach service delivery channel to proactively reach young people.
The COVID-19 context makes it difficult to estimate the effectiveness of these strategies on achieving KPIs. For the Youth PbR KPIs, the country and portfolio goals were met. However, there was no increasing trajectory of adolescent access over the three years of the programme – i.e., the average proportion of FP users under 20 years of age neither increased nor decreased significantly for either lot. The first Youth PbR KPI of 5% of service users to be under 20 years of age applicable for all countries was easily met for most countries, though some countries made considerable efforts to reach this indicator. The second Youth PbR KPI of 15% across the consortium was also reached by both lots. This was more ambitious and several countries remained under this threshold.
6 However, both Youth KPIs illustrate the limitations of using a single indicator across countries with very different contexts. There are reasons to suggest programming for young people became more difficult during COVID-19 lockdowns in several contexts, not least with the widespread school and university closures across countries, through which some awareness-raising interventions were focused.
Strategies adopted by WISH to reach more people in poverty included programming decisions about where to geographically locate services, the removal of service fees, non-discrimination training and a no-refusal policy. At the global level, most stakeholders mentioned the use of poverty-mapping heat maps for locating services and, in a few instances, these heat maps were used to close sites in areas that did not serve excluded populations and open sites in entirely new geographies.
Despite developing strategies in response to the PbR mechanisms, many countries did not meet their poverty PbR KPI in either the first or second year. The KPI was defined according to national benchmark rates, as per the World Bank’s absolute poverty threshold of USD 1.90 Purchasing Power Parity (PPP) per day.
7 Each country would measure service users who should be in poverty in proportion to the national benchmark level to achieve the PbR KPI.
8 The poverty PbR KPI had the most significant performance shortfall across the PbR KPIs and carried significant financial risk. This underperformance was in part attributed to the measures used. The most common measure used was the Poverty Probability Index (PPI), which estimates the poverty propensity of a given population using a small number of questions for a survey sample. The PPI was originally designed as an ‘ease of use’ measure for a household survey setting because of its features: it is a quick to collect, easy to calculate and low cost to administer, rather than a precise and accurate, measure (Schreiner, 2018) [
19].
The PPI and Multidimensional Poverty Index (MPI) are well-established estimates of poverty. However, they both use their unique metrics and baselines and therefore come to different results when measuring the same sample. They also use benchmark national surveys that are often out-of-date (as early as 2009) and do not reflect the contemporary situation. In addition, by using national benchmarks that covered all ages, the poverty rates of women of reproductive age were likely to be overestimated, making it more difficult to achieve the KPI.
9 Administering the MPI in a healthcare exit interview setting rather than a household survey setting brought its own challenges. For example, it was not possible to collect the anthropometric data to compute the nutrition indicator. This was mitigated by not measuring nutrition status and instead doubling the weight of the other health sub-indicator (child mortality).
As these methodological issues became apparent with the PPI, Lot 1 (MSI) shifted from using the PPI measure to capture the poverty data in the first and second year to using the MPI measure in the third year. Other changes were made, for example, only using the national benchmark average as it applied to women of reproductive age and dropping the nutrition measures that required close physical proximity to collect during COVID-19. Despite the efforts to address these methodological issues, there remain ambiguities, and this creates a degree of uncertainty about the actual performance on poverty in the WISH programme.
More broadly, the programme was successful in achieving the quantity-focused KPIs (CYPs and AU), and the sustainability PbR KPI. These quantity targets were associated with the greatest financial values, and many respondents felt these were prioritised whilst implementing WISH. For some respondents, the CYP targets were too high, adding significant pressure to their work. For other respondents, the competing PbR KPIs created a cost-effectiveness trade-off as the more remote geographical areas may have offered greater opportunities for reaching people in poverty and addressing unmet need for contraception but would do less well on the CYP goals at the same cost. However, for some respondents, the strategies for achieving the PbR KPIs were seen to be complementary: the same activities that were promoting adolescents or reach to people living in poverty would also be beneficial for the AU and CYP KPIs. Reaching people with disability was viewed by most stakeholders as a success of the WISH programme; however, because reaching people living with disability was only included as log frame indicator and not as a PbR KPI, this study was not able to assess its effects compared to the equity PbR KPIs - youth and poverty.
System effects
Across the programme, the PbR mechanism was viewed as being largely positive and respondents were highly motivated by the programme level incentives: “These kinds of approaches have a motivational quality. That is because once you understand that you need resources to run your programmes, but you [sic organization] only get paid once you do the work, you will have the motivation to do a better job and to strategize different approaches and interventions for your work.”
Other associated improvements are in the data, learning and evidence systems developed to meet the reporting requirements of the WISH programme. This included the data management systems and the client exit interview (CEI) data. Planning systems were also enhanced: PbR brought a clearer approach to planning to achieve results, including how this led to the development of strategies to address inequity and respond to new information as and when it was available. One respondent stated: “[PbR] makes you work, in order to bring the required result, you plan different strategies based on available options, and by executing different activities in different ways. So, in order to get the required result, it makes you focus and work with effort by implementing different strategies, because if there is no result there is not going to be any payment.” The PbR KPIs were not seen as detracting or diverting attention from other priorities such as quality of care, availability and informed choice as this was embedded within existing protocol and procedures, log frames and professional training.
There were several systemic challenges that limited the possible incentive effects of the PbR KPIs. Decisions on where to geographically locate service sites are complicated (and not entirely within the powers of the implementing partners, as this is overseen by local health authorities), and the idea of being able to continue to adapt service locations in response to new information on poverty was neither realistic nor necessarily desirable. In Ethiopia, one respondent stated: “More than half of the outreach clients are below the poverty line. The public facilities support is static. We can’t move them. We can’t change location once we start providing support.” In contexts where there are no other providers, there are also ethical implications with respect to (re)moving outreach services. As one respondent stated: “You give women implants, it’s harder to take one out, you want to deliver that”, but “how must those women feel, when those services aren’t there?”. There were, therefore, both ethical and contextual restrictions driving the selection of sites and restricting the ability to change sites once they were set, limiting adaptive programming.
Modality effects
A distinguishing feature of PbR as a contractual mechanism is that finance is at risk. For any PbR mechanism to be credible, there must be a genuine risk of non-payment and financial loss to the agent organisation and for suppliers to be rewarded for achieving beyond the required performance. For WISH, the total financial risk (‘fees retained for PbR’) was shared across the consortia, based on financial modelling and negotiations at the start of the programme. The risk allocation within consortium partners was complex and required significant negotiation and diplomacy amongst partners in terms of both understanding the risk involved and risk tolerance, particularly for those without prior experience of PbR.
The final financial outcome for PbR was found to be likely neutral (i.e., there were overall no losses incurred). There was a risk that the programmes would run at a loss, though this did not happen. The potential impacts on cashflow due to underperformance on the poverty PbR KPI results were bridged by timely performance on the other KPI payments, particularly the Youth KPIs. Further, there was a degree of flexibility in the PbR mechanism granted by FCDO due to COVID-19: the COVID-19 waiver in 2020 and the ‘post-COVID’ waiver in 2021, combined with careful cashflow management by the consortia, ensured that fees were not lost across the two lots. The flexibility represented by the waivers ultimately meant that the poverty KPI shortfalls would not lead to fees being withheld.
Financial risk was not standard across the consortia. How risk was cascaded to different institutional levels varied, and consortia members negotiated how risk was pooled, which meant that partners faced different degrees of risks. For example, the Youth (b) PbR KPI – the 15% average of reach to those aged under 20 across the portfolio – was achieved across the Lot 2 consortium early in the programme. However, one partner had chosen not to pool risk with other members at the negotiation phase and uniquely faced financial risk. The different ways that the financial and cashflow risk were distributed, including PbR being flexibly applied across consortium partners, worked to protect the cashflow of the organisations. This assumes the ability to pay for the services upfront; and only large, well-resourced organisations would be able to meet these criteria.
The PbR reporting (and its verification) itself represented a significant time and cost burden for team members involved, and therefore an opportunity cost, though this is difficult to quantify. This included the time and energy spent in PbR modelling and analysis, managing risk within the consortia, as well as in negotiating aspects of the PbR with FCDO. While “making sure we have quality in our data is a priority”, the processes of verification via the third-party monitoring body also require “a lot more documents, bureaucracy, etc.” Furthermore, “this takes a lot of people’s time. Then we have calls, to explain processes, etc., and this is a burden for us to explain and provide more information.” The reporting requirements of the PbR generated stress, one respondent stated: “[It] is also a stressful experience requiring a lot of effort to implement and collect the evidence because the programme is performance-based to get pay-outs.”
There were positive consequences on ways of working among the partners and the two consortia, including the rich learning exchange between the two lots, particularly between MSI and IPPF. An effect of this collaboration was in aligning changes and contractual amendments between the two lots over the course of WISH, including the discussions on poverty methodologies. In practice, while the two lots tended to converge around mutually beneficial contract and methodological amendments, differences between the two lots remained due in part to different preferences and organisational modus operandi. It is not clear how much additional time and cost was generated by negotiating with both lots for FCDO itself, although it is likely the amount of time and cost diminished over time because the two lots collaborated and presented a unified approach that may have reduced the overall transaction costs.