Strengthening scaling up through learning from implementation: comparing experiences from Afghanistan, Bangladesh and Uganda

While all three of the country initiatives sought to improve the quality of health services available to the rural poor, the intervention strategies employed across the three settings were quite different. The community scorecard in Afghanistan was thought by the research team and local stakeholders to be an appropriate intervention given the high levels of existing mistrust within communities, and particularly between communities and authorities (including health workers). The community scorecard was perceived as a means to engage multiple stakeholders, including provincial and district directorates, non-governmental organisations (NGOs), healthcare providers, councils and communities. At the same time, it was intended to strengthen social accountability and audit of healthcare services, enhance community governance and engagement toward state building, and build trust in a post-conflict environment. Further, it built upon an established system of balanced scorecards that had been employed nationwide to track the performance of health facilities and hospitals. The intervention in Afghanistan took place over a shorter period than those in Bangladesh or Uganda, and there was relatively limited evolution of the intervention during the FHS project so the intervention, as originally conceived, closely matched the final form of the intervention implemented by FHS; however, the intervention evolved after FHS had formally closed out its work. Specifically, an NGO was asked to test the same intervention in a different ethnic community, and after the successful conclusion of this, in 2015, the government passed a national policy mandating the community scorecard as a key strategy for all NGOs delivering the Basic Package of Health Services, and by 2017 the intervention was being implemented in five different provinces.

In Bangladesh, the project team started with a more fluid formulation of the intervention noting that they sought to support informal healthcare providers (VDs) in the intervention area through the provision of technology resources to formally trained healthcare providers or of decision guides. Initially, the project team planned to adopt two interventions, one, known as the ‘Health Box’, was intended as a self-diagnosis and treatment guide for communities, which could also be used by VDs, and the other was the telephone link to a medical call centre run by a private for-profit partner. The Health Box technology was dropped during the first year as it was found to be designed to assist in diagnosis and treatment of specific diseases such as HIV/AIDS and was not easily adaptable for a wider array of conditions.

The Bangladesh project team focussed their efforts on technologies to support VDs, and the call centre was launched in mid-2011. The call centre services were provided for a fixed rate fee set by the private for-profit firm. Unfortunately, by the end of 2011 it was clear that this system was not working. The number of clients using the call centre was low due to low levels of awareness and trust, resulting in insufficient financial gain for the firm to continue its services. Hence, the private for-profit firm lost interest in running the call centre and became unresponsive. Further, communication between patients and the call centre doctors was difficult due to the different dialects spoken. Subsequent to the withdrawal of services by the for-profit firm, ICDDR,B decided to launch its own call centre, building on what it had learnt from the earlier collaboration. At the time of writing, the ICDDR,B call centre is still functional but has not significantly scaled up its operations.

The FHS Uganda team consciously adopted a flexible intervention design, initially tentatively identifying a package of related interventions to improve the use of maternal health services but committing to work with stakeholders during the early years of the project through a participatory action research approach in order to refine the intervention. While, throughout this process, the three main pillars of the intervention remained the same (promoting community demand for maternity services, supporting improved quality of care, and enhancing access to services through measures to address transport costs), the actual intervention details changed substantively through the process. In particular, the use of both transport and healthcare vouchers were dropped early in the process as being both financially unsustainable and too logistically complicated to implement. Instead, the team sought to work with local savings groups to help women save funds to cover the costs of their own maternity care, and also instituted a system of recognition awards as a low-cost way to motivate health workers. While the interventions as implemented by the FHS team have not been fully replicated elsewhere, the interventions have been sustained by local actors and aspects of them continue to influence broader government policy.

The three different country teams each adopted very different strategies to scaling up their interventions; however, none of the teams articulated clear scale-up plans in their initial proposals. The Afghanistan team recognised relatively early in the implementation process that it sought to stimulate a change in government policy towards the use of the scorecard, and this insight was important in terms of the implementing team established and in particular the strong links built by the research team with the Department of Community-based Health Care in the Ministry of Health. However, the Afghan Government was not initially convinced by the results of the community scorecard intervention, observing that the initiative had not included Pashtun communities (only ethnic Hazara, Uzbek and Tajik communities) and that before seeking to scale-up, it would be important to implement the community scorecards in contexts that included Pashtuns. It was only after this further piloting, which was implemented with another NGO in Nangarhar province after the formal end of the FHS project in Afghanistan demonstrating similar evidence of improvements in health facility services, that in September 2015, the Ministry of Public Health gave official support to the community scorecard strategy incorporating it as part of the new Community-based Health Strategy.

In Bangladesh, the FHS implementing team quickly recognised (by the end of 2011) that there was an explosion of mHealth technologies with many different organisations experimenting with different mHealth applications. The FHS team sought to better understand the array of different mHealth models being implemented, through conducting a scoping study [24]. Of the many mHealth applications found, only the FHS initiative and one other social enterprise were seeking to connect VDs with formal doctors. However, the team’s awareness that there were many other mHealth initiatives, and that most worked through the private sector, perhaps influenced the team to consider a more spontaneous scaling-up strategy that focused on identifying successful aspects of different models, and allowing the market to adopt such innovations.

While the FHS Uganda team did not articulate a clear scale-up strategy at the start of its work, the broad approach adopted did not hinge upon an organised replication of the scheme. Instead, the FHS Uganda team sought firstly to design an intervention that could be managed and sustained by local communities and stakeholders themselves, and secondly to build relationships with a variety of stakeholders, including the Ministry of Health, parliamentary representatives, NGOs and development partners, so as to encourage them to adopt and roll out successful elements of the FHS approach.

All three research teams conducted a stakeholder analysis at the start of their work, although these were conducted with varying degrees of thoroughness. In Uganda, the stakeholder analysis suggested that the community was very supportive of the intervention, though concerned about feasibility considering local resource constraints [25]. The question of community acceptability was a greater concern in Afghanistan, where the team identified very strong support from the Ministry of Public Health from the outset, but low levels of trust, linked to the current and historical security situation, and particularly strong mistrust within communities towards public authorities. Further, the implementing team was initially concerned about the feasibility of ensuring representation in the community scorecard discussions of the full diversity of community members. However, the final round of focus group discussions conducted in the intervention communities found very high levels of participation and a sense that even the most marginalised community members had been involved in the process, ensuring gender equity and representation in the generation, scoring and management of performance indicators.

All three of the research teams identified a complex array of stakeholders that they needed to engage (Table 5) to support both the successful implementation of the intervention and subsequent scale-up. In practice, quite a lot of the research conducted, and especially ‘add on’ studies developed after the main protocols, focused heavily on understanding the attributes of stakeholders. For example, the Afghanistan study used focus group discussions to explore which community members were involved in the community scorecard processes, and how they perceived the process. In Bangladesh, research addressed how VDs perceived the call centre intervention [17], and later spread into studies of citizens’ use of mobile phones [19], with a view to understanding how the technology could best be used to promote health. In Uganda, the participatory nature of the research meant that the research team had very close engagement with all the stakeholder groups listed in Table 5. For example, throughout the implementation process, the team conducted regular review meetings with an array of local stakeholders involved in the project to review which parts of the intervention were working smoothly and which were facing obstacles. Ultimately, the team identified key findings for all the key stakeholder groups (village health teams, communities, savings groups, health workers, political leaders, health managers and district administrators).

Reviewing the project documentation across the three sites, there is surprisingly little explicit recognition or discussion of contextual factors that may affect scale-up. While teams invested significant effort in actor management, and nurtured relationships with local and national level actors whose support they perceived to be critical to the success and ultimately scale-up of the intervention, there is much less analysis of the context.

There were some partial exceptions to this. First, in Afghanistan, the unfolding security situation meant that the team provided regular assessments and reports on the security risks, but other dimensions of context were not fully studied. In Uganda, community engagement through the participatory action research identified factors important for continuity of interventions, including local political ownership of the intervention, building of local capacity and committed leadership from both the political and technical arms of the districts in which the intervention was implemented.

One explanation for the limited systematic analysis of context may be the fact that, in all three countries, the research teams were local and/or had long histories of engagement in the regions where the interventions were occurring. For example, Makerere’s work in the Eastern region of Uganda dates back to the beginning of FHS1 (around 2006), Johns Hopkins School of Public Health involvement in Afghanistan dates back to 2002, and ICDDR,B has been working in Chakaria promoting self-help for health since 1994 [26]. It is possible that, given these histories of extended engagement in the intervention sites, teams were already intimate with contextual factors and did not feel a need for further in-depth investigation of context.

In both Afghanistan and Uganda, the planned research focused primarily on the questions of feasibility and effectiveness, namely the extent to which the intervention could be carried out in that particular setting and whether or not it was effective. The Bangladesh team sought to examine both effectiveness and acceptability of the intervention.

Many of the findings from the implementation research studies related to the intervention itself, and in particular its acceptability, appropriateness, feasibility and implementation costs. These findings bore significant implications for the design and implementation of the intervention. For example, in Afghanistan, the studies underlined the importance of investing sufficiently in skilled facilitators who could inform communities about entitlements and mitigate unrealistic expectations. In Uganda, the implementation and documentation process pointed to the need for supportive supervision for CHWs and district implementation teams. In Bangladesh, the whole intervention design was brought into question as it became apparent that the intervention, as initially conceived, struggled to achieve acceptability with community members, and did not offer sufficient financial incentives to either the village doctors, or the private for-profit call centre.

In addition to lessons about the design of the intervention, the research also yielded insights into how stakeholders perceived the intervention and how best to frame and implement the intervention to promote its acceptability and perceived appropriateness. For example, in Afghanistan, in conservative communities, it was important to figure out modalities that allowed women’s views to be fed into the community consultative processes, even though, in some settings, the women themselves were unable to participate alongside the men. Further, the research team realised that local stakeholders found the intervention more acceptable when framed as part of a national system of cascading scorecards, rather than as a separate and stand-alone intervention. While all three of the country teams pursued qualitative research, such as focus group discussions, which could help cast light on acceptability and appropriateness of the intervention, many important findings emerged spontaneously from the research team’s close and regular engagement with stakeholders on the ground.

Frequently, the original objectives of the implementation research undertaken by FHS appeared unnecessarily narrow. While research protocols focused on feasibility, acceptability and effectiveness, the close involvement of the research teams in these implementation processes also cast light on factors such as cost, fidelity, sustainability and the perceived appropriateness of the intervention. These insights offered important and actionable lessons for implementing teams.

While there are obvious connections between the conceptual frameworks on scaling up [8, 9] and frameworks on implementation research [15], to the best of our knowledge, this is the first paper that explicitly integrates these two types of framework.

Although the projects presented here are only now reaching completion, and it is therefore too early to fully assess their influence upon scale-up and sustainability, the research conducted offered substantial insights into the barriers and facilitators of scaling up, including the design of interventions, and relevant attributes of stakeholders. Frequently, these insights could be translated into concrete changes in the design or framing of an intervention that might make it more feasible or acceptable. The research process in all three cases involved substantial stakeholder engagement, which, while burdensome on the research team, was critical to both developing lessons and encouraging the adoption of these lessons.

In both Bangladesh and Uganda, the interventions were consciously framed in a relatively flexible way that facilitated adaptation to new learnings about stakeholders, or the intervention design. This flexibility was of critical importance, and all three FHS projects went through substantial adaptations to the initial design, including dropping some elements of the original package of interventions (Bangladesh and Uganda), adding new elements to the package of interventions (Uganda), reframing or re-packaging the intervention (Afghanistan), changes in implementation partners (Bangladesh), and extending the time frame and the geographical spread of the intervention (Afghanistan). Other authors have pointed out that scaling up interventions or innovations involves intervening in complex systems [5, 27], such that one should expect constant adaptation, dynamic interactions and complex social processes. Certainly, the three case studies reflect this complexity, and the implementation research conducted offered a means to embrace the complexity and simultaneously navigate a way through it. It is also important to note the potential tension between adaption and fidelity. While fidelity of an intervention to its original design is often thought to be key to the success of scale-up, this needs to be balanced against adaptations that help ensure the appropriateness and acceptability of the intervention. Particularly in the early stages of intervention design and scale-up, adaptation will likely be more important than fidelity.

There were considerable differences in the implementation research designs adopted across the different country settings. While all three country research teams sought to assess effectiveness, they had differing foci in terms of other implementation questions (feasibility, acceptability, etc.). Further, there were differing degrees of flexibility in the research design, with Uganda perhaps being the most flexible and adopting a participatory action research approach, and Afghanistan, for primarily logistical reasons (namely the shorter period of study), being the least flexible. However, in addition to the formal protocols, all three teams were required by the FHS research consortium and the funders to also develop theories of change [28] and conduct stakeholder analyses early in the research process. The stakeholder analyses proved critical in identifying key stakeholders, and guiding teams’ thinking about how best to engage those stakeholders. Further, the theories of change necessitated an explicit articulation of how the proposed interventions would help achieve the desired objectives, and therefore perhaps facilitated later decisions about which components of an intervention could be dropped or replaced. Overall, these pragmatic project management tools combined effectively with the implementation research to enable teams to adapt implementation strategies to reflect learnings from their research, while maintaining an overall sense of purpose and strategy.

In Uganda, the Makerere research team also engaged in regular review and reflection meetings. To give one example, during the first year, participatory self-assessments were conducted where both the research team and the district health management teams involved in the intervention reflected on what they had learnt, the nature of their relationships, where there were gaps and how these gaps could be addressed. This process helped to identify where collaboration between the research team and the district health management teams could be improved, and where responsibilities needed to be handed over from the research team to district health management teams to strengthen effectiveness and sustainability. Such formal processes for reflection likely helped to reap the full benefits of implementation research.

The three cases reviewed here had varied approaches to scaling up, including a more vertically oriented approach (Afghanistan), a more horizontal approach (Uganda) and a replication approach (Bangladesh). In Afghanistan, the research focussed primarily on the feasibility, acceptability and appropriateness of the intervention. Presumably in this context, where government already used balanced scorecards at higher levels in the system, determination of feasibility and desirability may be sufficient to support scale-up. In contrast, the research in Bangladesh, where it was anticipated that private market-based actors may be key in scaling-up the intervention, focussed more on acceptability and adoption, i.e. attributes of the target community. In a sense, the Bangladesh study provided ‘market research’ for private sector entities interested in entering the market. However, these conclusions are tentative, and much more needs to be done to understand how best to tailor implementation research to different types of scale-up strategy.

As previously noted, this paper is written at the time of the close out of the FHS projects, and thus it is too early to fully appreciate lessons concerning scale-up and sustainability. In addition, all members of the writing team are internal to the FHS project, and indeed the lead author is also the project leader of FHS. This has certain advantages in that, as participants in the processes described here, the authors brought extensive knowledge of the contexts, interventions and implementation research. However, it may also lead to certain biases, such as an inclination to make the interventions supported appear successful. We sought to guard against this bias as far as possible by triangulating between different data sources, including the researchers' own recollections, but also historical project documentation to describe the initial ambitions of the projects and how they changed over time.

Many implications emerge from this paper regarding how best to design and undertake implementation research that is designed to support scale-up.

With regard to implementation research, a number of apparent good practices have emerged, many of which reinforce what has already been described in the published literature. For example, the need for engaging stakeholders from the start of the process and providing ongoing nurturing of these relationships is highlighted across multiple implementation research texts [29, 30], and was again found to be important here. Further, this paper also supports the need for flexibility in both the intervention being implemented (to respond to lessons from research findings) and in the research design (to respond to emerging new problems) [31].

While it is important that teams involved in implementation research are not overly ambitious in their objectives, teams should think more broadly about the kind of questions that their research can address, as well as being receptive to unexpected learnings.

Finally, while scaling up frameworks typically consider the characteristics of the target community and those of the implementing team separately, FHS research suggests that this is an oversimplification. There were multiple stakeholders involved in each of the country cases, who had differing degrees of responsibility for and engagement with the intervention. In no case was the community a passive recipient of the intervention. We also found substantial blurring between the research team and the ‘implementing team’. In all three of the cases considered here, the research team worked closely with stakeholders responsible for implementation, and had a direct implementation role themselves. In Uganda, where participatory action research was employed, community members were in effect part of the implementing team.

There is a growing interest in the notion of ‘embedded research’ [32, 33], whereby researchers are embedded within the organisations that they are examining. While only one of the three studies described here formally espoused a participatory action research approach, the close relationships, and indeed blurring of boundaries, across different types of stakeholder roles was of critical importance to the success of scaling up strategies. However, this relational complexity and the overlapping roles across research team, implementation team, and target community raise many questions around the usual research paradigm that emphasises the importance of research neutrality, and sees an intervention as largely separate from the community and context in which it is implemented. If the application of implementation research to support scale-up of interventions depends critically on intense stakeholder engagement, then more thought is needed about how best to manage these relationships through the scale-up process.