Process evaluation in the field: global learnings from seven implementation research hypertension projects in low-and middle-income countries

Preliminary data on how the projects planned their process evaluations, were collected from the 15 hypertension research teams in the network in 2015 and was used to develop a data collection tool. This tool was then used to collate case study information from seven projects based on the following themes: (1) description of the intervention, (2) objectives of the process evaluation, (3) approach (including theoretical basis, main sources of data and analysis methods), (4) main findings of the study and the process evaluation, (5) implications for the project, policy and research practice and (6) lessons for future process evaluations. The case study approach recognizes that projects were at different stages of intervention/ evaluation. Each process evaluation was nested within an intervention study that was either completed or nearly completed.

Information relevant to each of the agreed themes was documented by FL and JW using a data extraction sheet (see Table 1). The information was summarized and reported descriptively and narratively in relation to the themes above. Overarching issues were identified by the working group that had been established to oversee the project. The working group comprised researchers who had all been involved in the different process evaluations and helped to draw out the main implications of the process evaluation with respect to project and policy, as well as lessons to inform future process evaluations.

The seven process evaluations were from low- and middle-income countries (LMICs) that had either completed or nearly completed their process evaluations including Fiji and Samoa, South Africa, Kenya, Peru, India, Sri Lanka, Tanzania and indigenous communities in Canada, (Table 1). The countries referred to in this manuscript were countries in which projects were a) funded through the GACD and b) contained process evaluations at a sufficiently advanced stage to include in the analysis. These process evaluations were part of pragmatic trials of innovative interventions to prevent and manage hypertension in the areas of salt reduction, task redistribution, mHealth, community engagement and blood pressure control [8]. Although the studies generally took place in LMICs, there were varying geographical, cultural and economic settings within and across countries (Table 1). Most of the interventions (five) were tested in randomized controlled trials with one stepped wedged trial and one pre-post study design. The duration of the studies was three to 5 years (Additional file 1).

The specific objectives for each process evaluation were tailored to the broader objectives for each project and however, they were generally aimed at understanding factors that would have affected the implementation process and the impact of this process on trial outcomes. These objectives were achieved through the collection of qualitative information about context, mechanisms, feasibility, acceptability and sustainability of the interventions and quantitative information about fidelity (extent to which the intervention is implemented as intended), dose (how many units of each intervention are delivered) and reach (extent of participation of the target population) [2].Two process evaluations (in Peru and Sri Lanka) were limited to assessing barriers and facilitators which affected implementation, and in Peru the project had benefited from its previous formative research [9]. In India, the process evaluation had only evaluated the ASHA training and not the entire intervention. For the DREAM GLOBAL in Tanzania and Canada, the investigators have so far undertaken and reported on formative research as part of its process evaluation and have published their process evaluation framework protocol [7]. This formative research is aimed at assessing the major components of the intervention and how these components should vary among and between people, countries and cultures.

The theoretical approaches to process evaluation differed in the seven case studies, including use of the United Kingdom’s Medical Research Council (MRC) framework for process evaluation (Fiji/Samoa) [2, 10], the realist evaluation approach (South Africa and Kenya) [11], community-based participatory evaluation theory (Tanzania and Canada) [7], the RE-AIM framework (Kenya) [12], Kirkpatricks’ four level evaluation model (India) [13], phenomenology (Peru) [14] and Framework analysis method (Sri Lanka) [15] (see Table 2).

Most of the investigators used a mixed methods approach in their data collection and inquiry, integrating quantitative and qualitative data within a single study [18]. For all seven case studies, qualitative data were obtained using one or more of the following methods: interviews, observations, focus group discussions, implementer diaries, researcher notes and reports of meetings. Quantitative data were obtained from surveys, metrics of process measures, written tests and clinical examinations. In most of these cases, data were collected and analyzed by researchers who were members of the study team but were adept at analyzing these types of data and so were specifically employed to conduct process evaluation. In two studies, phenomenological and framework analysis approaches were applied with only qualitative data used in their evaluation. This was because their process evaluation solely focused on assessing barriers and facilitators.

Although the studies reported in this paper had very distinct interventions and outcomes, there were similarities in the findings of the process evaluations across the studies. The process evaluations of the salt reduction interventions in Fiji and Samoa demonstrated that the absence of a significant reduction in salt, in either country, could be explained by the fact that the interventions were not implemented with full fidelity (i.e. implemented in line with the study protocol). This was partly due to contextual factors including political and management changes, a cyclone in Fiji (affecting normal diets) – and partly due to lack of time for the intervention to take effect. However, the process evaluation also highlighted the fact that the projects had resulted in increased research capacity among government and research institutions that participated in the study. In addition, new government policies for salt were being integrated into the new Food and Nutrition security strategy in Fiji, while in Samoa government proposals for taxation of packaged foods high in sugar and salt, were being considered.

In South Africa, although control of hypertension was not improved, the lay health worker (LHW) intervention enabled the functioning of clinics to be streamlined, partly by improving the appointment system for chronic patients. The process evaluation enabled investigators to conclude that shifting certain medically and socially oriented tasks from nurses to LHWs could relieve the burden on nurses, improve delivery of chronic care, and improve functioning of primary care clinics. However, there was non-linearity between implementation process and outcomes at clinic level. The variability in implementation and outcomes between sites were likely a consequence of different levels of patient load and resources, nature of relationships and clinic management [19, 20]. For instance, in clinics with high patient loads LHWs were unable to complete all their tasks.

The process evaluation in Tanzania and Canada contributed to the formative stage of the intervention by identifying discrepancies between text messages created by researchers and those preferred by recipients, thereby enabling a change in the study design prior to commencement.

In Kenya, although analysis of the main trial outcomes had not been completed at the time of developing this paper, process evaluation data showed low fidelity in implementing certain components of the intervention, sub-optimal retention of skills among community health workers (CHWs), and gaps in recall of training elements by CHWs in some topics. Initial training of CHWs was more effective to help CHWs recognize complications, non-pharmacologic treatments and causes of hypertension, than to recognize the signs and symptoms of hypertension and the possible side effects of medication. In response to these findings, the project subsequently incorporated training that encompassed these other issues.

The salt substitute intervention in Peru was effective in reducing population levels of uncontrolled BP. The process evaluation enabled the investigators to determine that effective implementation of the intervention was attributable to (1) good relations and trust between researchers and the community facilitating the launching of the trial in the area, and thus take advantage of this established engagement platform aided the intervention’s uptake through trust; and (2) targeting of women during the intervention as the critical primary receptors of the intervention due to their role as food preparers in homes.

The process evaluation in India is still underway, but the process evaluation of the training of Accredited Social Health Activists (ASHAs) demonstrated the intervention was successfully implemented by the ASHA which improved skills, knowledge and motivation among the ASHAs [21, 22].

In Sri Lanka, patients and providers liked the triple pill because of its ease of use (single pill, once a day dosing) and significant BP control. At the beginning of the trial, providers expressed apprehension about initiating treatment with the triple pill in treatment naïve hypertensive patients. Over time, they became comfortable as no major safety issues were reported and the extent of BP lowering achieved was substantial. Providers expressed a willingness to prescribe the triple pill and patients were willing to use it if it was made available after the trial.

These findings from the seven cases demonstrate the role of process evaluations in describing the implementation process despite variation in study outcomes.

A series of implications for projects and policies were highlighted by the process evaluations. For salt reduction interventions, these included the need for adequate time between baseline and follow up for the implementation to take effect; the need for strong leadership (diverse, experienced and representative) and clear roles for multi-sector advisory bodies; regular communication with stakeholders; and consideration of consumer acceptability and affordability of salt-reduced products. For the LHW intervention, the importance of a supportive and well-resourced clinic environment, strong management of the Primary Health Care facilities, and motivated staff that relate well to the patients were identified as fundamental for successful task shifting operations. The importance of continued training, communication and programmatic support was highlighted through a number of process evaluations. The process evaluation in the CHW study in Kenya showed the need for additional education about signs and symptoms of hypertension and treatment side effects as well as intensive, repeated training regarding hypertension management. Similarly, findings from the ASHA program in India emphasized the need for culturally appropriate training materials, delivered using interactive and innovative methods. It also showed the need to align project tasks and responsibilities with CHW incentives as without this, their morale for work would decline. Implementing a low dose, combination treatment strategy in Sri Lanka highlighted the need for education and training for prescribing physicians around the benefits of early use of combination therapy for treatment of hypertension and ensuring availability of the combined therapy for hypertension.

A range of lessons for process evaluation as part of implementation research in LMIC have been identified through this study (Fig. 1). A common theme that emerged was that while mixed methods approaches can be time consuming and generate a vast amount of data, they significantly enhance understanding of the implementation of complex interventions as well as generate a wealth of learning to inform future projects. For instance, the semi-structured interviews used in the Fiji and Samoa process evaluation, produced qualitative data, i.e. experiences of government institutions, the food industry and other stakeholders. This information helped to explain that the lack of intervention effect on salt intake was likely at least partially attributable to the short intervention duration and the fact that policy changes had yet to take effect.

Incorporating process evaluation data collection tools into the intervention process from the onset was identified as crucial for process evaluation. Process evaluations cannot be conducted retrospectively as investigators cannot go back and collect the required data. For instance, for the clinic based LHW intervention in South Africa, it was ideal to observe and understand how nurses interacted with the intervention as it was being implemented. Thus, process evaluations should be fully embedded into the intervention protocol or a separate process evaluation protocol should be developed alongside the intervention protocol. In Sri Lanka, collecting process evaluation data before the study outcome data were available helped in exploring implementation processes without unintentionally influencing investigator or patient behavior in the study.

Some investigators who commenced their process evaluation after the intervention had begun, e.g. the CHW project in Kenya, felt that it may have been helpful to have started this earlier in the study life-cycle. Other investigators who incorporated process evaluation in formative research and situation analyses reported that this approach helped identify which specific process measures should be collected during the intervention. One study team deliberately did not collect process evaluation data until after the study was complete so as to not affect investigator or patient behavior in the study itself. Therefore multiple considerations should be taken into account when designing process evaluations.

Experiences from six of the projects support the MRC recommendation for strong relationship and consultations between researchers responsible for the design and implementation of the trial, outcome evaluation and process evaluation [2]. However, whilst the teams reported positively on coming together to exchange experiences at different stages of the project, it was felt that additional interim assessments of process throughout the project would have further strengthened implementation of the interventions. In some teams, discussion of the preliminary results of the process evaluation by the broader project group, including local country teams, was an essential part of the data synthesis and greatly enhanced the validity of the results by clarifying areas in which the researchers might not have understood the data correctly.

The DREAM GOBAL process evaluation demonstrated the need for formative research that informed the mHealth projects for rural communities in Tanzania and Indigenous people in Canada as well as the value of using a participatory research tool [28]. This tool helped to identify: a) key domains required for ongoing dialogue between the community and the research team and b) existing strengths and areas requiring further development for effective implementation. Applying this approach, it was found that key factors of this project, such as technology and task shifting required study at the patient, provider, community, organization, and health systems/setting level for effective implementation [7].