Elsevier

The Lancet

Volume 377, Issue 9759, 1–7 January 2011, Pages 85-95
The Lancet

Health Policy
Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations

https://doi.org/10.1016/S0140-6736(10)60810-0Get rights and content

Summary

Evaluation of large-scale programmes and initiatives aimed at improvement of health in countries of low and middle income needs a new approach. Traditional designs, which compare areas with and without a given programme, are no longer relevant at a time when many programmes are being scaled up in virtually every district in the world. We propose an evolution in evaluation design, a national platform approach that: uses the district as the unit of design and analysis; is based on continuous monitoring of different levels of indicators; gathers additional data before, during, and after the period to be assessed by multiple methods; uses several analytical techniques to deal with various data gaps and biases; and includes interim and summative evaluation analyses. This new approach will promote country ownership, transparency, and donor coordination while providing a rigorous comparison of the cost-effectiveness of different scale-up approaches.

Introduction

The Millennium Development Goals (MDGs) have stimulated interest and increased funding for programmes aimed at reduction of maternal and child mortality and the burden of HIV/AIDS, tuberculosis, and malaria. At the same time, the realisation that few programmes and initiatives have been evaluated properly,1, 2, 3 and interest in results-based financing approaches,4 are increasing pressure on implementers to undertake effectiveness evaluations.

A common evaluation framework is needed to allow future comparison of the performance of different initiatives—measured in terms of increasing coverage and achieving health effects—and their cost.5 Such a framework should include: (1) a conceptual model outlining pathways through which the initiative is expected to affect the MDGs; (2) a list of standard indicators of inputs, processes, outputs, outcomes, and impact, with clear measurement plans; and (3) guidelines for design of evaluations in a compatible way. Much progress has already been made on the first two objectives.6 This Health Policy article focuses on the third topic, reporting on evaluations of large-scale public-health programmes.

Our objective is not to establish the efficacy of new biological or behavioural interventions, or of packages of such interventions; these aims are best achieved with randomised controlled trials. We are interested in assessment of how well large-scale complex programmes deliver efficacious interventions, using different delivery channels in various health-system contexts.

Programme success is defined as gains in intervention coverage and in health effects under real-world conditions, when implementation tends to be less intense and more variable than in efficacy trials.7 Observational designs are needed because evaluators cannot control where, when, and how rapidly programmes will be implemented at scale by governments, international or bilateral agencies, and private voluntary organisations.

Design of health-programme evaluations has been dominated traditionally by experimental approaches used in medicine, in which specific individuals or clusters of people receive an intervention whereas others do not. Studies tend to be undertaken in controlled environments in which the influence of external factors is kept to a minimum or eliminated. In the real world, however, the intervention or programme of interest usually accounts for only a small part of variability in health outcomes. Figure 1 presents a simplified framework showing that maternal and child health outcomes can also be affected by socioeconomic and contextual factors, by changes in existing health services in the public and private sectors that are outside the scope of the programme of interest, and by other initiatives or interventions in health or other sectors present in the same geographic areas. Because changes in all the above factors can be happening concurrently with implementation of the programme under assessment, real-world effectiveness evaluations present challenges that cannot be properly addressed by the traditional approach of intervention versus comparison group. In addition to the reality that programmes are not scaled up in a vacuum, they also rarely start from a blank sheet. Pre-existing baseline levels and, particularly, trends in key indicators need to be taken into account.

We start by describing typical approaches to evaluation of large-scale programmes and move on to propose a new approach that addresses the mosaic of concurrent programmes and initiatives characteristic of most low-income countries with high rates of maternal and child mortality. We draw heavily on an evaluation study published in The Lancet,8 the Multi-Country Evaluation of Integrated Management of Childhood Illness (MCE-IMCI),9, 10 and subsequent efforts to develop designs for independent assessments of the Catalytic Initiative to Save a Million Lives,11 including the three-country rapid scale up of maternal, newborn, and child health funded by the Bill & Melinda Gates Foundation via WHO (the Partnership for Maternal, Newborn, and Child Health).

Section snippets

Programme areas only

This evaluation design is one of the simplest and entails recording changes over time in the intervention area, in terms of indicators of health service provision, utilisation, coverage, and impact. These evaluations usually attempt to note goals in terms of coverage (eg, 80% of children aged 6–59 months receiving two doses of vitamin A in the previous 12 months) or impact (eg, a 25% reduction in mortality over a specific period), and have been described as adequacy designs, because evaluators

A way forward: the evaluation platform design

In the preceding section, we addressed limitations of intervention-comparison designs in contexts for which many programmes and interventions are being scaled up simultaneously. We argued that a reductionist approach to evaluation based on isolation of programme effects is no longer appropriate for scaling up of initiatives to reach the MDGs in most low-income countries.

What is the alternative? How can evaluations of large-scale programmes be designed in ways that are scientifically rigorous

Conclusions

We propose a systematic approach for evaluation of scale-up of national programmes for maternal and child survival and potentially other public-health programmes that address specific diseases (such as HIV/AIDS, tuberculosis, or malaria). A national evaluation platform shows how ideas have evolved in response to changes in the development landscape in most countries of low and middle income, where governments are working with several partners to implement many simultaneous public-health

References (38)

  • JP Habicht et al.

    Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact

    Int J Epidemiol

    (1999)
  • J Bryce et al.

    Ten methodological lessons from the Multi-Country Evaluation of Integrated Management of Childhood Illness

    Health Policy Plan

    (2005)
  • The catalytic initiative to save a million lives

  • CA Brown et al.

    The stepped wedge trial design: a systematic review

    BMC Med Res Methodol

    (2006)
  • T McKeown

    The modern rise of population

    (1976)
  • S Gregson et al.

    Impact and process evaluation of integrated community and clinic-based HIV-1 control: a cluster-randomised trial in eastern Zimbabwe

    PLoS Med

    (2007)
  • NC Grassly et al.

    The effectiveness of HIV prevention and the epidemiological context

    Bull World Health Organ

    (2001)
  • J Amaral et al.

    Impact of IMCI health worker training on routinely collected child health indicators in northeast Brazil

    Health Policy Plan

    (2005)
  • CG Victora et al.

    Are health interventions implemented where they are most needed? District uptake of the integrated management of childhood illness strategy in Brazil, Peru and the United Republic of Tanzania

    Bull World Health Organ

    (2006)
  • Cited by (0)

    View full text