Figures
Abstract
Introduction
In 2010, the Global Fund provided more than 75% of external international financing for malaria control. The Global Fund uses performance based funding in the grants it finances. This paper analyses the indicators used to measure the performance of Global Fund supported malaria grants in Asia.
Methods
Indicators used in the performance frameworks for all Global Fund supported malaria grants in Asia were retrieved from grant database and grouped into impact, outcome, output and input categories and categorized by service delivery areas. Indicators of each group were compared over rounds. Indicators used in performance frameworks were compared with internationally adopted indicators included in the Monitoring and Evaluation Toolkit developed by the Global Fund and international technical agencies.
Results
Between 2002 and 2010, 1,434 indicators were included in the performance frameworks of the 48 malaria grants awarded in Asia, including 229 impact and 227 outcome indicators, 437 output and 541 input indicators, with an average of 29.9 indicators per grant. The proportion of impact and outcome indicators increased over rounds, with that of input indicators declining from 44.1% in Round 1 to 22.7% in Round 9.
Conclusions
Input indicators, which have predominated the performance frameworks of the Global Fund supported malaria programs in Asia have declined between Rounds 1 and 9. However, increased alignment with internationally adopted indicators included in the Monitoring and Evaluation Toolkit is needed to improve the validity of reported results.
Citation: Zhao J, Lama M, Sarkar S, Atun R (2011) Indicators Measuring the Performance of Malaria Programs Supported by the Global Fund in Asia, Progress and the Way Forward. PLoS ONE 6(12): e28932. https://doi.org/10.1371/journal.pone.0028932
Editor: Clive Shiff, Johns Hopkins University, United States of America
Received: July 25, 2011; Accepted: November 17, 2011; Published: December 19, 2011
Copyright: © 2011 Zhao et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: These authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
International financing for malaria control increased by 166%, from US Dollars ($) 0·73 billion in 2007 to $1·94 billion by 2009 [1]. The Global Fund to fight AIDS, Tuberculosis and Malaria (the Global Fund) accounted for 75% of international financing [2], with $5.3 billion committed for malaria programs in 83 countries, including $0.95 billion in Asia covering 32 countries in East Asia and the Pacific and South and West Asia [Box S1].
Malaria epidemiology is highly heterogeneous in Asia [3]: endemic in the south and west Asian and the Pacific countries, highly focal in the countries and areas of the Greater Mekong sub-region, such as Cambodia, Yunnan province of China, the Lao People's Democratic Republic and Viet Nam, restricted to particular geographical locations in Malaysia, the Philippines and the Republic of Korea, and no indigenous transmission in the Maldives since 1984. Most countries have both Plasmodium (P.) falciparum and P. vivax. Transmission in Afghanistan, North and South Korea, Sri Lanka and central areas of China is primarily due to P. vivax [3].
The Global Fund uses performance based funding when investing in AIDS, tuberculosis, malaria and health systems strengthening. Grants are implemented in two phases: the phase one for two years and the phase two for three years. Performance in the phase one determines funding for the phase two. Grant performance is measured quarterly or six monthly, when disbursements are made adjusted by performance. The principal recipient (PR) and the Global Fund jointly develop a performance framework [4]: a legally binding agreement signed by each, to monitor grant performance. The performance framework comprises indicators, targets, data sources and reporting requirements reflecting goals and objectives of the grant, local epidemiology, and strength of the local reporting systems. The PR and the Global Fund can agree to revise the performance framework following the first phase of program implementation reflecting performance, evolving epidemiology and contextual factors.
A performance framework typically includes 2 to 5 impact and outcome indicators to measure achievement of program goals and objectives, and up to 15 ‘programmatic’ input and output indicators to measure progress with major activities, which are grouped as service delivery areas (SDAs). Different categories of indicators are shown in Table 1. Each indicator has a time-bound target, with targets for impact and outcome indicators set for 1 or 2-year periods to assess performance at end of the phase one [5], and programmatic indicators set for 3 to 6 months and used to assess implementation performance for the period in question and to determine disbursement for the next period. To guide PRs in indicator selection and to ensure consistency of indicator wording and comparability of results across grants the Global Fund developed a Monitoring and Evaluation Toolkit (M&E Toolkit) with its technical and financing partners in 2004, with revisions in 2006, 2009 and 2011 emphasizing internationally adopted indicators used to measure outcome and impact [6].
Grant performance is rated according to achievements towards targets set in the performance framework. Performance rating informs disbursement decisions and funding awarded for the second phase of the grant [7], [8]. We analyze malaria programs supported by the Global Fund in in Asia, to explore for the first time the indicators used in performance frameworks to measure grant performance and the alignment of these indicators with internationally adopted indicators defined in the M&E Toolkit.
Methods
Data sources
We used the Global Fund grants database to identify and tabulate indicators included in performance frameworks (both Phase one and two) of all Round 1–9 malaria grants supported by the Global Fund in Asia over the period 2002 and 2010, and financing for these grants.
Data analysis
We grouped indicators into impact, outcome, output and input categories, then sub-grouped all but impact indicators according to SDAs (see Table S1 for a list of SDAs). We computed the number and proportion of indicators in each category and SDA, and the cumulative funding allocated to each SDA, comparing them over Rounds 1–9.The analysis is limited to the indicators included in the performance frameworks, does not include the actual reported results against the targets for the indicator.
The indicators included in the performance frameworks were assessed against the definition and wording of relevant indicators in the M&E Toolkit 2009 version and categorized as ‘aligned’, ‘partially aligned’ and ‘not aligned’ – aligned if an indicator in the performance framework matched that in the M&E Toolkit, partially aligned if key elements of an indicator were expressed using different wording to that in the Toolkit, and non aligned if the indicator used in the performance framework was totally different or not included in the M&E Toolkit. Alignment of indicators with M&E Toolkit was compared over rounds by SDAs and by indicator category using SPSS 13.0 (SPSS Inc., Chicago, IL). Examples of indicators with relevant categorization are shown in Box S2.
Results
Between 2002 and 2010, from Round 1 to Round 9, the Global Fund approved in Asia 48 malaria grants with a budget of $950 million. There were 1,434 indicators included in the performance frameworks of these 48 grants: comprising 229 impact and 227 outcome indicators, 437 output and 541 input indicators (Table S1).
While the number and proportion of indicators used in the SDAs relating to prevention, treatment and health system strengthening (HSS) this difference had no clear trend (P = 0.007). The top five SDAs accounted for 80% of the indicators: 23.6% (284/1205) for activities relating to insecticide treated nets (ITNs), 17.3% for facility and home treatment, 16.3% for training, 11.7% for diagnosis, 10.9% for behavioral communication change. Only two of the 1,205 indicators were included in the performance frameworks for activities related to indoor residual spraying, an intervention critically important in low malaria transmission areas.
The number of impact and outcome indicators increased over Rounds 1 to 9 (P = 0.007), with a rise in the ratio of these over output and input indicators (P = 0.002) (Figure 1). However, while the input indicators declined in proportion to the total number of indicators, by Round 9 these still accounted for 38% (range of 23% to 44% across rounds) of total.
The average number of indicators per grant, which remained constant over rounds, was 29.9 (range of 19.3 to 44.6): 4.8 (range 2.8 to 7.8) for impact, 4.7 (range 2.5 to 9.3) for outcome, 9.1 (range of 6.6 to 14.0) for output and 11.3 (range 5.0 to 17.1) for input indicators. (Figure 2)
Input indicators accounted for 37.7% over the rounds, decreasing in t proportion from 44.1% of total in Round 1 to 22.7% in Round 9 (Figure 2, P = 0.000).
Indicators related to training accounted for 16.3% of the total (196/1,205), declining from 26.4% in Round 1 to 9.3% in Round 5 (Table S1) increasing thereafter, but without an obvious trend.
Of the 1,434 indicators used, 43.2% was aligned, 28.0% partially aligned and 28.8% not aligned with the indicators used in the M&E Toolkit. The proportion of aligned indicators increased over the rounds, with a decline in the proportions of partially aligned and not aligned indicators (P = 0.000). While the indicators for prevention (P = 0.025) and treatment (P = 0.018) SDAs were increasingly aligned over Rounds 1–9, those for HSS did not change (P = 0.380) with 41.5% of these indictors not aligned with the M&E Toolkit.
Over the rounds, the number of indicators relating to ITNs (P = 0.024) and facility and home treatment (P = 0.003) were increasingly aligned, unlike those relating to coordination and supportive environment where 94% of indicators remained ‘not aligned (P = 0.001) (Table 2).
Overall alignment of input (P = 0.373) and output indicators (P = 0.108) did not improve over the rounds, with only 31.1% and 41.6% respectively aligned with M&E Toolkit. In contrast the alignment of impact (P = 0.000) and outcome (P = 0.052) indicators improved over time (Table 3).
For several SDAs, there was a clear asymmetry between the funding allocated and the number of indicators used to assess the performance (Table 4). For example, while 0.3% of the total budget was allocated to training over the period 2002–2010, the number of indicators for training accounted for 16.3% of the total (196/1,434). For diagnosis SDA, which accounted for 3.9% of the total budget the number of indicators were 12.7% of the total, similar to treatment SDA, which accounted for 17.2% of the total budget ($163 million) with 30.1% of the total indicators.
Discussion
We present the first analysis of indicators used in performance frameworks of Global Fund financed malaria programs and their alignment with internationally adopted indicators. Achievements against the targets set using these indicators determine the amount of funding received by grants. The indicators used in performance frameworks and the targets for these indicators form the basis for performance-based funding, and financing of grants. Hence, appropriate selection and definition of indicators is critical to ensure performance is appropriately measured and well performing grants duly rewarded. Inappropriate indicators may distort the performance rating and therefore grant funding.
Since 2009, renewed emphasis by the Global Fund on measurement of outcomes and impact of its investments has strengthened performance measurement of malaria grants in Asia, with increased use of impact and outcome indicators in grant performance frameworks. However, a large proportion of indicators in grant performance frameworks are still input indicators, especially those for training activities, reflecting poor attention to indicator selection in the period preceding 2009, giving undue emphasis in performance of SDAs driven by inputs rather than outcomes and thereby skewing financing towards grants that have achieved improved inputs but not necessarily outcomes or impact [8]. Hence, when negotiating performance frameworks with countries, the Global Fund will continue to reduce the number of indicators per grant and focus them on output, outcomes and impact. Additional analysis with actual program results is needed to quantify how the composition of indicators within the performance framework affects grant performance rating.
Although in Asia the malaria epidemic is heterogeneous, in the majority of Asian countries malaria remains localized. While massive ITN distribution would be an effective strategy in high malaria transmission areas, IRS would be especially effective in low malaria transmission areas [9], [10]. However, in Asia in the malaria grants supported by the Global Fund indicators for ITN are the most frequently monitored indicators, rather than those for IRS at odds with the prevailing epidemiology [11].
Alignment of indicators with the Global Fund M&E Toolkit increased in each successive round, especially with impact and outcome indicators. Consistency of indicator definition and wording across different grants is critical for the data aggregation at regional or global level and comparison across the regions or different epidemic situations. Malaria indicators in the current M&E Toolkit are more relevant to high transmission areas such as Africa where large majority of Global Fund investments are made, with fewer indicators relevant to low transmission areas, including those at pre-elimination and elimination stages which would be more relevant for Asia. The revised M&E Toolkit due for release in 2011 will expand indicators suited to low transmission countries and those in pre-elimination stage, which will provide further flexibility for PRs to select indicators relevant to the epidemic stage in the country.
Different descriptions of the same indicator result in duplication and create difficulty in comparing results. Standardization of each indicator in the new M&E Toolkit will improve the validity of performance framework as well as the consistency and comparability of results across rounds and regions/countries.
In conclusion, an improvement has been observed in the Global Fund performance frameworks for malaria grants in Asia from round 1 to round 9, as evident by decreased proportion of input indicators and increased proportion of outcome and impact indicators. Efforts shall still be made to select indicators, appropriate in the total number per grant and allocation of categories, to ensure the performance framework is measuring in a standardized way what it supposes to measure, and therefore improve the value for money of the Global Fund investments in malaria programs in Asia and in the world.
Supporting Information
Table S1.
Indicators in performance framework of malaria grants in Asia from Round 1 through Round 9, by service delivery areas (SDA) over the rounds.
https://doi.org/10.1371/journal.pone.0028932.s001
(DOC)
Box S1.
List of countries in the Asia region.
https://doi.org/10.1371/journal.pone.0028932.s002
(DOC)
Box S2.
Alignment of different indicators in performance framework with M&E Toolkit.
https://doi.org/10.1371/journal.pone.0028932.s003
(DOC)
Author Contributions
Conceived and designed the experiments: JZ SS RA. Analyzed the data: JZ ML. Wrote the paper: JZ RA ML. Reviewed and approved the final manuscript: JZ ML SS RA.
References
- 1. Snow RW, Okiro EA, Gething PW, Atun R, Hay SI (2010) Equity and adequacy of international donor assistance for global malaria control: an analysis of populations at risk and external funding commitments. Lancet 376: 1409–1416.
- 2.
Roll Back Malaria Partnership (2008) The Global Malaria Action Plan for a free malaria world. 2008, Roll Back Malaria Partnership, Geneva, Switzerland.
- 3.
World Health Organization (2010) World Malaria Report 2010. WHO, Geneva, Switzerland.
- 4.
The Global Fund. Performance Frameworks, Available: http://www.theglobalfund.org/en/me/documents/performanceframeworks/(accessed on October 10 2011).
- 5.
The Global Fund, Monitoring and Evaluation Toolkit, third edition. 2009, The Global Fund, Geneva, Switzerland.
- 6.
The Global Fund, Monitoring & Evaluation Toolkit, Available: http://www.theglobalfund.org/en/me/documents/toolkit/ (Accessed on October 10 2011).
- 7. Low-Beer D, Afkhami H, Komatsu R, Banati P, Sempala M, et al. (2007) Making performance-based funding work for health. PLoS Med 4: e219.
- 8.
Grant Performance Rating and Disbursement Decision, in The Global Fund Operations Policy Manual issued on August 8 2011, Pages 177 to 184. Available: http://www.theglobalfund.org/en/library/documents/ (Accessed on October 10, 2011).
- 9. Worrall E, Connor SJ, Thomson MC (2007) A model to simulate the impact of timing, coverage and transmission intensity on the effectiveness of indoor residual spraying (IRS) for malaria control. Trop Med Int Health 12: 75–88.
- 10. Yukich JO, Lengeler C, Tediosi F, Brown N, Mulligan JA, et al. (2008) Costs and consequences of large-scale vector control for malaria. Malaria Journal 7: 258.
- 11.
The Global Fund, Grant Portfolio. Available: http://portfolio.theglobalfund.org/en/Home/Index (Access on October 10, 2011).