Background
Methods
The RACD M&E tool
Module 1: reviewing key documents | Objective: to review the key documents and personnel involved in the RACD process and determine the availability and use of reporting forms at the health facility level, including: |
Organizational diagrams | |
Standard operating procedures (SOPs) | |
Malaria case notification, case investigation and RACD reporting forms | |
Activity and reporting flow diagrams | |
Module 2: assessing key malaria indicators | Objective: to compare and evaluate the accuracy of malaria case reporting, case investigations and RACD activities using indicators of: |
Completeness and timeliness of health facility reporting | |
Case investigation completeness and timeliness | |
RACD completeness, timeliness, screeninga coverage and positives identified | |
Module 3: evaluating standard operating procedures (SOPs) | Objective: to evaluate the baseline knowledge and practices of programme staff on SOPs and understand existing gaps and challenges in conducting case investigation and RACD activities, including: |
Minimum screening radius around an index case | |
Household and community individuals to target and screen | |
Practices for screening during follow-up visits | |
Challenges to conducting case investigations and RACD | |
Module 4: estimating the costs | Objective: to estimate the costs of conducting case investigation and RACD activities at district and provincial levels, including the main cost drivers for: |
Malaria surveillance personnel (paid and volunteer) | |
Commodities for malaria activities | |
Services and other costs |
Tool modules
Module 1. Reviewing key documents
Module 2. Assessing key malaria indicators
Module 3. Evaluating standard operating procedures
Module 4. Estimating the costs
Pilot phase
Study areas
Location | Pilot implementation period | Data collection period | Phase | Pilot scale | Health facility catchment area population | Total number of facilities | Number of staff interviewed |
---|---|---|---|---|---|---|---|
Aceh, Indonesia | June–September 2013 | June–September 2013 | Elimination | Five subdistrictsa
| 1,343,849 | 34/34 | 34 |
Jiangsu, China | June–August 2013 | January–December 2012 | Prevention of reintroduction | Three countiesb
| 10,149,000 | 6/6 | 10 |
Ranong, Thailand | January–March 2015 | January–December 2014 | Elimination | Five districtsc
| 177,089 | 10/10 | 15 |
Training for the pilot evaluation
Study procedures
Assessment of RACD M&E tool
Results
Tool modules
Module 1. Reviewing key documents
Inventory list for each health facility | Aceh | Jiangsu | Ranong | Total | ||||
---|---|---|---|---|---|---|---|---|
(n = 34) | (n = 6) | (n = 5) | (n = 45) | |||||
Available | % | Available | % | Available | % | Available | % | |
Is there a diagram of the malaria personnel organizational structure? | 30 | 88.2 | 6 | 100.0 | 5 | 100.0 | 41 | 91.1 |
Does it include both paid and volunteer personnel? | 3 | 8.8 | 6 | 100.0 | 5 | 100.0 | 14 | 31.1 |
Health facility case notification form available? | 24 | 70.6 | 6 | 100.0 | 5 | 100.0 | 35 | 77.8 |
Index case investigation form available? | 28 | 82.4 | 6 | 100.0 | 5 | 100.0 | 39 | 86.7 |
RACD form available? | 12 | 35.3 | 6 | 100.0 | 5 | 100.0 | 23 | 51.1 |
Reporting forms availability subtotal
|
64/102
|
62.7
|
18/18
|
100.00
|
15/15
|
100.0
|
97/135
|
71.8
|
SOPs for health facility diagnosis and notification? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
SOPs for index case investigation? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
SOPs for RACD? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
SOP availability subtotal
|
9/102
|
8.8
|
6/6
|
100.0
|
0/5
|
0.0
|
27/135
|
20.0
|
Diagram of process for health facility diagnosis and reporting? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
Diagram of process for case investigation? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
Diagram of process for RACD? | 3 | 8.8 | 6 | 100.0 | 0 | 0.0 | 9 | 20.0 |
Diagram availability subtotal
|
9/102
|
8.8
|
6/6
|
100.0
|
0/5
|
0
|
27/135
|
20.0
|
Module 2. Assessing key malaria indicators
Indicators | Aceh | Jiangsu | Ranong | Total | ||||
---|---|---|---|---|---|---|---|---|
No. | % | No. | % | No. | % | No. | % | |
Malaria cases reported to the database from health facilities | 112/120 | 93.3 | 42/42 | 100 | 510/510e
| 100.0 | 664/672 | 98.8 |
Malaria cases reported to the database within a specific amount of timea
| 106/120 | 88.3 | 42/42 | 100 | 259/510e
| 50.8 | 407/672 | 60.5 |
Malaria cases reported to the database that were investigated | 87/111 | 78.4 | 42/42 | 100 | 465/752 | 61.8 | 594/905 | 65.6 |
Malaria cases reported to the database that were investigated within a specified amount of timeb
| 79/87 | 90.8 | 42/42 | 100 | 394/465 | 84.7 | 515/594 | 86.7 |
RACD events that occurred (out of total RACD events that should occurd) | 57/58 | 98.3 | 19/19 | 100 | 271/419 | 64.7 | 347/496 | 70.0 |
RACD events that occurred within a specified amount of timec
| 47/58 | 81.0 | 19/19 | 100 | 229/271 | 84.5 | 295/348 | 84.7 |
Total population screened during RACD events | 931 | – | n/a | – | 18,505 | – | 19,436 | – |
Positive malaria cases identified through RACD | 3 | – | 0 | – | 26 | – | 29 | – |
Module 3. Evaluating standard operating procedures
Aceh | Jiangsu | Ranong | ||||||
---|---|---|---|---|---|---|---|---|
Responses | n | % | Responses | n | % | Responses | n | % |
1. Question: what is the minimum geographic radius to screen around an index case household during RACD? | ||||||||
100 m | 15 | 44 | 50–100 m | 4 | 40 |
1 km
|
2
|
18
|
200 m | 1 | 3 | 150–200 m | 2 | 20 |
2 km
|
2
|
18
|
500 m
|
13
|
38
|
1 km
|
1
|
10
| 3 km | 2 | 18 |
No radius | 2 | 6 | No radius | 2 | 20 | 5 km | 1 | 10 |
No response | 3 | 9 | No response | 1 | 10 | No radius | 4 | 36 |
Total |
34
|
100
| Total |
10
|
100
| Total |
11
a
|
100
|
Responses | Aceh | Jiangsu | Ranong | |||
---|---|---|---|---|---|---|
n | % | n | % | n | % | |
2. Question: Which individuals should you screen when conducting RACD around an index case household? | ||||||
Febrile only | 14 | 41 | 2 | 20 | 1 | 9 |
All (asymptomatic and febrile)
|
17
|
50
|
7
|
70
|
8
|
73
|
We do not screen household members of a positive case | 3 | 9 | 1 | 10 | 2 | 18 |
Total |
34
| 100 |
10
| 100 |
11
a
| 100 |
3. Question: what triggers screening in the community? | ||||||
Local cases only | 5 | 15 | 0 | 0 | 2 | 18 |
Local and imported cases
|
26
|
76
|
10
|
100
|
7
|
64
|
Imported cases only | 0 | 0 | 0 | 0 | 0 | 0 |
When local cases reach a minimum threshold | 0 | 0 | 0 | 0 | 0 | 0 |
Neighbors of index case never screened | 3 | 9 | – | – | – | – |
Total |
34
| 100 |
10
| 100 |
9
a, b
| 100 |
4. Question: how often do you screen neighbors around index case household? | ||||||
We always screen neighbors
|
13
|
38
|
6
|
60
|
11
|
73
|
Sometimes screen neighbors | 18 | 53 | 3 | 30 | 4 | 27 |
Never screen neighbors | 3 | 9 | 1 | 10 | 0 | 0 |
Total |
34
| 100 |
10
| 100 |
15
| 100 |
Module 4. Estimating the costs
Study area | Data collection period | Personnela
| Commodities, services and other | Total cost | Average monthly costb
| ||||
---|---|---|---|---|---|---|---|---|---|
All malaria activities | CI/RACD only | All other malaria activities | CI/RACD only | All malaria activities | CI/RACD only | All malaria activities | CI/RACD only | ||
Aceh | September 2013 (1 month) | n/a | $ 3469.56 | n/a | $ 257.55 | n/a | $ 3727.11 | n/a | $ 3727.11 |
Jiangsu | January–December 2012 (12 months) | $ 9101.13 | $ 4550.56 | $ 5513.89 | $ 5587.08 | $ 20,202.10 | $ 10,137.64 | $ 1683.50 | $ 844.80 |
Ranong | January–December 2014 (12 months) | n/a | $ 10,486.61 | n/a | $ 13,969.43 | n/a | $ 24,456.04 | n/a | $ 2038.00 |
Assessment of RACD M&E tool
Suggested changes to tool
Suggestions by programme implementers | |
---|---|
RACD M&E tool overall | Create user manuals for district- and national-level programme staff for each module to provide more detail on the inputs required for data collection |
Enable translations for each template sheet to increase use of tool | |
Create automatic data analysis pages in each module for district-level monitoring | |
Develop macros in Excel to allow provincial- and national-level users to compile results across districts | |
Module 1: reviewing key documents | Create preset drop-downs with yes/no responses for inventory checklist items |
Module 2: assessing key malaria indicators | Reduce the number of data entry sheets from 3 (health facility reporting, case investigation and RACD) to 2 (health facility data, district level data) by combining case investigation and RACD into a single data collection sheet |
Simplify the district-level indicators sheet for case investigation and RACD by including all the indicators on a single data entry sheet, separate from the review pages | |
Create option for preset drop-down lists to make data entry quicker, and to maintain consistency in the spelling of district and province names | |
Add indicator for RACD screening coverage | |
Module 3: evaluating standard operating procedures | Develop questionnaire in open source platform or improve data analysis in current Excel template to make it easier |
Allow programme users to modify the questionnaire module to match programme activities and needs | |
Module 4: estimating the costs | Reduce the number of data entry sheets from 3 (Personnel, Commodities and Services and Other) to 2 (Personnel and Commodities, Services and Other) into a single data collection sheet. Include a drop-down list to identify the expense |
Impact on programme decision making
Study site | RACD M&E tool impact on programme decision making |
---|---|
Aceh, Indonesia | Broadened use of RACD M&E tool to other districts in Aceh Province and other provinces within Indonesia |
Recommend the integration of the M&E tool into the current national malaria case reporting system (referred to as e-SISMAL) | |
Conducted refresher trainings with malaria officers and microscopists at district level on the knowledge and information at primary health centers through routine meetings | |
Recommended to provincial health office on the use of standardized case investigation and RACD forms for entire province, and need to undertake a notification form from the ministry of health that is under the responsibility of the surveillance unit | |
Advocated to district and provincial health offices to allocate more budget for supervision and field monitoring | |
Set up random monitoring of malaria program implementation in Aceh for quality assurance of activities and reporting | |
Recommend the development of a tool for M&E in malaria diagnosis QA system to be created and tested in Sabang and Aceh Besar Districts before scaling up nationally | |
Jiangsu, China | Carried out additional evaluations in Jiangsu and Yunnan Province in China |
Improved China’s 1-3-7 SOPs in Jiangsu, making the SOPs more suitable for the local context | |
Added indicators of completion rates for the China’s national 1-3-7 reporting framework to the routine diseases surveillance information system (CRDSIS) | |
Jiangsu Institute of Parasitic diseases (JIPD) malaria division carried out additional trainings on malaria reporting system management, epidemiology investigation and foci disposals for basic level CDCs staff in Jiangsu province | |
Ranong, Thailand | Adopted routine monitoring and evaluation activities into national surveillance guidelines |
Incorporated into online database for all reporting facilities standardizing indicators for routine reporting | |
Conducted refresher trainings on case investigation and RACD because gaps identified differences in how activities were conducted | |
Recommend to conduct a full national evaluation using the M&E tool | |
Recommended including a rapid reporting system because gaps were identified from remote clinic areas (malaria post, border malaria post) and malaria clinics without computers/internet have reports that are not timely/complete | |
Recommend integration of the case investigations into the primary health system to not miss case investigations and to be more timely/complete |