Achievements
Data from the in-depth interviews and focus group discussions demonstrate several achievements from the establishment of the district M&E officer cadre. These include improved health worker capacity to monitor and evaluate programs within the districts; improved data quality, management, and reporting; increased use of health data for disease surveillance and public health services planning purposes; introduction of district-led operational research activities; and increased availability of time for nurses and other health workers to concentrate on core clinical duties.
Improved district-level capacity to monitor and evaluate programs
District M&E officers helped build M&E capacity in the districts by providing in-service training and mentoring to health workers based at the district health offices, health facilities, and civil society organizations. Capacity-building efforts focused on the significance of M&E, data quality (timeliness, completeness, reliability, validity, and accuracy), and data analysis. Health workers received training on the existing data collection tools and data management as indicated by what this district M&E officer said during a focus group discussion:
“We [the M&E officers] have trained them [health workers] on the importance of data management, and they have started appreciating M&E…Data quality has improved a lot.”
Training and mentoring in M&E was conducted on-the-job or during scheduled district meetings. Data audit activities and facility support visits were used to identify capacity development needs for health workers, such as basic computer skills. A notable outcome of the capacity-building efforts has been the increased awareness and appreciation of M&E, and the creation of a culture where its recognition has led to its inclusion in district meetings. Two key informants during in-depth interviews respectively said:
“We also saw a lot of engagement with the districts now. People are beginning to ask a lot of questions regarding the science behind some issues that we included in the data collection tools… this is consciousness. People are beginning to question the inclusion of certain variables in those tools.”
“…If you could count the number of times you hear the word “M&E” in workshops nowadays, it is probably 10–20 times more than it was before they [the district M&E officers] came in. Now that is what I’m calling the development of an M&E culture in this country. That, if nothing else, is the ultimate achievement.”
Improved data quality, management, and reporting
Each cadre of interviewee, district M&E officers, program officers, and district managers, acknowledged the contribution of district M&E officers in helping improve data quality, management, and reporting. Improvements in data quality included accuracy, timeliness, validity, and completeness. Introduction of data audits and standardization of data collection tools contributed to the perceived improvements in accuracy and completeness of reporting. The development of tools to track the submission of reports from facilities and feedback given to facilities helped improve timeliness and reporting. Three district managers respectively remarked in focus group discussions:
“There has been some improvement in terms of timely submissions and completeness [of data and reports], though not a hundred percent."
“There has been reasonable improvement in the way the data is collected, [and] the way the data is entered, interpreted, and reports made.”
“Even as management, we now get proper and true representation of the [Health] data from the facilities. Now, we are confident of the validity of the data.”
District M&E officers also introduced systematic filing of health reports by maintaining centrally placed manual files and creating electronic databases, thereby improving data management. This improved both the availability and access to aggregate health data. In focus group discussions, a program officer and a district M&E officer, respectively, reported:
“Their [M&E officers’] data is safely kept in a way that if anyone comes into the office and is enquiring about something, it does not take long to find information…”
“At my district, I met with program officers to train them on making electronic files and saving data on them. Initially they were just recording manually, and you would hardly find a trace of that information later on when you wanted to make reference to it. Now, we keep both manual and electronic folders, and accessibility to data has improved.”
Increased use of health data for disease surveillance, planning and project management
The presence of the district M&E officers improved the use of local data for disease surveillance purposes. Data were analyzed, and disease trends were reported to the district health management teams. District M&E officers became important members of disease control teams and contributed to outbreak investigations. During a focus group discussion, a district manager expressed the following:
“They [M&E officers] are able to alert the head professionals when the statistics are high, especially the emergency of outbreaks like the diarrhoea outbreaks.”
In addition to surveillance, improved data quality better informed evidence-based planning at the district level. Health program monitoring and evaluation data has been used to guide the planning process and determine priorities. Two district managers remarked during a focus group:
“…even though we were using data for evidence-based planning, there wasn’t quality data that really informed us appropriately. But now, the district M&E officers… really work on this data and ensure that it is quality data that we need for planning. This is a priority in the district now, you are basing [decisions] on reliable data, unlike in the past we were using data which is not reliable.”
“When you put a priority, as part of plan, they will say that the evidence does not reflect that. You can’t put that as a priority because your numbers for this [are] not indicative.”
The district M&E officers were reported to monitor implementation of district health plans and provide district health teams feedback on progress towards objectives. This strengthened project management, and districts were able to keep track of implementation of prioritised activities. A district manager expressed the following:
“They [district M&E officers] crosscheck to see if we are on track, if we have attained our goals, if we are following the set objectives and timelines, and if the funds are directed towards the planned activities or diverted to other ‘emerging needs’, which are not a priority in the district.”
Introduction of district-led operational research activities
District M&E officers introduced operational research in the districts they supported in order to better understand health events. Operational research questions were generated from routinely collected data. Findings from research were shared with the district teams, and some, at a national HIV/AIDS research conference. A program manager and a key informant, respectively, stated:
“…I understand the district M&E officer did a needs assessment to try to address the issue of teenage pregnancy in district[X]. That needs assessment was used to inform the evidence-based plan.”
“..they started getting involved in conducting basic research and sharing their results at conferences, the first one being a National HIV/AIDS Research Conference where they presented some research activities that they were doing.”
Increased availability of time for nurses and other health workers to concentrate on core clinical duties
The district M&E officers took on M&E responsibilities that were previously conducted by nurses and other health workers as secondary activities to core program and clinical responsibilities. This allowed such health workers to concentrate on delivering quality clinical services and focusing on implementation of quality, priority health programs. A district manager reported:
“So, when the district M&E officers came in, they relieved the community health nurse in such a way that the community health nurse is able to go to facilities to attend to such programs as child health and others. The district M&E officer then took up [data responsibilities] for different HIV programs.”
Despite taking over M&E responsibilities, district M&E officers worked closely with other health workers coordinating different programs as most program data comes through them.
Lessons Learnt
Lessons learnt from the qualitative assessment of the M&E officer cadre included the importance of clarifying of roles for newly established cadres, aligning resources and equipment to expectations, and ensuring stakeholder collaboration. Another lesson learnt was the need to ensure retention of new cadres.
Clarity regarding how tasks are shifted was essential for the acceptance of a new cadre within the system
When district M&E officers were deployed to the districts, their duties had not been well articulated and communicated to them, their supervisors, nor their colleagues. As a result, they were often tasked with other activities outside M&E. District M&E officers felt that this lack of clarity related to roles and responsibilities was an impediment to developing cooperative working relationships with colleagues. In some instances, district M&E officers were perceived as a threat by program officers to their job security since the cadre had been deployed to the districts to take over some of the duties previously conducted by program officers. A key informant during an in-depth interview and a program officer in a focus group discussion remarked:
“But at the end of the day, we also needed to have prepared district teams themselves… That part [orientation] was not done, and that was a serious omission....”
“The issue is that their role was never clearly defined and therefore we didn’t know how we are to work with them. If their roles were clear to us, we would have come up with a way so that it works easier for everybody since we would be having an understanding.”
Expectations of a newly established cadre must be aligned to available resources
District M&E officers reported that they often struggled to execute their duties effectively due to inadequate resources. These included transport and communication infrastructure challenges, and unreliable access to computers and the Internet. Transport challenges oftentimes affected planned facility-level activities like data audits, while limited access to telephone and facsimile affected reporting and feedback. Unavailability of internet in some districts made it difficult for the district M&E officers to use web-based data reporting tools, such as the district health information system (DHIS). A district M&E officer and a key informant, respectively, reported:
“I have never used DHIS. They introduced DHIS 2.0, and I am supposed to be connected to the internet, but I don’t have it at my facility. I don’t see it working.”
“A lot of things were expected from them [district M&E officers], but without giving enough logistical support. There was no internet in the office. This was challenging for them.”
Planning for career growth and retention is critical
The district M&E officer cadre was initially donor-funded, on fixed-term contracts without a career trajectory, compared to other health workers employed directly through the public health service establishment. While plans were available for a smooth transition of the district M&E officers into the public service structure, being on contract led to job insecurity. As a result, some district M&E officers resigned from their positions. A program officer expressed the following:
“They are watching their colleagues moving up the ladder! They are just in one place. Even the good ones…. we are going to lose them if the trend continues.”
Realising the need to retain the district M&E officers, stakeholders agreed to offer an additional allowance as a form of incentive to retain the district M&E officers. Despite this effort by stakeholders, job security still remained a concern. Two key informants remarked:
“…strategies were put in place to retain them. And hence, there was what we called a “contractual allowance.”
“..the stakeholders met regarding retention of the officers and gave them 20% of their salary but this didn’t address the issue of [job]security.”
Stakeholder collaboration and building local capacity contributed to success and sustainability
Multiple stakeholders, including national institutions, donor agencies, and technical organisations, collaborated in the establishment and implementation of the district M&E officer cadre. These stakeholders worked together as members of the technical working group that oversaw the establishment of the cadre, and brought together various areas of expertise that were critical for the cadre. The recruitment of nationals of Botswana for the district M&E officer positions contributed to sustainability of the M&E discipline within Botswana. Despite some of the M&E officers resigning, they joined other organisations within the country as M&E officers and have continued to contribute to the national M&E efforts. Two key informants indicated:
“And I would say one of the biggest successes of the project, these were all local individuals. They were all hired locally and even if they move, they are still in country, working with other partners and donors, still in the same field. So I would say we did build capacity in the country when it comes to monitoring and evaluation”
“… a lot of the district M&E officers now are still in the field of monitoring and evaluation, with different partners, with different United States Government-funded partners and government.”
Additionally, senior M&E officers at the national level were mentored to ensure sustained capacity to provide technical support to the district M&E officers. For continuous capacity building in M&E, stakeholders also developed self-guided training materials, consisting of workbooks and a classroom-based curriculum. These were meant for ongoing, self-guided use by district M&E officers. Two key informants said:
“Mentoring was offered to national-level senior M&E officers to ensure that the district M&E officers continue getting the right support.”
“Stakeholders developed training materials which should really be utilised; they should be availed so that these officers utilize these materials for their own growth on the job and skills development.”