Qualitative evaluation of RTI interventions
Based on our preconceptions (reflexivity principle) and on the iterative discussions performed until the theoretical saturation point was reached, the analysis issues were consolidated eventually into three main categories, namely driving forces for selection of RTI interventions, programmatic characteristics of interventions, and monitoring and evaluation aspects.
A common concern expressed by most interviewees was the limited scale of implementation of RTI interventions. Despite initial plans for implementation at scale across the whole country, an overwhelming number of these interventions are implemented only in few major cities, and in a very scattered way (Table
1), reducing therefore their potential impact. A noticeable example is Zero Tolerance, considered later in this report.
Selecting RTI interventions in Peru
There was a general consensus that the main motivation for selecting RTI interventions varied in different settings, although a salient driving force highlighted was the pressure exerted by the media and the community. This finding is in line with our assumption that interventions are not chosen on the basis of existing evidence, but also reveals additional driving forces, most notably the media influence, which has also been reported in other settings [
27]. As an interviewee explained:
Every day we read or watch news on crashes and victims. Most frequently drivers are shown as those who cause them. The media forces politicians to choose quickly any intervention, no matter if it is effective or not, because they fear the media and also the community anger.
(Officer/Implementer, Technical Secretariat of Road Safety Council, Lima)
Drivers are consistently publicized as those with the greatest responsibility in the occurrence of road traffic crashes, being perceived as actors with deviant behaviours, and thus the interventions targeted to this group occupy a prominent place at all levels in the country. The media also promotes widely education interventions; an approach that has therefore received prominent attention by policymakers, although the evidence questions its effectiveness, especially when implemented in isolation [
1]. An interviewee put it this way:
We all know that most drivers have inadequate behaviours and psychological problems, and that they cause the crashes. Thus we would like to change those behaviours, we would like to educate them.
(Technical officer of Road Safety Education, Lima)
Interestingly, the interviewees agreed with the media on the importance of strengthening current students and drivers' education interventions despite informing them about available evidence from Peru suggesting otherwise [
27]. This indicates that it is not enough to ensure that research information is shared. It is more important that it is packaged appropriately and specifically for use by policymakers and that policymakers are equipped with the necessary skills to discuss and evaluate the quality of evidence they are presented with [
28].
Another salient factor influencing the selection of RTI interventions was the availability of refundable external funds conditioning the loans to the development of road safety strategies, suggesting the need to strengthen national and local initiatives aimed at identifying the priority aspects related to RTI interventions, while also taking advantage of external sources of funding and policy influence. One example of such an external driving force is a World Bank loan in the 1990s that required the Peruvian government to assign 10% of the loan to the development of a comprehensive road safety strategy. As a policymaker explained:
The World Bank gave money to our country when Fujimori was president, and said that they were going to borrow us 100 M bucks, but that government should spend them in road safety. Thus a consultancy was commissioned, and this concluded that the country needed a cross-sectoral road safety organization. And thus the National Council of Road Safety was created in 2004.
(Policymaker, Technical Secretariat of Road Safety Council, Lima)
At regional level, the need to comply with municipal or regional level mandates; efforts by NGOs focusing on road safety; and the decentralization process at regional and local governments were all considered as important driving forces. They illustrate the need of considering context-specific aspects when planning and implementing RTI interventions at sub-national level.
Main programmatic drawbacks related to RTI interventions
Interviewees consistently identified programmatic drawbacks in the policy process of design and implementation of RTI interventions in Peru, which include:
(a)
Lack of clear and sustained political and budgetary support for both national and local level interventions, which are aggravated by frequent change of high-level policymakers. As an interviewee explained:
"...political will has been absent in all the efforts. We always notice a lack of interest. Whenever a new minister is in charge, which happens all the time, he has to be informed again and again, and we have to wait for him to become familiar with the problem.
(Coordinator, Violence Observatory, Universidad Peruana Cayetano Heredia, Lima)
(b)
Lack of clear planning and implementation guides, which lead to a high degree of improvisation when selecting potential RTI interventions, and when implementing them. An interviewee put it this way:
"...not only for traffic injuries, we need to made and effort of thinking on all things that happen in our country. But today things are done without thinking. We spend a lot of money implementing things without planning them carefully, without measuring their impact. Then they are abandoned and no one knows why they didn't work. And we do not learn from our mistakes.
(Coordinator, Violence Observatory, Universidad Peruana Cayetano Heredia, Lima)
(c)
Lack of regular, planned training and supervision activities; and scarcity of dedicated staff to implement and enforce RTI interventions. Policymakers and implementers felt that besides structural and budgetary limitations, scarcity of trained human resources and of a supportive supervision system limit the implementation of RTI interventions. One technical officer explained:
With the problem of road traffic injuries, we have structural and budgetary limitations, and limited human resources, but also insufficient training and supervision. We don't even have ad-hoc physical spaces for performing training activities.
(Technical officer, Training Unit, Municipality of Pucallpa, Ucayali)
(d)
Ineffective coordination between the different sectors involved, which is particularly important when designing and implementing complex, crosscutting RTI interventions. A policymaker put it this way:
Definitely, if you don't act in a multisectoral way, you will fail. The safety road issue needs cooperation, coordination, if you want to have an impact.
(Central level policymaker, National Strategy for Road Traffic Injuries Ministry of Health, Lima)
(e)
Insufficient community participation, which reveal the need to consider more seriously the opinions and the active participation of community members when planning and implementing the interventions. This perception is line with our anticipated theoretical framework, and reduces the likelihood of an effective enforcement of several regulatory interventions. As a policymaker explained:
Very few interventions in our country consider actually community participation. Fortunately, our education program for road safety includes community members that are very committed to contribute. They are very important because they talk to their fellow citizens, and they are influential.
(Responsible, Road Safety Program, San Borja Municipality, Lima)
(f)
Lack a reliable and fully functional information system. This limits seriously the assessment of the actual burden posed by RTIs and the possibility of measuring reliably the impact of RTI interventions. As an interviewee explained:
We need to work with clear objectives, we need to have a good database on road traffic crashes... we need those indicators to measure our work.
(Responsible, Road Safety Program, San Borja Municipality, Lima)
These perceptions were broadly in agreement with our theory framework, although interviews revealed further context-specific aspects not anticipated, such as lack of adequate regulations for local transport vehicles.
At regional and local level, the most important programmatic issue raised was weak regulatory framework and enforcement of regulations. In Ayacucho and Pucallpa in particular, interviewees were concerned about the lack of adequate road safety regulations to address the continuous increase of motorcars or moto-taxis, with additional emphasis expressed on the limited success in the enforcement of the existing regulations. As local policymakers explained:
I think that we need a tight control. We really need that the police make an effective control of road safety infractions... We have too many regulations, but they are not well planned. And they are not enforced, you know. People don't care about those regulations.
(Responsible, Training Unit, Municipality of Pucallpa, Ucayali)
"...There are too many moto-taxis. And their drivers don't care about road safety regulations. But we don't have good regulations for them. They are fragile and dangerous as well. We also need to educate drivers of moto-taxis.
(Responsible, Training Program for moto-taxis, Coronel Portillo Municipality, Pucallpa)
With regard to suggested actions to improve the implementation of current RTI interventions, the most important ones were the need for a stronger multisectoral coordination of scaling-up activities; and the need to develop strategies to overcome the cultural, social and geographical diversity of the country, and to address the specificities and modalities for enforcing regulations in the different regions. As a local policymaker explained:
...well, the best lesson learned is the need of coordinated work: authorities, transport colleagues, all actors. Then we can have more efficient actions, better results. But we should remember that each place has its own problems, moto-taxis for instance, are a problem here, but not so much in Lima. Also, you know that use of seat belts are going to be very difficult in Ucayali, because it's so hot here. And can you imagine seat belts in moto-taxis? Really hard.
(Policymaker, Road Safety Training Unit, Ministry of Transport, Ucayali)
Regarding the media influence at local level, as the following quote highlights, a wide but responsible participation and support from the media was also identified as an important and crucial step in the successful implementation of RTI interventions and their enforcement:
Well, journalists are very important, but they shouldn't be so sensationalists, they should be more responsible when they release the news. Only in that way they are going to help really to improve the problem of road crashes.
(Policymaker, National Strategy for Road Traffic Injuries Ministry of Health, Lima)
Monitoring and evaluation
Finally, the interviewees consistently pointed out that none of the interventions have systematic monitoring and evaluation activities, and no clearly identified process and impact indicators to evaluate progress along the way. When they exist, they are most often related to the measurement of administrative goals, rather than to the impact of the implemented interventions themselves or to identification of aspects needing improvement. This reveals that monitoring and evaluation limitations are far more dramatic than we had assumed in our theoretical framework:
...no, we don't monitor, we don't have the monitoring tools, we don't have evaluation indicators, we don't have the necessary budget.
(Policymaker, Communication Unit, Road Safety Decentralization Program, Lima)
It is hard to talk about monitoring, you know, especially when implementation of activities is still limited. We have explored this, however, and facilitators should be trained, so they can monitor and offer counseling.
(Policymaker, Safety Road Education Program, Ministry of Education, Lima)
Informal evaluations... yes, we do. But actual evaluation of results and monitoring, no, because it is not sustainable. Probably in the future, after massive training of drivers.
(Policymaker, Training Courses Section, Coronel Portillo Municipality, Pucallpa, Ucayali)
We have a supervisor who tries to ensure a small evaluation at the end. We were thinking to give an incentive to schools that show improvement in road safety. We would like to perform a theater representation.
(Implementer, Safety Road Project, Ayacucho Municipality)
Although they acknowledged the existence of various information systems in the police, Ministry of Transport, Ministry of Health, and the private sector, which could be used as useful information sources for monitoring and evaluation efforts, participants expressed concern about the quality and completeness of the assembled information, the limited access to the existing information, and the lack of coordination and integration of these different information systems. This finding, which we did not specifically include in our theoretical framework, highlights that a construction of a functional and integrated information system for RTIs remains a pending task at national and local level:
...we have agreed to share the information. The police, the judicial system, the insurance companies and other sectors have agreed. But there are no concrete results. For example, we wanted to use the district police information, but they didn't want. They said that there was not superior approval. We need written agreements, maybe a law. We need to improve the quality of the information. It needs to be complete. If there are not data, we don't have monitoring and evaluation, you know.
(Policymaker, Health District, Lima)
"...we have just started a joint work with the statistics sections of diverse sectors through the regional government. A Coordination Committee has been established to allow sharing of information on violence and road traffic crashes. We have met to coordinate. We hope to have access to different information sources.
(Chief, National Police, Ayacucho)
Stakeholders meeting
The meeting generated a wide array of ideas and suggestions applicable to the design, implementation and evaluation phases of the policy process. Many of them were consistent with those provided in the in-depth interviews presented above. The most important or recurring ones are highlighted here.
First, the participants emphasized the need to promote intersectoral alliances both for normative and operational aspects of RTI interventions, as otherwise overlapping, and sometimes-competing efforts will hamper the impact and effectiveness of different interventions.
Second, they insisted on the need to strengthen enforcement of road safety regulatory strategies. In particular, they emphasized that the police needs to be supported from inside and outside to improve its social surveillance role. This would include strengthening of all aspects of the police system, including audit and accountability mechanisms, training, logistic and budgetary aspects. The lack of availability of even basic supplies such as alcohol measurement devices and of combustible fuel for the patrols were highlighted as dramatic examples that jeopardize an effective participation of police in the implementation of RTI interventions such as control of speed limits and alcohol-impaired driving.
Third, the participants highlighted the importance of sustaining and improving technical training and regular supervision of human resources involved in different aspects of RTI implementation efforts, at both national and local level.
Fourth, they felt that improvement of quality and accessibility of information related to RTIs is a pending accomplishment. Although several information systems are in place, there is a critical need for a thorough process to improve the structural and functional aspects of each information system, to identify core indicators, to have easy access to meaningful data, and to effectively coordinate and integrate the generated data to get reliable and updated epidemiologic profiles of RTIs at national and sub-national level.
System-wide effects of zero tolerance
The brainstorming session and email discussions proved to be a very instructive and useful exercise that can be easily repeated for other interventions. The output of this exercise for Zero Tolerance is summarized in Table
2. Zero Tolerance inspections are currently limited to main departure points in Lima and a few large cities of the country. Buses and drivers undergoing inspection are not chosen randomly or systematically. While inspection of vehicles and drivers should occur with the joint participation of Ministry of Transport officers, police officers and a public prosecutor, in reality, they are performed sporadically by a Ministry of Transport officer, who is often short of the necessary equipment and public support for an effective enforcement of the corrective measures. Early in its implementation, unrealistic expectations for the potential impact of Zero Tolerance were raised both by the media and politicians. It was burdened, therefore, with ample discredit and wide public criticism that resulted in decreased political commitment and limited allocation of resources. The most important perceived negative system-wide effects were unplanned diversion of human resources and unfulfilled aims to setup a functional information system for monitoring and evaluation. Important potential positive effects such as empowerment of passengers and accountability of public bus drivers were only partially accomplished (Table
2).
Table 2
Prioritized perceived system-wide effects of Zero Tolerance
1 | Narrow target group therefore low impact (inter-provincial public buses only) | - | High | High | All sectors-All building blocks |
1 | Decreased political commitment due to wide criticism | - | High | High | Central and local government-Leadership and Governance |
1 | Decreased budget and logistic resources due to criticism | - | High | High | Central and local government-Financing |
1 | Diverted HR and understaffing in involved sectors (MoT, Police, Ministry of Justice) | - | High | High | Crosscutting-HR & Delivery |
1 | Decreased police & passengers support to MoT officers | - | High | High | Crosscutting-Delivery |
1 | Increased accountability perception | + | Low | High | Drivers and vehicle owners-Governance and accountability |
1 | Opportunity for passengers empowerment | + | Medium | High | People-Governance and accountability |
1 | Unfulfilled aim to setup information system for M&E | - | High | High | MoT and Police-Information |
2 | Overburdened police workforce | - | High | High | Police-Delivery |
2 | Opportunity for coordinated crosscutting activities | + | Medium | High | All sectors-Delivery |
3 | Unrealistic media-driven expectations | - | Medium | Medium | Media-Leadership & Governance |
3 | Discrimination feeling of private transport sector (the "bad guys") | - | High | Medium | Private transport sector-All building blocks |
4 | Overlapping of police and MoT officers' activities | - | Medium | Low | MoT & Police-All building blocks |
4 | Traffic crowding at check points | - | Medium | Low | Crosscutting-Delivery |
Overall, the brainstorming exercise revealed that although there is room for substantial redesign of Zero Tolerance at different building blocks of the health system and the different sectors involved, policymakers may also wish to reconsider whether it deserves continued implementation, due to its very narrow target group (only public inter-provincial buses), and thus the remote likelihood of any measurable impact on the overall rate of RTIs.