Background
Overview of PRS development
Methods
Delphi exercise to reach consensus on core PRS items
Delphi participants and procedure
Rounds 1 and 2
Round 3
Technical consultation to refine and finalize draft PRS
Piloting the draft PRS in existing programmes
Results
Delphi results
Participant characteristics
Round 1 (N = 48) | Round 2 (N = 29) | Round 3 (N = 20) | ||||
---|---|---|---|---|---|---|
N | % | N | % | N | % | |
Organization | ||||||
UN | 5 | 10.4% | 3 | 10.7% | 2 | 10% |
University | 16 | 33.3% | 10 | 35.7% | 8 | 40% |
NGO | 14 | 29.2% | 8 | 28.6% | 5 | 25% |
Hospital/clinic | 2 | 4.2% | 0 | 0% | 0 | 0% |
Donor | 4 | 8.3% | 3 | 10.7% | 3 | 15% |
Government | 5 | 10.4% | 3 | 10.7% | 1 | 5% |
Other | 2 | 4.2% | 1 | 3.6% | 1 | 5% |
Professional backgrounda
| ||||||
Programme planner/implementer | 25 | 52% | 13 | 44.8% | 9 | 45% |
Researcher/academic | 31 | 65.3% | 15 | 51.7% | 11 | 55% |
Medical/clinical | 16 | 32.7% | 8 | 27.6% | 7 | 35% |
Management | 19 | 37.8% | 10 | 34.5% | 8 | 40% |
Other | 6 | 12.2% | 5 | 17.2% | 2 | 10% |
Region | ||||||
Global | 33 | 68.8% | 21 | 72.4% | 14 | 70% |
Africa | 6 | 12.5% | 3 | 10.3% | 2 | 10% |
Western pacific | 2 | 4.2% | 1 | 3.4% | 1 | 5% |
Southeast Asia | 3 | 6.3% | 2 | 6.9% | 2 | 10% |
North America | 3 | 6.3% | 2 | 6.9% | 1 | 5% |
Europe | 1 | 2.1% | 0 | 0% | 0 | 0% |
Item rankings and suggestions
Round 1 (N = 43) | Round 2 (N = 29) | Round 3 (N = 20) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Original items (systematic review) | Rating scores | Category of importance (% of respondents)a
| Revised items post round 1 | Rating scores | Category of importance (% of respondents)a
| Revised items post round 2 | Item rankingb
| ||||
Mean (SD) | Ess. | Imp. | Notimp. | Mean (SD) | Ess. | Imp. | Notimp. | ||||
Programme Preparation
|
Programme Preparation
|
Programme Preparation
| |||||||||
1. Programme name | 6.9 (2.2) | 58.1 | 37.2 | 4.7 | 1. Programme name | 7.2 (1.7) | 62.1 | 34.5 | 3.5 | 1. Programme name | Suppl. |
2. Overall goal/objectives – anticipated impact | 8.8 (0.6) | 97.7 | 2.3 | – | 2. Objectives and anticipated effects | 8.7 (0.7) | 96.6 | 3.5 | – | 2. Objectives and anticipated effects | Ess. |
3. Target population | 8.4 (1.4) | 93.0 | 4.7 | 2.3 | 3. Target population and area | 8.5 (0.8) | 96.6 | 3.5 | – | 3. Target population and area | Ess.c
|
4. Organization/agency | 6.4 (2.0) | 48.8 | 44.2 | 7 | 4. Partners and stakeholder involvement | 7 (1.5) | 65.5 | 34.5 | – | 4. Partners and stakeholder involvement | Suppl. |
5. Funding source | 6.2 (1.7) | 39.5 | 58.1 | 2.3 | 5. Funding source | 6.4 (1.7) | 44.8 | 48.4 | 6.9 | 5. Funding source | Suppl. |
6. Programme design process | 7.2 (1.6) | 69.8 | 27.9 | 2.3 | 6. Programme design process | 7 (1.5) | 62.1 | 37.9 | 6. Programme design process | Suppl. | |
7. Theoretical foundation | 7.3 (1.8) | 79.1 | 16.3 | 4.7 | 7. Theory and/or logic model | 7.6 (1.4) | 82.8 | 13.8 | 3.5 | 7. Theory and/or logic model | Ess. |
8. Program manual | 6.7 (1.6) | 55.8 | 41.9 | 2.3 | 8. Program manual | 5.8 (1.8) | 41.4 | 48.3 | 10.3 | 8. Program manual | Suppl. |
9. Implementation strategy | 7.8 (1.4) | 81.4 | 18.6 | – | 9. Implementation strategy | 7.8 (1.2) | 82.8 | 17.2 | – | 9. Implementation strategy | Ess. |
10. Evaluation plans | 8.4 (1.1) | 93.0 | 7.0 | – | 10. Evaluation plans | 7.9 (1.2) | 85.7 | 14.3 | – | 10. Evaluation plans | Ess. |
11. Ethical considerations | 7.4 (1.4) | 72.4 | 27.6 | – | 11. Ethical considerations | Suppl. | |||||
12. Dissemination plans | 6.5 (1.9) | 55.2 | 41.4 | 3.5 | 12. Dissemination plans | Suppl. | |||||
11. Piloting of activities | 79.1 | 20.9 | – | 13. Piloting of activities | 7 (1.2) | 62.1 | 37.9 | – | 13. Piloting of activities | Suppl. | |
Programme
| – |
Programme
| – |
Programme
| |||||||
Implementation
|
Implementation
|
Implementation
| |||||||||
12. Components/activities | 8.4 (1.1) | 88.4 | 11.6 | – | 14. Components/activities | 8.3 (1.2) | 89.7 | 10.3 | – | 14. Components/ activities | Ess. |
13. Complexity | 7.2 (1.4) | 66.7 | 33.3 | – | 15. Complexity | 6.8 (1.2) | 37.9 | 62.1 | – |
Merged with #14
| |
14. Standardisation | 7.5 (1.3) | 81.4 | 18.6 | – | 16. Standardisation and tailoring | 7.7 (1) | 86.2 | 13.8 | – | 15. Standardisation and tailoring | Ess. |
15. Innovation | 6.2 (1.7) | 44.2 | 48.8 | 7.0 | Merged with #14 | ||||||
16. Materials | 7.2 (1.3) | 72.1 | 27.9 | – | 17. Materials | 6.8 (1.6) | 58.6 | 37.9 | 3.5 | 16. Materials | Suppl. |
17. Timing (when) | 8.0 (1.4) | 83.3 | 16.7 | – | 18. Timing (when) | 7.9 (1.4) | 79.3 | 20.7 | – | 17. Timing (when) | Ess.c
|
18. Setting (where) | 8.5 (1.0) | 92.9 | 7.1 | – | 19. Setting (where) | 8.6 (0.8) | 96.6 | 3.5 | – | 18. Setting (where) | Ess. |
19. Dose and intensity (how much) | 8.4 (1.1) | 92.9 | 7.1 | – | 20. Dose and intensity (how much) | 8 (1.2) | 86.2 | 13.8 | – | 19. Dose and intensity (how much) | Ess. |
20. Provider characteristics (by whom) | 7.3 (1.5) | 71.4 | 28.6 | – | 21. Provider/staff characteristics (by whom) | 7.3 (1.4) | 72.4 | 27.6 | – | 20. Provider/staff characteristics (by whom) | Suppl. |
21. Provider/staff training | 7.3 (1.6) | 69.1 | 31 | – | 22. Provider/staff training | 7.1 (1.7) | 65.5 | 31.0 | 3.5 | 21. Provider/staff training | Suppl. |
22. Provider reflexivity | 6.4 (1.9) | 54.8 | 35.7 | 9.5 | 23. Provider reflexivity | 6.2 (1.4) | 46.4 | 53.6 | – | 22. Provider reflexivity | Suppl. |
23. Participant recruitment | 7.8 (1.4) | 78.6 | 21.4 | – | 24. Participant recruitment | 7.5 (1.3) | 75.0 | 25.0 | – | 23. Participant recruitment | Suppl. |
24. Participants (who) | 8.2 (1.1) | 92.9 | 7.1 | – | 25. Participants (who) | 8.2 (1.2) | 86.2 | 13.8 | – | 24. Participants (who) | Ess. |
25. Participant preparation | 7.0 (1.6) | 66.7 | 31.0 | 2.3 | 26. Participant preparation | 7.1 (1.4) | 60.7 | 39.3 | – | 25. Participant preparation | Suppl. |
26. Methods used to deliver activities (how) | 8.0 (1.2) | 85.7 | 14.3 | – | 27. Methods used to deliver activities (how) | 8 (1.2) | 82.8 | 17.2 | – | 26. Methods used to deliver activities (how) | Ess. |
27. Efforts to ensure fidelity of participants | 7.7 (1.3) | 82.9 | 17.1 | – | 28. Efforts to increase and sustain participation | 7.6 (1.3) | 79.3 | 20.7 | – | 27. Efforts to increase and sustain participation | Ess.c
|
28. Efforts to ensure fidelity of providers/staff | 7.9 (1.1) | 88.1 | 11.9 | – | 29. Efforts to ensure provider adherence to protocol | 7.7 (1.2) | 85.7 | 14.3 | – | 28. Efforts to ensure provider adherence to protocol | Ess. |
30. Monitoring of the programme implementation | 7.6 (1.2) | 82.1 | 17.9 | – | 29. Monitoring of the programme implementation | Ess. | |||||
29. Acceptability | 7.9 (1.1) | 88.1 | 11.9 | – | 31. Acceptability | 7.3 (1.4) | 72.4 | 27.6 | – | 30. Acceptability | Suppl. |
30. Appropriateness | 7.7 (1.2) | 85.7 | 14.3 | – | 32. Appropriateness | 7.2 (1.2) | 69.0 | 31.0 | – | 31. Appropriateness | Suppl |
31. Feasibility/Practicality | 8.0 (1.1) | 90.5 | 9.5 | – | 33. Feasibility/Practicality | 7.9 (1.0) | 89.7 | 10.3 | – | 32. Feasibility/Practicality | Ess. |
32. Adoption | 8.5 (1.0) | 92.9 | 7.1 | – | 34. Adoption | 8.2 (0.9) | 96.6 | 3.5 | – | 33. Adoption | Ess. |
33. Coverage/Reach | 8.1 (1.1) | 90.5 | 9.5 | – | 35. Coverage/Reach | 8 (1.1) | 93.1 | 6.9 | – | 34. Coverage/Reach | Ess. |
34. Attrition | 8.1 (1.1) | 88.1 | 11.9 | – | 36. Attrition | 7.7 (1.1) | 82.8 | 17.2 | – | 35. Attrition | Ess. |
35. Unexpected end of programme | 8.5 (0.8) | 97.6 | 2.4 | – | 37. Unexpected end of programme | 8.1 (1.2) | 89.3 | 10.7 | – | 36. Unexpected end of programme | Ess, |
36. Reversibility | 7.1 (1.3) | 65.8 | 34.2 | – | 38. Reversibility | 6.5 (1.3) | 46.2 | 53.9 | – | 37. Reversibility | Suppl. |
37. Contamination of activities | 7.3 (1.4) | 69.1 | 30.9 | – | 39. Contamination of activities | 7.3 (1.0) | 74.1 | 25.9 | – |
Merged with #35
| |
38. Fidelity | 8.2 (1.0) | 90.2 | 9.8 | – | 40. Fidelity | 8.4 (0.9) | 92.9 | 7.1 | – | 38. Fidelity | Ess. |
39. Reasons for low fidelity | 7.9 (1.1) | 88.1 | 11.9 | – | 41. Reasons for low fidelity | 7.9 (1.1) | 85.2 | 14.8 | – |
Merged with #38
| |
40. Sustainability | 8.3 (1.1) | 88.1 | 11.9 | – |
Moved to #49
| ||||||
41. Costs of implementation | 8.2 (1.0) | 92.9 | 7.1 | – | 42. Costs of implementation | 8.1 (1.1) | 93.1 | 6.9 | – | 39. Implementation costs/resources | Ess. |
Programme Evaluation
|
–
|
Programme Evaluation
|
–
|
Programme Evaluation
| |||||||
42. Process evaluation methods | 8.5 (0.9) | 95.1 | 4.9 | – | 43. Process evaluation methods | 8.3 (1.0) | 89.7 | 10.3 | – | 40. Process evaluation methods | Ess. |
43. Effect of implementation process on results | 8.3 (1.0) | 95.0 | 5.0 | – | 44. Effect of implementation process on results | 8.1 (1.0) | 89.7 | 10.3 | – | 41. Effect of implementation process on results | Ess. |
44. External events affecting implementation | 8.1 (0.9) | 95.1 | 4.9 | – | 45. Factors affecting implementation | 8.2 (0.9) | 93.1 | 6.9 | – | 42. Factors affecting implementation | Ess. |
45. Ethical considerations | 8.2 (0.9) | 95.1 | 4.9 | – |
Moved to #11
| ||||||
46. Implementation barriers and facilitators | 8.1 (0.9) | 97.6 | 2.4 | – |
Moved to #50
| ||||||
47. Strengths and limitations | 8.2 (0.9) | 95.2 | 4.8 | – |
Moved to #50
| ||||||
48. Outcome evaluation methods | 8.5 (1.2) | 90.5 | 9.5 | – | 46. Outcome evaluation methods | 8.4 (0.8) | 96.6 | 3.5 | – | 43. Outcome evaluation methods | Ess. |
49. Unexpected/negative effects | 8.3 (0.8) | 97.6 | 2.4 | – | 47. Unexpected programme effects | 8.2 (0.9) | 93.1 | 6.9 | – | 44. Unexpected programme effects | Ess. |
50. Differential effects | 8.0 (1.1) | 88.1 | 11.9 | – | 48. Differential effects | 8.2 (1) | 89.7 | 10.3 | – | 45. Differential effects | Ess. |
49. Sustainability | 7.8 (0.9) | 92.6 | 7.4 | – | 46. Sustainability | Ess. | |||||
50. Strengths and limitations (lessons learnt) | 8.4 (0.9) | 96.4 | 3.6 | – | 47. Strengths and limitations (lessons learnt) | Ess. |
Overarching issues raised by the Delphi participants
Theory/logic model. These things aren't meaningful to everyone. More background/rationale rather than an important component of implementation.
Some items were revised accordingly, while others will be further elaborated in a WHO document that will describe and explain how to use the PRS.The definition of Sustainability is a little odd. Would it be clearer to say something along the lines of 'the ability to maintain the programme and its effects over time?
The relevance of Fidelity depends on the circumstances. If one is conducting a study, you need to be able to determine if you have poorer than expected performance whether the problem is one of design or execution. So, in that setting, knowledge of fidelity is important.
There was also feedback related to items missing to describe programme results, for example by asking where “anticipated/expected effects are reported” given that the evaluation domain “doesn't seem to include reporting of the impacts apart from unexpected/negative effects and differential effects”. While the original purpose of the PRS tool was to provide guidance for the reporting of programme development, implementation and evaluation processes rather than results, this issue was further discussed during the technical consultation as described below.I've found it somewhat difficult responding to questions, not having a clear enough sense of what's meant by "program" and, when "reporting" is referred to, who is to be reporting to whom? In places, the language suggests that what's referred to is some kind of pilot effort. In some places, the language suggests some short-term "intervention" (like a training activity). In other places, it sounds more like on-going service delivery.
Technical consultation results
Organization | Country* |
---|---|
African Population and Health Research Center (APHRC) | Kenya |
Aga Khan Development Network | France |
BBC Media Action | United Kingdom |
Gynuity Health Projects | USA |
ICDDR, B | Bangladesh |
Institute of Population and Public Health | Canada |
Institute of Tropical Medicine | Belgium |
International Federation of Red Cross and Red Crescent Societies | Switzerland |
MaiMwana Project | Malawi |
Maternal and Child Survival Program | USA |
Packard Foundation | Pakistan |
Pathfinder International | USA |
Population Council | Kenya |
Population Sciences International (PSI) | Zambia |
Rutgers | Netherlands |
U.S. Agency for International Development (USAID) | USA |
United Nations Population Fund (UNFPA) | Multilateral |
WHO, Alliance for Health Policy and Systems Research | Multilateral |
WHO, Department of Maternal, Newborn, Child and Adolescent Health | Multilateral |
WHO, Department of Reproductive Health and Research | Multilateral |
Piloting results
Organization | Country | SRMNCAH topic covered | Type of programme | Scale | Duration |
---|---|---|---|---|---|
BBC Media Action | Somalia | Maternal, neonatal and child health | Social and behaviour change communication | National | 3 years |
UNFPA | Afghanistan | Reproductive, maternal and adolescent health | Prevention, treatment and social integration services | Global (one country selected) | 5 years |
USAID | Jordan | Sexual and reproductive health | Service delivery (private and non-governmental partnerships to expand family planning access) | National | 5 years |
Rutgers | Kenya | Sexual and reproductive health | Service and information delivery and uptake related to sexual and reproductive health | Global (one country selected) | 5 years |
I also found it hard not to want to give a qualitative assessment of the information provided on each item, i.e. where there was some information reported on an item but I didn’t feel the reporting was complete/of a high standard. Guidance on how to approach partial/weak reporting on certain items when completing the tool would be helpful.
This issue was further discussed during the virtual meeting during which the reviewers agreed that larger programmes might need to be broken down into smaller components (e.g. country levels, or specific topics) as part of the reporting process.If this list will become the norm of reporting, it means the reports will remain many and big for such a large programme, whereas this is not always considered useful. Currently, there is even a tendency from our donor to require smaller reports, mostly based on reflection. Therefore, it should be clear how reports based on this PRS tool will be used.
PRS version 1.0
The PRS is a tool that can be used for reporting on the planning, implementation and evaluation of SRMNCAH programmes. The PRS can be used throughout the programme lifecycle, guiding not only the reporting of processes and outcomes but also the programme design and development. Instructions for using the PRS • For each reporting item, provide the source and page number where the information can be located. • If the information provided for an item is deemed insufficient, state “not reported”. • While users of the PRS should consider the relevance of all items, some items may not be applicable to the programme or the specific report. If an item is irrelevant or beyond the scope of the programme, indicate “N/A” • Larger programmes may need to break their reporting into more specific components and topics. | ||
Section and item name | Item description | Reported (source and page) Not reported N/A |
Programme Overview |
Why was the programme started and what did it expect to achieve?
| |
1. Rationale and objectives | a. Programme rationale, i.e. why the programme was initiated (nature and significance of the issue or problem being addressed). | |
b. Goals and objectives. | ||
c. Anticipated short- and long-term effects of the programme at different levels (e.g. individual, household, facility, organization, community, society). | ||
2. Start and end date | a. Planned start and end date of the programme. | |
b. Delays and/or unexpected end of the programme along with reasons why. | ||
3. Setting and Context | a. Where the programme took place, e.g. country name(s), specific locations, urban/rural environments. | |
b. Overview of the context (e.g. political, historical, sociocultural, socioeconomic, ethical, legal, health system) pertinent to the programme. | ||
4. Stakeholders | a. Programme target population (key sociodemographic characteristics e.g. age, gender, ethnicity, education level) | |
b. Implementing organization(s). | ||
c. Partners and other stakeholders (e.g. local authorities, community leaders). | ||
d. How the different stakeholders were involved in programme development and/or implementation. | ||
5. Funding source(s) | Name of programme donor/funding source(s). | |
6. Theory of change and/or logic model | Theory of change, assumptions, and/or logic model framework underlying the programme, with details for how this guided the programme design, implementation and evaluation plans. | |
7. Human rights perspectives | a. If and how gender, equity, rights and ethical considerations were integrated into the programme. | |
b. If and how an accountability framework was adapted to define the programme’s commitments and how it would be accountable for these commitments. | ||
Programme Components and Implementation |
What did the programme do and how?
| |
8. Programme planning | How activities were decided upon and why (e.g. based on results of a situational or stakeholder analysis, identification of current gaps and needs in programming, or criteria such as the evidence-base, scalability, sustainability of activities). | |
9. Piloting | Piloting of the programme activities elsewhere or within the programme, and if so how, when, where, by whom and with what results. | |
10. Components/Activities (Please repeat for each component) | Detailed description of the core programme components/activities, including: • What was done • How (implementation methods/delivery processes/approaches). • When (frequency, intensity, duration). • By whom (characteristics, skills, training and responsibilities of implementing personnel (i.e. staff, providers, volunteers). • For whom (target population for each activity). • Support materials used and where these can be accessed. | |
11. Quality assurance mechanisms | a. Mechanisms used to ensure the quality in the implementation of activities (e.g. supervision and support of personnel, refresher trainings, product quality checks). | |
b. Efforts used to increase and sustain participation of stakeholders (e.g. incentives). | ||
Monitoring of Implementation |
How did the programme keep track of what was done?
| |
12. Monitoring mechanisms | How the programme implementation process was monitored, including the collection and analysis of indicators to identify problems/solutions. | |
13. Coverage/Reach and Drop-out | a. Uptake (utilization) each programme activity reported by key sociodemographics characteristics. | |
b. Coverage of the programme activities, including differential reach in or outside of the target population. | ||
c. Non-participation and dropout among the target populations, along with key sociodemographics and reasons for why. | ||
14. Adaptations | a. Whether the programme was delivered as intended, e.g. discrepancies between programme design vs. the actual implementation of components, degree of match between programme content and theory of change. | |
b. On-going adaptation of the programme activities to better fit the context, and the fidelity to the activity plan. | ||
15. Acceptability | Acceptability of the programme among stakeholders, e.g. assessment of whether the programme was considered to be reasonable and relevant. | |
16. Feasibility | Assessment of the feasibility of the programme, e.g., the extent to which it could be carried out in the particular context or by the specific organization. | |
17. Factors affecting implementation | Description of key barriers and facilitators to programme implementation, including contextual factors (e.g. social, political, economic, health systems). | |
Evaluation and Results |
How was the programme evaluated, and what were the findings?
| |
18. Evaluation | a. Type of evaluation(s) conducted (e.g. process evaluation and/or outcome evaluation, quantitative or qualitative). | |
b. Evaluation methods. How, when (timing and phases e.g. baseline, midline, end line) and by who the programme was evaluateda. | ||
19. Results | a. Description of the programme results (key process, output, outcome indicators), differentiating between short/mid/long-term effects. | |
b. Whether the programme effects differed across key sociodemographic characteristics and/or geographical areas. | ||
c. Whether the programme had unexpected effects (beyond what was anticipated in the design) on the target population, health services and/or the communities | ||
20. Costs | a. Summary of the required resources for implementation (e.g., financial, time, human resources, materials, administration) | |
b. If and how a cost analysis or cost-effectiveness analysis was conducted. | ||
Synthesis |
What are the key implications?
| |
21. Lessons learnt | Appraised weaknesses and strengths of the programme, what worked well and what can be improved. | |
22. Sustainability | Reflections on the sustainability of the programme over time, e.g. the expected ability to maintain the programme activities, engagement of stakeholders, outcomes achieved, effects, partnerships. | |
23. Scalability | Description of the scale-up of all or some programmes activities, or any plans for scale-up. | |
24. Possibilities for implementation in other settings | Reflections on the context-dependence of the programme and (and with what degree of effort) it could be implemented in/adapted to other settings. | |
Additional information (optional) | References and/or links to additional sources of information in relation to the programme. | |
Any additional comments related to the items reported above |