Background
Is Research Working for You? A Self-Assessment Tool and Discussion Guide for Health Services Management and Policy Organisation
|
Developed by the Canadian Health Services Research Foundation (CHSRF) in 1999 to examine and facilitate discussion around the capacity of health service management and policy organisations to use research evidence in making decisions [6, 9]. Focuses on four areas of assessment: acquire, assess, adapt and apply, with questions related to these ‘four As’ and a discussion guide for the participants to discuss research utilisation in the organisation in a group setting. The tool was validated with respondents in Canada [10] and in Colombia, Argentina, Mexico and Georgia [4], and results suggest that it was useful to understand organisational capacity; however, there were questions about how to best apply the discussion portion given how useful it could be but also subject to imbalanced power dynamics among participants in hierarchical settings. |
Key limitations include (1) strict focus at the organisation level, with little to no assessment of individual or systems capacities and how they influence the uptake of research evidence, and (2) despite application across country settings, government participation appears to be limited. |
SUPPORT Tools |
The Supporting Policy Relevant Reviews and Trials (SUPPORT) project developed a set of tools for increasing well-informed and evidence-informed decision-making targeting primarily policymakers, non-governmental organisations and civil society groups, both in low- and middle-income and high-income country settings [11]. The tools address four broad areas related to policymaking: “1) supporting evidence-informed policymaking, 2) identifying needs for research evidence, 3) finding and assessing evidence, 4) going from research evidence to decisions” [11]. As part of the SUPPORT Tools, a specific tool for organisational capacity to support the use of research evidence to inform decisions was also developed [8]. The tool consists of seven sections that assess: |
• Organisational culture and values to support the use of research evidence to inform decisions |
• Setting priorities for obtaining research evidence |
• Obtaining research evidence |
• Assessing the quality and applicability of research evidence and interpreting the results |
• Using research evidence to inform recommendations and decisions |
• Monitoring and evaluating policies and programmes |
• Supporting continuing professional development |
It is not clear whether the SUPPORT Tools have been formally tested for reliability and validity, or whether there is a standard method for involving particular types of participants to obtain more standard measures and benchmarks. For the specific tool on organisational capacity, the focus is on the organisation with no specific assessment of individual or systems capacities. |
Data Demand and Information Use (DDIU) in the Health Sector
|
Developed by MEASURE Evaluation focusing on generating and collecting health information data with a view towards informing policymaking [7]. The ‘Checklist for DDIU Assessment’ is not a fixed instrument but provides guidance on assessing technical, organisational and behavioural/individual constraints that can affect the demand and supply of data. The approach is not prescriptive and specific, but asks broader questions about potential barriers to collecting, sharing and using data. While DDIU takes on individual, organisational and systems capacities, its focus is on generation and use of data rather than research evidence. |
Other Instrument
|
Boyko et al. [5] developed an instrument based on the theory of planned behaviour to evaluate the intention of policymakers to use research. The instrument was tested with Canadian policymakers and stakeholders who had participated in deliberative dialogues about relevant topics, and was found to be reliable. However, due to the small sample size, the instrument’s validity could not be assessed. This instrument is focused almost exclusively at the individual level, with assessments of individual attitudes and expectations towards use research. Additionally, it is not clear whether this instrument would be applicable in other settings, especially in low- and middle-income countries. |
Conceptual framework
Peer-reviewed journals | Economic development databases |
---|---|
Health Policy and Planning | 3ie |
Health Research Policy and Systems | EuropeAid |
Implementation Science | GIZ |
Milbank Quarterly | OECD |
Public Administration Review | UKaid |
Social Science and Medicine | USAID |
WHO Bulletin |
-
Value:a.Prioritise research utilisationb.Institutionalised in organisational culture
-
Expectation:a.Structure in place to (dis)incentivise research utilisation, including monitoring and evaluation to ensure that research is incorporated in decisionsb.Peer expectations and influence
-
Infrastructure:a.Investment to ensure research utilisation takes place, including funding, staff time, technical resources (hardware, software, IT support), facilitiesb.Access to experts when necessaryc.Skills development/investment through trainings and mentoring
-
Process:a.Clear processes for research evidence to be introduced into decision-making processes, be shared and discussed, including discussion fora (ongoing and for specific issues) and staff participationb.Communication processes around research utilisation
-
Issue recognition: structures and processes whereby actors outside of the MoH draw attention to existing or emergent issues
-
Consultation: formal and informal mechanisms for consultation between MoH actors and outside actors around agenda items that are driven by research evidence
-
Outside influence: processes whereby MoH attempts to influence outside audiences regarding their behaviour, priorities and decisions, specifically those driven by research evidence
Methods
Instrument development
Instrument components
Study design
Sampling framework
Countries | WHO Region | Economic statusa
| Population (Thousands) |
---|---|---|---|
Fiji | Western Pacific | Lower middle income | 868 |
Moldova | Europe | Lower middle income | 3559 |
South Africa | Africa | Upper middle income | 50,856 |
Zambia | Africa | Lower middle income | 13,747 |
Lebanon | Eastern Mediterranean | Upper middle income | 4259 |
Bangladesh | South-East Asia | Low income | 150,493 |
India | South-East Asia | Lower middle income | 1,241,491 |
Pakistan | Eastern Mediterranean | Lower middle income | 176,745 |
MoH Unit | Type of respondentsa
| Number per unit |
---|---|---|
Ministerial level | Minister, Vice/Deputy Minister, Administrative head (e.g. Principal Secretary) and Head of Health Service (e.g. Director General) or their equivalents | 3–5 persons per MoH, depending on organisation structure |
Policy and planning unit (or equivalent) | Head and Deputy Head | 2 respondents |
Research unit (if one exists) | Head and Deputy Head | 2 respondents |
Monitoring and Evaluation unit | Head and Deputy Head | 2 respondents |
Public health programmes | Head and Deputy Head | 2 respondents |
Specific health programmes, if multiple programmes (e.g. maternal/child health, nutrition, HIV)b
| Head of programme (random selection of 5 respondents from heads of such programmes) | 5 respondents |
Hospital services (or health services) | Head and Deputy Head | 2 respondents |
International relations | Head of unit | 1 respondent |
Regional/district units | Head of unit (random selection of 10 respondents from heads of such units) | 10 respondents |
-
MoH officials at the central level departments and divisions (not regional and sub-regional levels);
-
MoH officials who have responsibilities that are directly health-related;
-
MoH officials operating under the political leadership of the MoH.
Data collection
Data analysis
Validity
Reliability
Results
Study sample
Countries | Initial respondents | Retest |
---|---|---|
Fiji | 24 | 4 |
Moldova | 25 | 4 |
South Africa | 20 | 0 |
Zambia | 26 | 0 |
Lebanon | 26 | 4 |
Bangladesh | 24 | 4 |
India | 30 | 4 |
Pakistan | 28 | 4 |
Total | 203 | 24 |
Preliminary analysis
EFA results
Validity and reliability
Original item no | Item text | F1a
| F2a
|
---|---|---|---|
1 | Using research evidence is a priority in the MoH | 0.883 | |
2 | Leadership in the MoH supports evidence-informed data | 0.951 | |
3 | Decision-makers in the MoH give consideration to any recommendations based on research evidence | 0.694 | |
4 | There is a transparent process for how research evidence is used in decisions at the MoH | 0.678 | |
8 | Has the MoH conducted any activities to promote the use of research evidence in the last year? | 0.476 | |
10 | The MoH has a process to check regularly whether I use research evidence in my work | 0.607 | |
14a | Our staff has enough time to evaluate research evidence | 0.736 | |
14c | Our staff has enough resources to evaluate research evidence | 0.684 | |
15a | Our staff has enough time to compare what the MoH does to what the research evidence says | 0.924 | |
15c | Our staff has enough resources to compare what the MoH does to what the research evidence says | 0.791 | |
16a | Our staff has enough time to link research evidence to key issues facing decision-makers | 0.944 | |
16c | Our staff has enough resources to link research evidence to key issues facing decision-makers | 0.972 | |
17a | Our staff has enough time to provide recommendations based on research evidence to decision-makers | 0.860 | |
17c | Our staff has enough resources to provide recommendations based on research evidence to decision-makers | 0.961 | |
18a | Who does this for the MoH?… – Search for and retrieve research evidence – MoH staff | 0.782 | |
19a | Who does this for the MoH?… – Interpret research evidence – MoH staff | 0.925 | |
20a | Who does this for the MoH?… – Synthesise into one document all the relevant research evidence, information and analyses for a specific issue – MoH staff | 0.829 | |
21a | Who does this for the MoH?… – Compare what the MoH does to what the research evidence says – MoH staff | 0.831 | |
22a | Who does this for the MoH?… – Link research evidence to key issues facing decision-makers – MoH staff | 0.823 | |
23a | Who does this for the MoH?… –Provide recommendations based on research evidence to decision-makers – MoH staff | 0.815 | |
26 | The MoH gets involved with researchers as partners in decision-making | 0.627 | |
31 | The MoH has a good process to advocate its priorities based on research evidence to the public, such as to promote behaviour change | 0.749 | |
32 | The MoH has a good process to advocate its priorities based on research evidence to health workers, such as to promote changes in clinical practice | 0.844 | |
33 | The MoH has a good process to advocate its priorities based on research evidence to other ministries, such as to justify the costs of health interventions | 0.600 | |
34 | The MoH has a good process to advocate its priorities based on research evidence to professional organisations, such as to promote new roles for different health workers | 0.661 | |
35 | The current policy environment is supportive of the MoH using research evidence for its decisions | 0.607 | |
36 | The current government is supportive of the MoH using research evidence for its decisions | 0.557 | |
37 | Stakeholders outside the MoH actively engage the MoH to contribute research evidence to inform decisions | 0.586 | |
50 | Our unit has regular access to a computer for acquiring and analysing research evidence | 0.631 | |
51 | Our unit has regular access to the Internet at work for accessing research evidence online | 0.706 |
-
Organisational capabilities (Factor 1): processes and practices that reflect MoH commitment to using research evidence in its decisions and for influencing others outside the MoH, including allocation of staff to take specific steps in its use.
-
Individual capacities (Factor 2): access to sufficient resources necessary for MoH staff to assess evidence and provide recommendations.
Conceptual framework step | Organisational capabilities | Individual capacity | Total |
---|---|---|---|
Example | Example | ||
Recognition | Using research evidence is a priority for the MoH | – | 6 (6 ORG) |
Acquisition | MoH staff search for and retrieve research evidence | Our unit has regular access to the Internet at work for accessing research evidence online | 2 (1 ORG+1 IND) |
Cognition | – | Our staff has enough time to evaluate research evidence | 3 (3 IND) |
Discussion | The MoH gets involved with researchers as partners in decision-making | – | 1 (1 ORG) |
Reference | MoH staff synthesise all the relevant research evidence, information and analyses for a specific issue | Our staff has enough resources to compare what the MoH does to what the research evidence says | 5 (2 ORG+3 IND) |
Adaptation | There is a transparent process for how research evidence is used in decisions at the MoH | – | 2 (2 ORG) |
Influence | The MoH has a good process to advocate its priorities based on research evidence to the public, health workers, etc. | Our staff has enough resources to provide recommendations based on research evidence to decision-makers | 11 (7 ORG+4 IND) |
Country | Mean score | Standardised Z-score | Total respondents |
---|---|---|---|
South Africa | 24.2 | 0.72 | 20 |
Zambia | 21.8 | 0.35 | 26 |
India | 20.8 | 0.19 | 30 |
Fiji | 20.4 | 0.12 | 25 |
Moldova | 20.4 | 0.12 | 20 |
Lebanon | 17.6 | –0.32 | 27 |
Pakistan | 16.5 | –0.48 | 28 |
Bangladesh | 16.3 | –0.53 | 24 |
Factor 1 | Factor 2 | ||
---|---|---|---|
Country | Mean | Country | Mean |
South Africa | 18.4 | Pakistan | 6.0 |
Zambia | 16.4 | South Africa | 5.9 |
Fiji | 16.1 | Zambia | 5.4 |
India | 16.0 | Moldova | 5.2 |
Moldova | 15.2 | India | 4.9 |
Lebanon | 13.3 | Fiji | 4.4 |
Bangladesh | 12.1 | Lebanon | 4.3 |
Pakistan | 10.5 | Bangladesh | 4.1 |