Background
Methods
Inclusion and exclusion criteria
Domain | Description | Conditions conducive to KT |
---|---|---|
General climate | Explores the extent to which funders and other stakeholders (i.e. universities, researchers and users of research) value and/or support efforts to link research to action | • Funder mandate to support KT activities • Universities and research institutions consider KT in their tenure/promotion process • Researchers value the promotion and use of evidence • Research users value evidence and its use |
Production of research | Explores the extent to which research is produced in a way that is aligned with policy priorities; this domain suggests that researchers engage in priority-setting to ensure that users’ needs are identified and then develop scoping reviews, systematic reviews and single studies to address these needs | • Funders engage potential evidence users to identify policy needs and priorities • Funders and ethics review boards value systematic reviews |
Push efforts | Includes efforts to create action based on messages arising from research; this domain includes developing actionable messages for end-users, disseminating research results and building research capacity to conduct these strategies | • Regularly identify actionable messages from systematic reviews • Develop user-friendly messages from evidence • Work with credible messengers for each user group • Use research-informed strategies to encourage and support action based on the messages • Evaluate their KT efforts |
Pull efforts | Includes efforts by end-users to acquire, assess, adapt and apply research; this domain includes capacity-building efforts for policy-makers to use research | • Engage in the self-assessment of abilities to acquire, assess, adapt and apply research • Develop structures and processes to help them use and promote research |
Exchange efforts | Includes efforts to develop partnerships between researchers and users, and the extent to which the partnerships jointly address relevant questions; this domain includes KT tools that facilitate exchange like policy dialogues | • Ongoing relationship building to develop an understanding of the cultural and other differences between the contexts of researchers and research users • Creations of meaningful partnerships where the roles and expertise of both researchers and research users are recognised |
Integrated effortsa | This is not a domain in the Lavis et al. framework; we have adapted the original domain ‘efforts to facilitate user pull’ and expanded this to integrated efforts, which we define as using brokering to facilitating push, pull and exchange efforts; recognising that the push, user pull and exchange efforts are not mutually exclusive, integrated efforts aim to institutionalise KT infrastructure to facilitate a combination of activities encompassed in all three of these models of KT [17] | • Multidisciplinary leadership, comprised of researchers, funders and policy-makers, that are guided by a clear goal • Regular priority-setting processes • Facilitate and conduct push efforts using actionable messages • Engage in a variety of efforts to facilitate KT (e.g. one-stop shopping resource of relevant and quality systematic reviews, rapid-response unit to provide evidence summaries) • Facilitate exchanges between research producers and research users (e.g. policy dialogues) [17] • Develop and sustain KT infrastructure by institutionalising organisational knowledge brokers [17] |
Evaluation | Explores the extent to which stakeholders participate in evaluating their KT activities; this domain also concerns assessing sustainability of KT initiatives | • Funding is allotted for evaluation of KT efforts • Funders, researchers, intermediary groups and user groups participate in rigorous evaluations of efforts to link research to action |
Capacity-buildinga | Includes efforts to improve stakeholder capacity in any of the KT models (push, pull and exchange) and explores the extent to which these activities were successful in skills development; capacity-building is an element in many of the Lavis et al. domains but is not a domain of the original framework; it was adapted for use in this study | • Researchers partake in continuing education programmes to develop KT skills (e.g. systematic reviews, priority-setting)b • Researchers partake in skills-development programmes to build capacity for developing and executing push effortsc • Research users partake in skill-development programmes to build capacity for acquiring, assessing, adapting and applying researchd • Capacity-building programmes to support researchers and research users to engage in mutually beneficial partnershipse • Stakeholders (e.g. funders, researchers or intermediary groups) partake in skills-development programmes to build capacity to develop and execute efforts to facilitate KTf |
Results
Study characteristics
Indicators organised by domain
1. General climate | |
Output indicators | |
1.1 Number of activities identified 1.2 Availability of synthesised and packaged evidence 1.3 The organisation has the skills, structures, processes and a culture to promote and use research findings in decision-makinga 1.4 Feedback on context/culture 1.5 Facilitators, barriers, lessons learnt | |
Outcome indicators | |
1.6 Increased demand or value of KT products or knowledge from policy-makers 1.7 Number of times evidence is mentioned in policy/parliamentary discussions 1.8 Increased awareness of importance of EIP initiatives 1.9 Changes in government allocated funding | |
2. Production of research | |
Output indicators | |
2.1. Number of peer-reviewed journal articles 2.2. Citations per article 2.3. Citation of research results by other researchers 2.4. Journal impact factor 2.5. Number of projects per research approach 2.6. Funds invested per project 2.7. Project duration 2.8. Number of projects liaising with users 2.9. Number of projects that led to subsequent research 2.10. Researcher feedback on project alignment with priorities 2.11. Mean score of scientific accuracy | 2.12. Mean score of readability 2.13. Mean score of usability 2.14. Mean score of ease of access 2.15. Applicability of research for decision making 2.16. Developed priority report (i.e. research agenda, list of priorities, country assessment) 2.17. Revision with stakeholders 2.18. Feedback on support and/or awareness 2.19. Feedback on priority development 2.20. Feedback on priorities |
Outcome indicators | |
2.21. Changes in policies or programmes consistent with evidence produced 2.22. Policy-makers, stakeholders and researchers report that relevant and understandable health research evidence is more readily available and cite this research evidence in media | |
3. KT activities: push efforts | |
Output indicators | |
3.1. Number of downloads 3.2. Number of page visits (total and unique) 3.3. Number of countries visiting the website 3.4. Number of page views per visit 3.5. Number of requests for materials 3.6. Extent of media exposure 3.7. Referrals made to distributed materials 3.8. Number of materials distributed 3.9. Transmitted to relevant stakeholder (discussed at policy dialogues, dissemination workshops) 3.10. Disseminated materials are read and understood 3.11. Efforts have been made to adopt the disseminated knowledge 3.12. Platform survey responses 3.13. Usage analytics of promotional products 3.14. Research is presented to decision-makers in a useful waya 3.15. Multiple formats of written and/or other forms of presentation (e.g. newsletter, website summary, interim report, oral presentation) 3.16. Presentation formats include layman’s terms and recommendations 3.17. Where appropriate, presentation formats are concise (e.g. less than two pages) | 3.18. Users contacted researchers to discuss results 3.19. Relevant documents disseminated in hardcopy 3.20. Website or online evidence database is established 3.21. Number of dissemination workshops 3.22. Percentage of grantees presenting at conferences 3.23. Percentage of grantees submitting work for publication 3.24. Percentage of grantees with published research at time of review 3.25. Feedback from grantees on competence and opportunities for dissemination 3.26. Number/amount of grant (applications) 3.27. Number of researcher internships 3.28. Number of trainees publishing research 3.29. Feedback on improved quality of research results 3.30. Percentage of research applications headed by a national 3.31. Increased interest by young nationals in research 3.32. Establishment of a PhD programme 3.33. Number of projects supported |
Outcome indicators | |
3.34. Number of project findings used/expected to be used in policy 3.35. Number of projects leading to/expecting to change behaviour 3.36. Increase in inquiries and applications 3.37. Phasing out of external funding | |
4. KT activities: pull efforts | |
Output indicators | |
4.1. Seeking, Engaging with, and Evaluating Research (SEER)a 4.2. Organisational Research Access, Culture and Leadership (ORACLe)a | |
Outcome indicators | |
4.3. Staff Assessment of enGagement with Evidence (SAGE)a | |
5. KT activities: exchange efforts | |
Output indicators | |
5.1. Grants for collaboration 5.2. Research projects are produced with policy-makers 5.3. Disciplinary backgrounds of contributing authors 5.4. Invitations to publish special issues 5.5. Partners views on using research results 5.6. Negotiation occurs during the research process 5.7. Negotiated items are clearly understood by all 5.8. Deciding on objectives together 5.9. Built mutual trust 5.10. Communication tools established 5.11. Sharing of information and responsibility 5.12. Transparency 5.13. Share profits equally 5.14. Build on achievements 5.15. Communication is clear 5.16. Communication is relevant 5.17. Communication is timely 5.18. Communication is respectful 5.19. Density and centrality 5.20. Connectedness of networks 5.21. Partners mention each other 5.22. Partners are flexible about meeting partner’s changing needs and revising research plans and timelines 5.23. Partners understand the limits of each other’s flexibility 5.24. Partners understand research findings, their limits and their implications for Ministry work 5.25. Conflict is dealt with openly, informally and promptly | 5.26. Trust has increased between partners 5.27. Comfort has increased between partners 5.28. Openness has increased between partners 5.29. Partners begin speaking a common language regarding research 5.30. Partners facilitate removal of barriers for each other’s work 5.31. Linkage with partner enhances partner linkage with community/other stakeholders 5.32. There is joint commitment to the research project 5.33. There is an increase in joint activity around the project 5.34. Clear leadership of partnerships 5.35. Team mentality 5.36. Early engagement of members 5.37. Number of members 5.38. Number/percentage of members present at activities 5.39. Level of engagement 5.40. Number/percentage of partners active 5.41. Member affiliation and profession 5.42. Joint meetings occur at most stages of research 5.43. Joint meetings occur to discuss research dissemination and utilisation plans 5.44 Feedback on linkage and exchange mechanisms 5.45. Number of partners involved in KT activities 5.46. Stakeholders involved |
Outcome indicators | |
5.47. Partners are perceived as experts in the research/policy area and are referred to as such to others 5.48. Value of network 5.49. Feedback on awareness and perceptions of network 5.50. Partnerships are built and sustained | |
6. KT activities: integrated efforts | |
Output indicators | |
6.1. Number of KTPs viewing their work as a long-term initiative 6.2. Number of KTPs engaging in priority-setting with stakeholders 6.3. Number of KTPs building capacity for priority-setting 6.4. Number of KTPs producing/in process of KT products (by type, e.g. evidence briefs, clearinghouses, rapid response services, deliberative dialogues, systematic reviews) 6.5. Number of KTPs that built capacity for KT (evidence briefs; deliberative dialogues; accessing, assessing and using research evidence) 6.6. Number of KTPs training research users in KT (systematic reviews, evidence briefs, deliberative dialogues) 6.7. Number of organisations using the products 6.8. Functional website or clearinghouse providing KT resources 6.9. Amount of resources utilised in knowledge brokering activities (e.g. cost, time, materials) 6.10. Number of KT materials 6.11. Products (e.g. website, policy dialogues, evidence briefs) aligned with and address priorities | 6.12. Topic of KT materials 6.13. Number of KT materials translated/available in different languages 6.14. Policy dialogues about high-priority policy issues take place regularly 6.15. Scoring of quality dimensions (mean, standard deviation) 6.16. KT activities regarded as beneficial for bringing together stakeholders and facilitating the development of partnerships |
Outcome indicators | |
6.17. Uptake and/or influence of evidence (reports, policy briefs, recommendations, other) in decision-making 6.18. Financial and organisational support to the KTP | |
7. Evaluation | |
Output indicators | |
7.1. Number of KTPs evaluating KT product(s) quality 7.2. Perception of sustainability (no outcome indicators identified) | |
8. Capacity-building | |
Output indicators | |
8.1. Number of activities 8.2. Type of activity 8.3. Number of people invited 8.4. Number of people attended 8.5. Number of people trained 8.6. Reasons for participation non completion 8.7. Participant occupation 8.8. Participant affiliation 8.9. Participant education level 8.10. Participant gender 8.11. Participant age 8.12. Participant’s number of years in current position 8.13. Participant’s level of policy influence 8.14. Country of participants 8.15. Participant’s experience with evidence-informed policy-making 8.16. Training workshops for policy-makers and researchers are designed and implemented regularly 8.17. Mean programme ratings and feedback 8.18. Intent to return 8.19. Survey response rate 8.20. Percentage increase in pre/post scores of skill abilities (e.g. access evidence, synthesise evidence, policy dialogues, evidence briefs, collaboration etc.) and value of knowledge use | 8.21. Comments in the media reflect capacity changes 8.22. Ability to acquire research 8.23. Increased research capacity 8.24. Change in research/policy-maker relationship 8.25. Comments in the media reflect relationship changes 8.26. Perceived EIP skills/changes in skills (acquire, assess, adapt, apply)a 8.27. Number of participants reporting benefits 8.28. Awareness of key government documents 8.29. Perceived change in skills and confidence to interact with experts 8.30. Perceived impact on current position and/or future career advancementa 8.31. Contribution to decision-making by partners and policy-makers |
Outcome indicators | |
8.32. Feedback on behavioural changes 8.33. Number/percentage of trainees reporting intent to use skills gained |
Measure descriptions of the four validated indicators and tools: | |
• ORACLe Measures: capacity (policies that encourage or mandate the examination of research in policy and programme development; tools, systems, training and programmes to assist with accessing, appraising and generating research). | |
• SEER Measures: capacity (value placed on research, perceived value organisation places on research; confidence in skills and knowledge to access, appraise and generate research; perceived availability of organisational tools, systems, training and programmes to assist with accessing, appraising and generating research); research engagement actions (self-reported extent of accessing, appraising and generating research, and interaction with researchers); research use (self-reported use of research). | |
• SAGE Measures: research engagement actions (accessing research, appraising research (for quality and relevance), generating new research or analysis, and interacting with researchers) and research use (four types of research use are considered: instrumental, tactical, conceptual and imposed) in each policy document and the context in which the policy document was produced, including barriers and facilitators. | |
• Is Research Working for You? Tool This tool was developed in 2009 by the Canadian Foundation for Healthcare Improvement (formerly the Canadian Health Services Research Foundation) and includes 88 items to measure culture for EIP and use of evidence at the individual and/or organisational level. |