Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review

An estimated 85% of investment in health and biomedical research is wasted every year due to redundancies, failure to establish priorities based on needs of stakeholders (particularly end-users of knowledge), poorly designed research methods, and incomplete reporting of study results, leading to billions of dollars lost globally [1‐3]. Stakeholders include those who are affected by, have an interest or stake in research [4], while knowledge users are subgroup of stakeholders who are likely to use research findings to make informed decisions about health systems and practices [5]. Knowledge users include but are not limited to patients and their informal caregivers or surrogate decision-makers (e.g., family, friends), healthcare providers (e.g., physicians, occupational therapists), policy-makers (e.g., Minister of Health, health officer), health system managers (e.g., hospital administrators, health unit managers), and policy analysts.

The overarching goal of knowledge user engagement in health research is to co-produce knowledge that is relevant and useful to those making real-world health decisions [6]. Early engagement of knowledge users in the research process may help establish research priorities and increase relevance of findings [7, 8]. To facilitate the use of research in decision-making, health systems and research funders are encouraging the engagement of knowledge users and other stakeholders in research [9].

Knowledge synthesis, such as a systematic review or a scoping review, is particularly useful for decision-makers because these research products provide a summary of the expansive evidence on a particular topic to inform decisions based on the totality of evidence [10, 11]. Co-production of evidence whereby researchers and knowledge users work together to conduct research increases the policy-relevance of research questions and fosters integration of findings into policy and practice [12‐14]. However, the opportunities and approaches to engaging a wide range of knowledge users remain largely unexplored. Evidence is required to guide the process of engaging knowledge users in knowledge synthesis to identify engagement approaches that are effective, efficient, and meaningful. In addition, co-production of research by researchers and knowledge users requires additional time and funding, and it is imperative that the limited resources available for health research are used appropriately.

We undertook a scoping review to map the literature on engaging knowledge users in the knowledge synthesis process. Engagement of policy-makers, policy analysts, and health system managers were of particular interest, as these knowledge users are increasingly commissioning knowledge synthesis research products to meet their decision-making needs. The research questions (RQs) for our scoping review are provided below and outlined in our published protocol [15]:

(RQ1) In what context were policy-makers, health system managers, and policy analysts engaged (e.g., health system setting, high-income countries (HICs) versus low and middle-income countries (LMICs))?

(RQ2) What strategies exist to engage policy-makers, health system managers, and policy analysts in the knowledge synthesis process?

(RQ3) In studies describing strategies for engaging policy-makers, health system managers, and policy analysts, what outcomes do they measure to evaluate engagement mechanisms (e.g., attitudes, beliefs, knowledge) and what are the results (e.g. benefits, unintended consequences)?

(RQ4) What are the barriers and facilitators in engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process?

The included documents were predominantly conducted at the level of a national healthcare system and focused on health services delivery in the context of high-income countries. We did not identify any distinguishing trends in engagement when we compared knowledge user engagement across country income groups and other contextual factors. We did not identify differences in results over time or across settings, for phases of engagement, or how the engagement was conducted. This might be because the practice of engaging knowledge users in knowledge synthesis is still relatively new.

Knowledge users were most commonly engaged as key informants who were engaged through structured meetings or workshops and surveys, focus groups or interviews. Knowledge users were engaged only once during the knowledge synthesis process in two-fifths of the documents, twice in one-quarter of the documents, three times in one-tenth of the documents, and across all four stages in nearly one-quarter of the documents. None of the documents conducted a formal evaluation of engagement and measurement tools specific to engagement were not identified. Sixteen barriers and facilitators were identified. The most common facilitator was content expertise/awareness of the knowledge user, whereas the most common barrier was lack of time or opportunity for engagement.

There are numerous perceived benefits to engaging policy-makers, policy analysts, and health system managers in knowledge synthesis. Examples include more comprehensive literature searches, improved rigor of knowledge synthesis findings, greater clarity of results [59] as well as greater relevance, uptake, and usefulness of results. However, the results of our scoping review suggest that very little research has been conducted in this area. The research that has been conducted is purely descriptive in nature and a formal evaluation of engagement approaches and outcomes was not identified. A future study could evaluate engagement using a variety of methods, such as documenting how the knowledge synthesis process and results were modified after engagement or testing engagement at different points of the knowledge synthesis process to see how engagement influences research impact.

We identified several factors that may enhance engagement of knowledge users in knowledge synthesis process that are within the researcher’s control, for example, engaging knowledge users before the synthesis begins; clearly outlining expectations regarding stakeholder’s role and time commitment; identifying funding opportunities to work closely with policy-makers; providing time for question and answer opportunities; conducting ice breaker activities; providing materials in advance of meetings; considering knowledge user comments as being equal to those received from researchers; being sensitive to knowledge user’s time; presenting results to knowledge users; and using a neutral facilitator. As none of these have been formally evaluated, we cannot comment on the effectiveness of any of these approaches. As such, the type and intensity of engagement should be meaningful and tailored to available resources, including time and funding. To better define knowledge user engagement in knowledge synthesis, researchers should discretely identify the desired benefits and impacts and effectiveness of engagement and develop systematic and reproducible methods and indicators for formal evaluation.

There were four main phases when engagement took place, including conception and design of research, search and data collection, data synthesis and interpretation, and knowledge dissemination and application.

Knowledge users were most often engaged as key informants across the four stages of the knowledge synthesis process to obtain advice, feedback, and opinions. However, there is increasing interest globally in co-design and co-development of research with knowledge users and using research to inform public policy [60, 61]. Co-creation of science is gaining momentum to integrate research and decision-making cycles and incorporate knowledge generation in complex policy planning and implementation, ultimately enhancing the usability and impact of research [12, 62, 63]. It will be important to test the utility of the co-design and co-creation of knowledge synthesis in the future.

Two conceptual frameworks were identified that provided a structure and mechanism to facilitate knowledge user engagement in knowledge synthesis. These were the 7Ps of Stakeholder Engagement and Six Stages of Research framework [58] and a conceptual framework on the models and mechanisms for engaging policy-makers in systematic reviews that focus on health policy and systems research [47]. An additional framework can also be used: the online survey patient and public engagement questionnaire (PPEQ) [64]. Members of our research team are currently conducting a study to test the level of engagement of knowledge users in a systematic review using the PPEQ [65], which will provide clarity to the field.

Two additional frameworks [66, 67] to engage stakeholders in knowledge synthesis were published after the literature search date and completion of our scoping review. Haddaway and colleagues [66] discussed a framework including approaches for engaging stakeholders during systematic reviews in the field of environmental management. Land and colleagues [67] described an empirically tested five-step approach for stakeholder engagement in prioritization and planning of environmental evidence syntheses that the Mistra Council for Evidence-based Environmental Management has been using. These frameworks may also be of relevance to knowledge synthesis within health and should be examined more closely in the future.

The strengths of our scoping review include a comprehensive literature search of multiple electronic databases as well as unpublished sources. We also followed the rigorous scoping review methods suggested by the Joanna Briggs Institute. We engaged with the principal knowledge user (EVL) throughout the review process who provided input in our research questions, review protocol, eligibility criteria of papers, reviewed this manuscript, and helped interpret our findings. In terms of dissemination plans, in addition to publication of this manuscript, we will prepare a1-page policy brief which will be made available on our website (https://​knowledgetransla​tion.​net/​) and present at international conferences. Team members will also use their networks to encourage broad dissemination of results.

There are limitations to our scoping review process. To increase feasibility, we limited inclusion to documents made available in the past 20 years. However, this is likely not a substantial limitation, as all of the included documents were made available in the past 10 years. We also limited inclusion to studies written in English, which may have resulted in the exclusion of eligible studies from LMIC settings for RQ1. Given the large number of documents included, the data were abstracted by one reviewer and verified by a second reviewer. However, the data are likely valid, as a pilot-test was conducted prior to embarking on data abstraction with the entire team and a second reviewer who is an experienced research coordinator on the team verified all data. Often the included documents did not distinguish between stakeholders (i.e., those who are affected by or have an interest or stake in research [4]) and knowledge users (i.e., a subgroup of stakeholders who are likely to use research findings to make informed decisions about health systems and practices [5]), which is likely due to inconsistent use of the terms in the literature. As such, our results are likely applicable to both stakeholder and knowledge user participants. Furthermore, the reporting of knowledge user engagement methods varied considerably in their completeness across the literature, and as such, our data are limited by the details described in the literature. For example, most papers described steps to engage knowledge users but did not provide details on non-response or unsuccessful engagement.

Engaging policy-makers, policy analysts, and health system managers in knowledge synthesis usually occurs at the beginning or end of the knowledge synthesis process. However, ongoing engagement throughout the review process may lead to more relevant and user-friendly results. The type and intensity of engagement should be meaningful and tailored to available resources, including time and funding. Researchers should document and evaluate engagement activities in knowledge synthesis on an ongoing basis. It is important to document and test knowledge user engagement in knowledge synthesis in the future, to advance the field.