Capturing lessons learned from evidence-to-policy initiatives through structured reflection

The framework developed by Lavis et al.[13] was used to guide data collection. The framework identifies four domains for assessing country-level efforts to link research to action: i) the general climate for research use; ii) the production of research that is both highly relevant to and appropriately synthesized for research users; iii) the mix of clusters of activities used to link research to action which include push efforts, efforts to facilitate user-pull, user-pull efforts, and exchange efforts; and iv) the evaluation of efforts to link research to action. The framework also identifies the corresponding elements that are conducive to linking research to action in each domain (Table 3). The elements of this framework are based on potential innovations that are being implemented to link research to action but that require further rigorous evaluation. This framework was previously adapted to assess activities of researchers[14, 15] and health research funding agencies in linking research to action in LMICs[16].

Participants were notified of the objectives of the interviews and were approached for the interview after securing their verbal approval. All interviews were conducted face-to-face. Interviews were tape-recorded only after obtaining the permission of participants. All interview data was treated as confidential and the anonymity of participants was preserved. Care was taken to ensure that comments could not be attributed to a single individual (or KTP) to protect the confidentiality of respondents.

Ten group interviews were conducted with a total of 39 key informants. Group size ranged from two to eight participants per group. Group interviews with as few as two participants were previously used in several qualitative studies[17‐20]. Group interviews lasted for one to two hours on average. In addition, seven individual interviews were conducted and lasted for an average of 45 minutes. Six group interviews were tape-recorded and were transcribed verbatim. Extensive notes were taken for the remaining group interviews and for the individual interviews and they were summarized shortly thereafter.

Interviews with KTP leaders were guided by a semi-structured interview tool developed by the KTPE team. An adapted version of the interview tool was used for interviewing policymakers, stakeholders, and KTPE team members, which allowed them to focus on questions based on their area of expertise. The development of the semi-structured interview tool was guided by the framework adapted from Lavis et al.[13] and involved reviewing several iterations by team members to ensure its robustness. The interviews covered topics on general climate for supporting (or hindering) efforts to link research to action; activities and outputs undertaken by KTPs including priority setting, engaging in push efforts, facilitating user-pull, pull efforts, and exchange efforts; KTP’s evaluation efforts and their perceptions of the results of their activities; facilitators for supporting KTP work, challenges encountered, and lessons learned that can assist in overcoming challenges; and critical factors that can help to ensure the sustainability of KTPs. Interview tools are enclosed in Additional files1 and2.

The document review helped to identify efforts undertaken by KTPs to link research evidence to action and derive key lessons for improving country-level efforts to support EIHP in LMICs. The document review was cross-checked with findings from interviews and observation of deliberations. Material examined included a report summarizing the proceedings of the International Forum on EIHP in LMICs. The report was prepared by the WHO International Forum Organizing Committee and it provides an overview of the Forum including plenaries, workshops, and small group sessions that took place. It is organized around the framework for assessing country-level efforts to link research to action and highlights key messages and lessons learned that emerged during the Forum[21]. The document was reviewed and summarized in a data collection sheet that included the title of the document and the data related to the four domains for assessing efforts to link research to action framework.

Observations of deliberations at the International Forum on EIHP in LMICs were recorded. Materials examined included notes on observation of deliberations and hands-on workshops, and meeting minutes of four plenaries and six sessions from the Forum. Sessions from the Forum were sampled based on their relevance to the work undertaken by these KTPs. Participants in these sessions shared their experiences on the ways in which KTPs can encourage and promote EIHP initiatives and engage different actors in efforts to support EIHP. The observations focused on the climate for and activities of KTPs, facilitators for supporting KTP work, challenges encountered, and lessons learned that can assist in overcoming these challenges. Observations were recorded by three members of the research team. They were then compiled and compared against each other. Observations helped to verify information obtained through the interviews and the document review. Each document was reviewed and summarized in a data collection sheet.

Data analysis was guided by the framework developed by Lavis et al.[13]. Data analysis was conducted by the lead author (FEJ) using a grounded iterative approach, and shared with the other team members (JNL, SC, TP, KAM) during several stages of the analysis as a way to confirm, challenge, and feedback into the further refinement of themes. Preliminary results were also shared with a group of doctoral students studying EIHP to the same effect.

Data from the interviews, document review, and observation of deliberations were holistically analyzed such that data collected from the three sources were gathered using the same data analysis sheet, rather than analyzing data from different sources separately. Thematic analysis was used. Open coding was conducted first, whereby findings were broken into similar concepts and ideas. Axial coding was then conducted, which involves organizing the emerging concepts into themes[22]. Themes were pre-identified based on the study objectives, interview questions, and additional themes emerged during the analysis. Themes were then categorized based on the framework by Lavis et al.[13]. Recurring themes and emerging patterns across respondents from different KTPs and the corresponding countries were then analyzed. Illustrative quotations were identified to support the narrative description of the themes.

The extent of support for and value given to efforts to link research to action was variable among KTPs. Results from the interviews, document review, and observation confirmed that the support provided to KTPs by policymakers and stakeholders as well as the financial support provided by international funders were key for enabling EIHP initiatives (Table 4). The main challenges to supporting EIHP work were the lack of skilled human resources to undertake KT activities and the difficulty in managing multiple roles within KTPs.

Despite efforts by funders, policymakers, and stakeholders to support EIHP, findings suggested that much should be done for building support for EIHP initiatives in LMICs. Increasing the awareness of policymakers and stakeholders on the importance of EIHP, including those outside the health sector, was often highlighted as a requirement for creating a climate that supports KT activities. For example, discussions at the Forum highlighted the importance of mapping institutional structures at the national level to understand the optimal ways and emerging opportunities for targeting policymakers and stakeholders, which would also require that KTPs build their capacity in lobbying and understanding the political context. Additionally, KTP leaders repeatedly urged funders to play a stronger role in supporting KT activities in LMICs mainly through increasing financial support provided to KTPs, strengthening technical capacity of KTPs to undertake KT, as well as providing mentorship opportunities to KTPs and more flexibility with funding (for example in terms of providing KTPs with more time to discuss and explore innovative KT ideas) (Table 4).

Findings revealed that efforts exerted by KTPs under the domain of research production are still limited. Interviews indicated that only a few KTPs engaged in producing systematic reviews. The lag in or lack of local research production and the urgent need to build the production of high quality, policy-relevant systematic reviews and local single studies were repeatedly emphasized across interviews, the document review, and observations (Table 4).

Push efforts constituted the bulk of KTP efforts to link evidence to policymaking. All KTPs built their capacity on developing evidence briefs for policy and they all developed evidence briefs, with significant variation in the number of produced briefs across KTPs at the time of interviews. A common perception among the interviewed KTPs was that evidence briefs were useful tools for influencing policymaking. This view was illustrated by a KTP leader:

Participants acknowledged that the process of developing evidence briefs was burdensome and lengthy; this was mainly attributed to a lack of skilled human resources to prepare evidence briefs and poor quality of local information to support evidence briefs. As such, the need for further capacity building to ensure that organizations and individuals within countries have the skills to prepare evidence briefs was repeatedly emphasized. Additionally, participants emphasized the need for researchers to better align their research with high priority policy issues in their countries and accelerate their efforts in disseminating research evidence and making it readily accessible for policymaking (Table 4).

Establishing rapid response services (RRS) and “one-stop shopping” through online clearinghouses were undertaken by some KTPs in order to facilitate user-pull, with varying degrees of implementation across KTPs. While challenges encountered under this domain varied across KTPs, a set of common challenges emerged from the interviews and were confirmed by the document review and observations. These common challenges were the lack of skilled and dedicated personnel to conduct efforts to facilitate user-pull and the difficulty encountered by KTPs in accessing and finding local evidence mainly due to the unwillingness of policymakers and research institutions to share local data. As such, the need for building the capacity of KTPs to undertake efforts that facilitate user-pull and for scaling up financial support to pursue capacity building for such initiatives were frequently emphasized (Table 4). REACH Uganda, EVIPNet Burkina Faso, and ZAMFOHR highlighted how collaboration among them helped to develop their skills in implementing RRSs. They emphasized that this type of knowledge sharing is essential for scaling up approaches to facilitating user-pull in other countries.

Efforts to enhance the capacity of research users to acquire, assess, adapt, and apply research were rarely undertaken by KTPs. Common challenges to pursuing user-pull efforts included resistance to change and strong political influences. Additionally, a number of KTPs reported that high turnover in top level policymakers in the government, including political appointees and civil servants, contributed to the leakage of capacity, severed relationships, and shuffled priorities, as a KTP leader explained:

In order to support user-pull efforts, interviews and the document review indicated that there is a need to assess and build capacity among research users to acquire, assess, adapt and apply research to help ensure their active engagement in EIHP initiatives. Such efforts should be combined with establishing institutional structures and routine processes (e.g., criteria for promotion and legislation) for research users to encourage their engagement in EIHP activities. For example, E2P Nigeria contacted policymakers through formal organizations in order to encourage their participation in EIHP initiatives and provided certificates following capacity building sessions (Table 4).

Efforts to engage in deliberative processes and collaboration among research users, researchers, and funders constituted a core component of EIHP initiatives. Deliberative dialogues informed by evidence briefs were perceived by interviewees as the “most commendable [tools for enhancing EIHP]” and “activities [KTPs] were most proud of”. Interviews reported that deliberative dialogues had the potential to influence the use of evidence in policy formulation, influence the perceptions of policymakers, stakeholders, and researchers regarding the availability of research evidence, and strengthen the relationship between policymakers, stakeholders, and researchers. At the same time, interviewees emphasized that these are their own perceptions about the short-term influence of deliberative dialogues and that it is still early in their KTP work to comment on the long-term impact of deliberative dialogues on the use of evidence in policymaking. These perceptions are illustrated in the following quotes by KTP leaders:

“Options that were proposed [in the deliberative dialogue informed by an evidence brief] were integrated in the strategic plan for accelerating maternal survival.”

“[During deliberative dialogues] it was very interesting to see how actors with different interests and views on the healthcare system were willing to discuss new ideas, and contribute without putting aside their differences. This was very encouraging for us.”

Findings revealed that key components that facilitated the engagement of KTPs in conducting deliberative processes were skilled human resources to moderate deliberative processes and the support of policymakers and stakeholders and their willingness to participate in such initiatives. In addition, being hosted within the Ministry of Health (MOH) facilitated interactions between KTPs and policymakers and stakeholders.

Suggestions for strengthening exchange efforts included extending interactions beyond those between researchers and policymakers at the government level to include a diverse range of research users including community members, donors and the international community, and the media. Increasing opportunities to engage with other KTPs doing similar work in order to share experience and best practice was also highlighted, as a KTP leader stated:

A common theme that emerged from the interviews and was confirmed by the document review and observations was the need for intensifying monitoring and evaluation (M&E) efforts. Generally, M&E activities were seldom undertaken by KTPs in LMICs. This lag in conducting M&E activities was attributed to the lack of capacity within KTPs to conduct M&E activities, particularly in implementing M&E approaches and tools, and in analyzing data, as well as the prevailing perception among KTP leaders that M&E activities were particularly challenging endeavors.

When asked about the key outcomes resulting from EIHP activities, interviewees emphasized the increased awareness among policymakers, stakeholders, and researchers on the importance of EIHP initiatives. This was illustrated by a policymaker:

“[The KTP in our country] has become a household name in the MOH as evident by the level of participation [of policymakers and stakeholders] in EIHP initiatives.”

Additionally, strengthened relationships among policymakers, stakeholders, and researchers were common outcomes of the KTP work, as mentioned in the interviews. Furthermore, evidence briefs reportedly helped inform policymaking at the government level. For example, policymakers from Nigeria attested to the utility of evidence briefs developed by E2P Nigeria in informing decisions for launching the Ebonyi State Helminth Control Programme. Other frequently mentioned outcomes included an increased demand for KT products by policymakers, enhanced capacity among policymakers for accessing, assessing, and using research evidence, and enhanced capacity among KTP members for developing evidence briefs and dialogues.

Key to achieving outcomes and bringing about change was strong leadership support particularly from policymakers at the government level, as emphasized in the interviews, document review, and observation of deliberations. Discussions during the interviews and at the Forum of key facilitators for conducting KTP work acknowledged the central role that strong leadership plays in the process of establishing collaborative partnerships between research funders, research producers and users, and for promoting EIHP initiatives. KTP leaders reported that committed high-level policymakers within the government (e.g., advisors of the Minister of Health) were among the essential facilitators to their KT work.

Ensuring the sustainability of EIHP initiatives after the end of funding was identified by interviewees as a major challenge confronting KTPs. Three interrelated factors were commonly cited to play a strong role in influencing the sustainability of KTPs.

The first factor that emerged was the institutionalization of KTPs within the structures (or processes) of the government. There were various arrangements for hosting KTPs. Three KTPs were hosted by the MOH, two were hosted by a university, one by a government research institution, and one by a hospital, while one was an independent non-governmental organization (NGO) (Table 2). Findings on the structure, pros, and cons of the different KTP arrangements are summarized in Table 5.

Institutionalization was thought to help overcome challenges related to retaining capacity and funding, as indicated by interviewees, document review, and observation of deliberations. For example, institutionalization of KTPs within the MOH ensured the continuous funding of EIHP activities and salaries. This can be demonstrated by EVIPNet Ethiopia, which was established by the National MOH as the Directorate of Technology Transfer & Research within the EHNRI; it is composed of staff paid by autonomous public authorities to conduct KTP work. Its strategic direction, activities, and KT priorities are informed by policymakers at the MOH. Participants noted that close linkages to the MOH strengthens the sustainability of KTP work, provides proximity to policymakers, and increases the prospects that KTP work is utilized in policymaking.

At the same time, it was emphasized that KTPs should pay attention to maintaining their autonomy from political interests and governmental control and should protect themselves from the instability and turnover in top level policymakers at the government level, which are mainly influenced by political changes in authority. Being located within a university or a research institute whilst having close collaboration with and support from policymakers and stakeholders was a suggested alternative for institutionalization:

“The instability in the MOH and lack of appropriate capacity makes the university a better alternative [for our KTP].”

Finally, the third key factor that emerged to influence sustainability was the need for building and retaining capacity within KTPs, with special focus on conducting M&E activities and developing skills in advocacy and “selling the idea” of EIHP (Table 4).

The establishment of KTPs in a broad range of LMICs is a promising development in supporting EIHP. KTPs report that they have contributed to increased awareness of the importance of EIHP initiatives and strengthened relationships among policymakers, stakeholders, and researchers, while acknowledging the urgent need for KTPs to conduct more robust M&E efforts to assess the outcome and impact of their activities on health policymaking. Furthermore, deliberative dialogues informed by evidence briefs for policy have been among the most commended tools by interviewees for informing policymaking, partly because they provided a public face for the work of KTPs. Additionally, deliberative dialogues can potentially strengthen interactions among policymakers, stakeholders, and researchers, which in turn increases the prospects for research use in policymaking[23].

It is important to note that KTPs were not able to specify how research evidence was used in policymaking (e.g., was the use of evidence instrumental, conceptual, or symbolic? Did it inform policy development or implementation?). The impact of implementing evidence-informed policymaking (e.g., improvement on population health status) has not been assessed and there were no clear mechanisms for undertaking such initiatives. However, this structured reflection provides a snapshot in time of the work of KTPs and their efforts to scale up activities to link research evidence to policymaking and assess the impact of their work, which is still ongoing.

Findings from this structured reflection indicated that KTPs have scaled up push and exchange efforts to link research evidence to policymaking. However, KTPs’ production of research, efforts to facilitate user-pull, user-pull efforts, and evaluation efforts need to be intensified in their view. In order to facilitate these efforts, findings suggested building the production of local evidence, building the capacity of KTPs and research users to undertake such activities, and increasing financial support to pursue capacity building and M&E activities.

Variations in achieving EIHP activities among KTPs have been due to the differences in support systems and value placed on efforts to link research to action available to KTPs in their countries. Strong leadership and support from policymakers at the government level in addition to funding and technical support (e.g., availability of mentors) from international funders were key facilitators to conducting EIHP activities. However, the lack of skilled and dedicated human resources to conduct push efforts, efforts to facilitate user-pull and exchange efforts, and to evaluate these efforts impeded KTPs.

Ensuring the sustainability of EIHP initiatives after the end of funding was identified as a major challenge confronting KTPs. KTPs reported that institutionalization within government structures or processes has helped to retain human resources and secure funding for their KTP activities, whereas KTPs hosted by universities highlighted the advantage of attaining autonomy from political interests while maintaining close collaborations with and support from policymakers and stakeholders.

The study has several strengths. First, the data were collected using three different sources (interviews, document review, observation of deliberations), which helped enhance understanding of KTP experiences and allowed cross-checking of findings. Second, the “framework for assessing country-level efforts to link research to action” that was used for analysis helped build a comprehensive understanding of activities, the set of facilitators and challenges influencing each of these activities, and key learnings for enhancing these activities within each component of the framework. Third, this study is one of the very few reporting on the early experiences of KTPs in LMICs and can help identify opportunities for improving ongoing activities of KTPs and informing the experiences of other growing KTPs in LMICs.

With regards to limitations, our sample of interviewees was drawn from KTP staff and leaders and then from policymakers and stakeholders. Eliciting the opinion of a larger number of policymakers and stakeholders at the country level can potentially provide additional in-depth insights on the work of KTPs from the perspective of the groups that KTPs eventually aim to reach and influence. This can provide valuable information for assessing initiatives conducted by KTPs, tailoring EIHP initiatives to their needs, and improving the work of KTPs in each setting. That said, KTP leaders come from different backgrounds, such as policymakers at the government level and stakeholders from civil society as well as researchers, and have provided their perceptions on EIHP activities from these perspectives. Additionally, the document review and observations and synthesis of deliberations from the International Forum included findings from the perspectives of policymakers, stakeholders, and researchers, in addition to KTP leaders.

Findings from this study on the barriers and facilitators to KTP work are congruent with those previously reported by policymakers, stakeholders, and researchers on EIHP, particularly from LMICs[7, 24‐30]. Gaps in research production and limited alignment with regional priorities, as well as lack of skills among research users to acquire, assess, adapt, and use research evidence, were also previously reported to hinder EIHP activities in LMICs[25‐28]. There is little empirical evidence on effective KT approaches and how KT strategies can be tailored for different contexts and disciplines[31‐35]. The positive views held by interviewees regarding evidence briefs and deliberative dialogues are supported by a recent survey of policymakers, stakeholders, and researchers from LMICs, whereby respondents viewed evidence briefs and deliberative dialogues very favorably and reported strong intentions to act on what they learned in the briefs and dialogues[36]. Lessons drawn from the experience of ZAMFOHR included the necessity of performing comprehensive situation analyses to understand the evidence-to-policy climate and the operational niche for the KTP and the importance of networking to benefit from the experience of other KTPs, funding opportunities, and technical support[7]. ZAMFOHR’s experience, as well as findings from other studies, suggested focusing the KTP program on building the capacity of its own staff, policymakers, stakeholders, and researchers to engage in EIHP[7, 26‐29], which also confirm lessons learned from KTPs in the current study. Furthermore, making changes to the existing institutional structures and incentives provided to researchers and research users were suggested strategies to encourage them to engage in EIHP initiatives[25, 28‐30].

Findings from this study suggest the need for intensifying efforts to increase research on effective KT strategies and understanding KT tools in different contexts. Specifically, further research is needed to systematically determine how the ways in which evidence briefs, deliberative dialogues, and other KT strategies are designed, their content and the context in which they are developed will influence their usefulness in supporting EIHP[9, 36, 37]. A recent systematic review reported that contextual factors, particularly the institutions, interests, and values of a given context, as well as issue-related factors (such as whether issues are polarizing, salient, or familiar to policy actors) can influence views regarding evidence briefs among their intended users[38]. Similarly, future research may explore the ways in which contextual factors and policy issues can influence other strategies (e.g., capacity building targeted at research producers and users) for supporting the use of research evidence in policymaking processes.

There is a general assumption that evidence-informed policymaking will inevitably improve outcomes; however, there is no robust evidence base to support/disprove this assumption. As such, assessing the impact of policies informed by evidence briefs for policy and deliberative policy dialogues (e.g., improved health of the population) is needed to complete the M&E cycle and to make the case for supporting EIHP. Further research comparing governance structures of KTPs in different contexts and the factors influencing their sustainability is also much required, given the current volatility of these initiatives as indicated in this study.

This structured reflection provided four key lessons to improve the activity of KTPs and to help inform those interested in establishing KTPs in their own countries. Firstly, concerted efforts should be directed towards capacity building for preparing evidence briefs, conducting deliberative dialogues, implementing a RRS, developing and maintaining online clearinghouses, and evaluating these activities among those engaging in these EIHP activities. A systematic review reported that training in using systematic reviews and providing peer-group support facilitated the uptake of evidence from systematic reviews[31]. Capacity building ought to go hand-in-hand with raising the awareness of policymakers, stakeholders, and researchers alike on the importance of EIHP in addition to developing their skills to engage in EIHP activities (e.g., skills of policymakers to access, assess, and apply research; and skills of researchers to align research production with policy priorities, produce systematic reviews, effectively disseminate their research, and engage with policymakers). In Nigeria, for example, a training workshop that focused on capacity development for EIHP and building partnerships between policymakers, stakeholders, and researchers showed significant improvements in participants’ knowledge and their understanding of the health policymaking process and the use of evidence[8]. Different capacity-building approaches can be implemented and assessed with both researchers and research users and tailored to different contexts; for example, comparing the use of group workshops with one-to-one meetings on building the capacity of policymakers, stakeholders, and researchers[39].

Additionally, funders are in a position to move the agenda of EIHP forward by providing training on KT skills and mentoring or coaching for KTPs, policymakers, stakeholders, and researchers as well as by advocating for KT through rewarding/celebrating the work of those promoting KT[40]. At the same time, such efforts should be combined with establishing institutional structures and incentives that support EIHP activities (e.g., criteria for promotion or legislation to mandate the use of research evidence). A systematic review on the use of research evidence in public health policymaking suggested that changing the culture within which policymakers work (in terms of structures, rewards, and training) such that more value is placed on the use of research evidence might encourage its use[41]. For example, the Ministry of Health and Long-Term Care in Ontario requires training for civil servants in finding and using research evidence, incorporating assessments of the use of research evidence as part of performance reviews. It also requires civil servants making submissions to the minister or to cabinet to document the key sources for research evidence that were searched and declare that relevant findings were used to inform the submission[42].

Secondly, the further expansion of financial support from funders, government sources, or revenue-generating activities will be needed to support and retain capacity within KTPs as well as to ensure the sustainability of these activities. Additionally, funding agencies could play a stronger role in promoting EIHP by requiring the presence of a KT component as a condition for the funding of proposals and by facilitating the sharing of experiences across different KTPs and international and local actors. For example, the Michael Smith Foundation for Health Research proposes five key functional areas that funders can undertake for effective KT, these are advancing KT science, building KT capacity, managing KT projects, funding KT activities, and advocating for KT[40]. This funding agency suggests specific examples of funding agencies’ activities, including funding knowledge synthesis and other KT strategies as well as KT model testing, providing awards for research use and uptake (e.g., to adapt and implement research evidence), and developing partnerships with governmental organizations and NGOs in order to leverage limited human and financial resources and align with existing and new initiatives and research[40].

Thirdly, there is a need for each KTP to think strategically about institutionalizing itself in a way that provides greater access to support and a better opportunity to influence policies. Where the KTP locates itself is a critical variable in its organization and operations and should be based on a comprehensive situational analysis of contextual factors related to EIHPs[7]. It should also be aligned with the structures (or processes) of the government. Additionally, more attention needs to be given to ensuring leaders at all levels with a vision to support EIHP are identified and supported in their work.

Finally, more attention needs to be paid to integrating M&E activities as key components of KTPs right from their conception. M&E activities should be viewed as tools for obtaining the buy-in of research users and funders for supporting KTPs where appropriate, which in turn would help to ensure the sustainability of KTPs. It is important that KTPs integrate their M&E work starting from the planning phase of KT work and that they allocate sufficient resources to implementing M&E. The KTPE team developed a range of tools that can assist KTPs in conducting M&E activities including survey tools to evaluate the perceptions of users of evidence briefs and deliberative dialogues, a brief survey administered every two years to a sample of policymakers, stakeholders, and researchers to assess outcomes of KT activities, and case studies to capture impacts of KT activities on the use of research evidence in policymaking. Very few systematic reviews examine implementing research findings into policy[41]. There are various challenges to evaluating the impact of research use in policymaking; these include metrics that are difficult to define, the fact that the use of research evidence differs depending upon the context, and the resource- and time-intensive nature of impact studies[40]. As such, further research can concentrate on assessing the outcomes and impacts of conducting KT activities on the use of research in policymaking in addition to assessing the impact of evidence-informed policies on the health system and the health of the population.