Background
Considerable resources are invested in applied health and social care research yet there is an acknowledged gap between what is known and what is acted upon [
1]. There is a growing interest in addressing this gap, and a simple starting point is the need to communicate research findings appropriately to audiences that are in a position to act. Research funding bodies expect their grant holders to disseminate their findings however, there is a lack of clarity about what is meant by dissemination [
2]. Wilson et al. in their scoping review of conceptual frameworks to support dissemination identified 33 frameworks. Only 20 of these had theoretical underpinnings, for example on the use of persuasive communication such as within the Lavis framework [
2,
3]. In addition, few funding bodies offer guidance on dissemination beyond the need to do so [
2].
We acknowledge that there is considerable interest in co-produced approaches to research planning, activity, and dissemination, and there are a number of initiatives funded with this in mind. Examples include the Integrated Knowledge Translation Research Network in Canada, and the National Institute for Health Research Collaborations for Leadership in Applied Health Research and Care (NIHR CLAHRC) in England. These collaborations are designed to better bridge the knowledge to action gap [
4‐
9].
However, it is not unreasonable to suggest that the default dissemination approach for researchers’ is concerned with the conduct and reporting of research in a high quality ‘peer reviewed’ journals. System factors drive these behaviours, for example, developing an academic career requires evidence of a publication track record which despite the rise of altmetrics [
10] is still largely viewed through the lens of scientific journals and their associated impact factors. In the UK, this is reinforced by metrics related to the Research Excellence Framework (REF) where institutions (and arguably careers) are measured against publications in high quality, international journals. The introduction of case studies demonstrating societal impact may ameliorate this, and Greenhalgh and Fahy’s review of the 2014 submissions identified targeted knowledge translation activities in 38 of the 162 impact case studies [
11].
Academic publications serve the scientific process but have limitations in terms of bridging the gap between knowledge and action [
12]. Given, as we outline above, that many of the quality and esteem criteria in academia are related to such publications, we acknowledge that they should be part of the dissemination process, and they are also an important step in the production of summary evidence on what is known. Graham’s conceptualisation of knowledge into action, for example, divides the process into two concepts: knowledge creation and an action cycle [
1]. He suggests that as well as reporting in peer reviewed journals, researchers have an additional ethical duty to publish research results which are capable of ready implementation, and which may support the knowledge into action gap. He uses a funnel to symbolise knowledge creation and argues that as (empirically derived or experiential) knowledge travels through the funnel it is increasingly tailored to meet the needs of stakeholders. At the end of the funnel the ‘third generation’ knowledge takes the form of ‘tools’ or products for example, guidelines or decision aids. These tools are designed to present the knowledge in a format that meets stakeholders’ (end users’) needs [
1]. This is a helpful model which conceptualises two elements to knowledge mobilisation and their connectivity.
Our study focuses on the knowledge creation element of this cycle, and seeks to identify further methods of disseminating knowledge beyond conventional scientific publication in health services and health systems research. Our contribution to this field stems from experience within one of the NIHR CLAHRCs in the UK. These are collaborations tasked with addressing the research-practice gap through partnership between academia and practice. Many CLAHRCs operate through co-producing knowledge within these partnerships, which in itself supports research mobilisation into practice. Rycroft- Malone et al.’s evaluation of the CLAHRCs suggests that their potential
‘to close the metaphorical ‘know–do’ gap was dependent on historical regional relationships, their approach to engaging different communities, their architectures, what priorities were set and how, and providing additional resources for implementation, including investment in roles and activities to bridge and broker boundaries’ [
8].
Working within a CLAHRC has afforded the opportunity to reflect (using our case study library available from
http://clahrc-yh.nihr.ac.uk/resources/case_study_library to identify research outputs that may have potential in crossing the boundary into policy and practice. For example, some of the outputs from the Keeping Warm in Later Life (KWILLT) project [
13] went beyond academic publication and used a segmentation model to produce 6 pen portraits to summarise the research findings for policy makers. Pen portraits are descriptive narratives to explain health behaviours in complex environments. In the KWILLT project the pen portraits described groups of older people who are vulnerable in cold weather, and the reasons that make them vulnerable. These were developed in order to inform targeted interventions for these groups in policy and practice. Specifically, the pen portraits helped both to make the research visible and facilitated action on the part of local and national commissioners. [
14] The pen portraits were used as a resource in the National Cold Weather Plan; they informed policy making in local government, and supported volunteer organisations to target advice to vulnerable older people. As Rycroft- Malone et al. [
4] acknowledge, some effort is needed to plan cross boundary work and the CLAHRC pen portraits supports this. We think the RDATs are a promising option as they can act as a ‘boundary object’ [
4,
15] which Fox [
15] describes as a construct to improve the uptake and transfer of research into other areas such as social policy and public services. RDATs then could represent ‘tailored’ knowledge [
1] which is potentially immediately useful to policy and practice. This idea, albeit requiring testing and refining, seems to be potentially helpful.
The term ‘actionable is repeatedly used in the implementation science literature occurring, for example, 843 times within the journal
Implementation Science. It is often the basis of discussion in both old and new media presentation of scientific findings, but there is little conceptual clarity around this term. We suggest it is generally understood in quite broad terms, and taken as meaning that the findings of a research study are implementable by practitioners or policy makers, and is useful to them. Further, in applied research settings it is arguably a given that there is a need for research to be actionable (directly) or inform action (conceptually). However, there is likely less clarity on what constitutes an actionable output and little evidence on how to produce RDATs nor any debate on where such tools fit in evidence hierarchies [
16].
Our study sets out to address this first issue, and aims to contribute to understandings about how to develop a RDAT, and in particular the importance of resource, end user involvement and embedding of such activity in the research process. The focus of this study was to seek some clarity about research outputs that might help to translate research for policy makers and practitioners. In the context of this study the end user are therefore policy makers and practitioners. Although we have consulted patient representatives on some decision making (see methods section), they are not the intended target audience for policy and practice related RDAT, so are therefore out of scope. Our aims were firstly, to develop a consensus definition of what is meant by the term ‘actionable tool’. Secondly, to come to some consensus on characteristics of an actionable tool that would support its use across the research-practice boundary, including how they might be integrated into the methods, conduct and reporting in projects. We also hoped to gain some agreement of examples of RDAT. Finally, throughout the study process we have reflected on the emerging definition and its potential to contribute to crossing the know-how boundary.
Results
Results of the first round
The key product of round one was the initial actionable tool definition (see Fig.
1). Participants also identified some factors in the research process that might be considered important in the development of appropriate actionable outputs from research, including the use of co-productive techniques to support this. Such issues were explored in round two.
The workshop also identified some examples of research outputs that might be considered a RDAT such as patient decision aid, clinical protocol, algorithm, and balanced score card. This initial list was used to structure items in the Delphi which asked whether an example could, using the definition, be considered to be a RDAT.
Results of the second round
Definition and characteristics of RDAT
In this round we gained consensus in a number of areas. We did not get agreement on first candidate definition of an actionable tool (Fig.
1) as the definition was considered too abstract and academic by our panel. However, we did get consensus that a definition of RDAT should include distinct descriptive elements (79.41% agreement, median 2). We also had some consensus in what these elements might include. These were that it should include the medium for communication mindful of the target audience (76.47% agreement, median 1) and that it should have a prompt (76.47% agreement, median 2), although the qualitative data revealed additional complexity associated with a prompt. Consensus was nearly reached on the ‘knowledge object’ characteristic of the tool (73.5% agreement, median 1.5), whereby the research knowledge is contained, and can be identified in the tool.
We found stronger, and more consensus around factors that would helpful in the development of actionable tool within the research process including the notions of involvement of end users from early stages, including at the grant application (80.65% agreement, median 1), and the research design (83.37% agreement, median 1). It was also seen as important that funds and resources should be allocated in the project budget (80.65% agreement, median 1). The expert panels’ responses around the definition enabled us to develop a second adapted definition (see Fig.
2), and thematic analyses of their open text answers suggested the other considerations, which contributed to the round 3 questionnaire.
Examples of RDATs
We asked the panel’s opinion on whether the list of research outputs identified in round one could be examples of RDAT. Panel members were given an example to help with their decision making. Consensus was only nearly reached for one example, the patient decision aid (75.8%; median = 2) and none of the examples was completely rejected by the panel. Table
2 below illustrates the diversity of views in relation to our other example RDATs.
Table 2
Delphi panel scores for example RDATs
Service specification | 51.5 | 15.2 |
Service evaluation/research tool | 48.5 | 21.3 |
Worksheet | 39.4 | 27.3 |
Simulation model | 21.2 | 24.3 |
Clinical decision aid | 57.6 | 18.2 |
Audit Tool | 39.4 | 36.4 |
Executive summary | 27.3 | 57.5 |
Patient decision aid |
75.8
| 12.1 |
Algorithm for clinical decision making | 66.6 | 15.2 |
Risk Assessment Tool | 51.5 | 21.3 |
Balanced Scorecard | 24.3 | 21.3 |
Teaching and learning pack | 45.5 | 15.2 |
Local protocol | 57.6 | 24.2 |
National protocol or guideline | 48.5 | 21.3 |
Social marketing materials | 24.3 | 39.4 |
Film | 15.3 | 42.4 |
Patient Reported Outcome Measure | 30.4 | 27.3 |
Patient Reported Experience Measure | 27.3 | 30.4 |
Panel members were able to offer explanation of their scoring. In terms of assessing whether a potential tool was a RDAT, the main difficulty was the lack of consensus (as yet) around a definition. A key message from the panel’s comments were around understanding the context in which the tool could be used, exploring support needed in using the tool, as well as the views of the end user in how useful it might be in a given context. This was further reinforced through discussing these early findings with our PPI advisory group. We also explored the types of tools identified in round one with PPI advisory group, and gave an example of the patient decision aid, illustrating this with a real example of a cataract patient decision aid. The tools were viewed quite negatively by the PPI group who valued the ‘expertise’ of discussions with their clinician rather than the interactive tool. They also felt the tools made assumptions about what was important to them in terms of decision making.
Results of round 3
We experienced 30% drop out rate of participants between rounds two and three. Twenty-three responded to the survey of the 33 invited. However, we achieve consensus on the new definition given in Fig.
2 (78.2% agreement, median 2).
The new definition was well received. It was viewed as clear and concise and ‘helps the user consider the important elements for translating research findings into everyday practice’ (round 3 participant). The written responses identified further nuances and understandings, particularly in relation to the prompt and its potential for action. Participants continued to identify the importance of context and the impact of external factors in terms of whether something was implemented or not. For example, the tool might aim to prompt behaviour but external factors to the setting might be the instrument of action, for example, a policy directive or a budget necessity. Another clear view was that the prompt within the tool could aim to encourage discussion within a team or organisation that may or may not lead to action. Although consensus was reached on the visibility of the research within the tool, there was debate around the importance of this to the end user, linked to their knowledge beliefs in deciding whether to use it, or whether it would promote action.
Higher levels of agreement were achieved in the generic content enabling factors that might make the tool more used and therefore promote action. These findings are reported in Table
3. Two of the questions related to what to communicate about the research. Qualitative responses highlighted that understanding the context of the research and its distinctive message was thought to enable the end user to assess its transferability to their context, and identify its unique worth:
‘what is this telling me and next- is this relevant to [my] work area?.’ It was thought that a clear articulation of these two areas is likely to lead to behaviour change.
Table 3
Summary results for 3rd round of the Delphi
Ability to tailor to local context | 95.7 | 1 | 0 |
Information on outcome measures to support the tool in practice | 95.6 | 2 | 1 |
Study setting or context | 91.3 | 2 | 1 |
Information on who needs to take action | 91.3 | 1 | 0 |
Information about what the study adds to the evidence | 87.0 | 1 | 1 |
Information to support implementation | 82.6 | 2 | 1 |
Testimonies from other users | 82.6 | 2 | 1 |
Inclusion of evaluation tools | 82.6 | 2 | 1 |
The tool prioritises actions | 78.2 | 1 | 1 |
Timescales for implementation | 78.2 | 2 | 1 |
In the second set of questions, consensus was achieved in all items with the exception of the tool including self assessment guidance for its use (73.9% agreement). There was clear consensus that end users would find the following items important: information on who needs to take action (91.3% agreement); ability to tailor to local context (95.7% agreement); information on outcome measures to support the tool in practice (95.6% agreement) information to support implementation (82.6% agreement); testimonies from other users (82.6% agreement); inclusion of evaluation tools (82.6% agreement); the tool prioritises actions (78.2% agreement); and details timescales for implementation (78.2% agreement). This suggests our experts had clear views about what a tool should contain and these revolved around factors that support its use and further action.
Discussion
This project has been able to develop a definition of a RDAT that reached consensus after three rounds from a Delphi expert panel. Members of the panel were experts in knowledge mobilisation, policy makers, researchers and practitioners. Our consensus definition has therefore been informed by the tacit knowledge of individual participants made explicit through the consensus generating mechanism of a Delphi. The quality of the definition will be judged by others, and we recognise there is further work to be undertaken in order to test it empirically. We hope in time the definition will be refined, and the concepts contained within it further explored to understand more about the metaphorical know – do gap.
Delphi panels are helpful in reaching such consensus especially when the matter is complex and unclear as they act as virtual panel of expertise [
18] but it is important to recognise there are limitations to the method and these are applicable to this study. The limitations centre around the ability to recruit a range of experts and maintain their involvement [
19]. Our experts largely came from the knowledge producer, policy and practice community interested in implementation of research in practice. Future work needs to involve service users [
24]. Considerations on context and facilitation in the use of RDATs should also be explored with a range of practitioners.
Another limitation was the focus and boundary of this research, which examined the research-practice gap in applied research with a focus on policy and practice. We recognise there are other areas were RDATs might be a useful concept for example, in basic science or in technologies and its link to applied research. This was not explored in this project.
We had drop out between rounds two and three of nearly one third, and this may be down to the extensive nature of round 2 which required consideration of many candidate RDAT. All were busy clinical-academics, and time pressures for competing the survey may be demanding in such schedules. There may have been disappointment in not pursuing agreement on types of tools in the third round, although this decision was based on the interim findings of round 2.
Further, the initial round workshop was shaped around two theoretic models in relation to knowledge mobilisation. This included Graham’s funnel mode [
1,
25] in the Knowledge into action cycle, and the concept of RDATs as a boundary object. We recognise there are other models, and that the wider panel members may have been distracted by our conceptualisation in this way.
Conclusion
Finally, there was a strong consensus that planning the research process, and the production of such research outputs should be undertaken in partnership with end users from writing grant applications onwards. This resonates with much of the knowledge mobilisation literature. For example, Kothari et al. promote a model of Integrated Knowledge Translation, where collaboration occurs from the beginning of the project onwards as end users have a unique knowledge of the context and the potential for implementation of the research [
9]. Others support the theory of coproduction within the research process which views of the Delphi panel also reflected [
26]. Coproduction demands that knowledge is generated within the context of use [
27] and calls on a blurring of boundaries between this producers and users of research [
26,
28] which also supports notions of co-production. There is also evidence that these more collaborative approaches are helpful in supporting impact in services [
29], which may be helpful in evidencing how research has produced societal impacts, which is increasingly important for research communities and well as for practice. What this study adds is that an expert panel has endorsed this approach with respect to generating RDATs in addition to peer reviewed journals, and describes the characteristics of such outputs that have a potential for action and impact. It also provides a consensus about planning and protecting resources in the research process to do this.
An issue for debate arising from these findings is whether the academic community is prepared, or able to undertake such activity. Coproduction involves addressing power differences [
30,
31], and involves skills such being able to communicate with different audiences including different disciplines and professions. It demands a balance of flexibility and creatively with maintaining standards of research rigor. It involves time, navigation around difficulties, and epistemological tolerance [
14]. This requires further exploration, with considerations on how we develop these skills in the applied health research workforce.
The study results have relevance for both research funders and research producers. For funders, there are messages about providing resources in their calls for funding RDAT as a distinct element of research projects. Tender briefs and application processes could ask for evidenced of RDATs within the research design, and encouraging researchers to consider such outputs in dissemination strategies. For researchers there are recommendations about how to develop RDATs within the research process.
Our work has highlighted that achieving agreement on a typology of RDATs is more problematic. Our findings continue to iterate that ‘context is key’ in relation to whether a RDAT has a potential for impact and change. This is reinforced by the implementation science and knowledge mobilisation literature [
32]. The findings also suggest that it is the end user who decides this. However, the expert panel have been able to identify key considerations that may support the implementation and use of RDATs. The reality is that there may well be no ‘magic bullet’, and that a number of approaches may be useful [
29]. However these results suggest that funding the production of, and tracking the use of RDATs could be an important contribution to using research knowledge in practice.
Acknowledgements
The authors of this paper were supported by the NIHR Collaboration for Leadership in Applied Health Research and Care Yorkshire and Humber (NIHR CLAHRC YH).
www.clahrc-yh.nihr.ac.uk. The views and opinions expressed are those of the author(s), and not necessarily those of the NHS, the NIHR or the Department of Health.
We would like to thank members of the Delphi panel and PPI group for sharing their insights with us. We would also like to thank Melanie Gee, information scientist/research fellow at Sheffield Hallam University for searching and retrieving example RDAT to support the refinement of our definition.