Background
Methods
Data sources and search
Study selection: inclusion criteria
Study selection: screening
Data abstraction
Data charting and collation
Consultation
Results
Literature search
Characteristics of the included articles
Studies [reference] | Country | Study design | Setting | Type of end user or participant | Number of end users or participants |
---|---|---|---|---|---|
Albert 2007 [38] | Mali | Qualitative study | National government: pharmaceutical decision-making group | Health policy maker | 19 |
Armstrong 2012 [39] | Australia | Qualitative study | Health related organisations | National stakeholders of relevance to employment and health, advocacy, member organisation for a workforce or national policy | 9 |
Atack 2010 [40] | Canada | Qualitative study | Different health care settings (research hospital, community hospital, community health centre, long-term care facility, district/regional health authority, government ministry/department, academic health science centre, others) | Health care manager | 42 for Technology Acceptance Model survey (TAMS) and 12 for interviews |
Campbell 2009 [42] | Australia | Qualitative study | Public health and health service government | Health policy maker and researcher | 38 policy makers/41 researchers |
Campbell 2011[41] | Australia | Qualitative study | Independent research organisations and Department of Health | Health policy maker and researcher | 8 policy makers/11 researchers |
Ciliska 1999 [21] | Canada | Qualitative study | Public health organisations | Health care managers and total, 277 were eligible, 242 participated in the first survey; 225 participated in the second survey | |
Dobbins 2001a (CR to Ciliska 1999) [22] | Canada | Qualitative study | |||
Dobbins 2001b (CR of Ciliska 1999) [23] | Canada | Qualitative study | |||
Dobbins 2004 [30] | Canada | Qualitative study | Public health/health promotion, government | Health care managers and policy makers | 46 |
Dobbins 2004 [31] | Canada | Qualitative study | Public health (programme managers, directors, epidemiologists, medical officers of health, provincial consultants, local board of health members) | Health care managers and policymakers | 51 participants |
Dobbins 2007 [43] | Canada | Qualitative study | Public health decision-makers: managers, directors, medical officers | Health care managers and policy makers | 16 |
Dobbins 2009a [24] | Canada | RCT | Public health departments | Health care manager | 108 |
Jewell 2008 [44] | USA | Qualitative study | Conference attendees | Health policy maker | 28 (11—public health; 15—legislators; 2—both) |
Lavis 2005 [45] | Canada, Scotland, Norway, UK | Systematic review and qualitative study | Studies of decision-making by health care managers and policy makers, interviews of managers, policy makers, website review | Health care managers and policy makers | 17 studies and 29 participants |
Packer 2000 [52] | England | Qualitative study | 5 health authorities | Health policy makers: health authority contacts including public health consultants, public health specialist registrars, a research analyst, and a public health nurse | 11 health authority contacts |
Ritter 2009 [46] | Australia | Qualitative study | Drug policy units | Health policy maker | 31 participants |
Rosenbaum 2011 [47] | Norway, Argentina, China, Colombia, South Africa, and Uganda | Qualitative study | National or international health service or policy-related work in health departments, national insurance programmes, hospitals, or aid organisations | Health care managers and policy makers | 18 participants |
Shepperd 2013 [48] | England | Qualitative study | Regional health authorities | Clinicians and Commissioners (including 2 general practice commissioners, 3 pharmacists, 6 public health consultants, and 4 health caremanagers) | 23 clinicians; 15 commissioners (2 general practice commissioners, 3 pharma, 6 public health consults, 4 managers) |
Suter 2011 [49] | Canada | Qualitative study | Provincial health authority | Health care managers and policy makers | 13 participants |
Vogel 2013 [50] | Canada, England and Wales, Finland, Germany, Italy, the Netherlands, Norway, Scotland, and Spain, Cameroon, Ethiopia, Mozambique, Zambia, Uganda, and South Africa | Qualitative study | National policy organisations | Health care managers and policy makers | 112 respondents |
Yousefi-Nooraie 2009 [51] | Iran | Qualitative study | Capacity building workshop on systematic review methods in Iran | Health care managers and policy makers | 131 participants |
Barriers to use of systematic reviews
Attitudes | Knowledge | Skills | Behaviour | |||
---|---|---|---|---|---|---|
Lack of agreement | Lack of outcome expectancy | Inertia of previous or actual practice/lack of motivation | Lack of awareness/familiarity | Lack of skills | SRs/intrinsic factors | Extrinsic factors |
Avoid providing specific recommendations for action based on SR; do not believe in causal linkage [45] | Lack of motivation to use reviews will require changing policy makers’ attitudes [51] | Administrators’ limited understanding of the function of SRs [44] | Participants’ poor conceptual understanding of systematic reviews [47] | The review itself did not appear to be user-friendly due to inaccessible language and dense layout [39] | Policy inconsistency within health care system (differing interests, different policies run in parallel, performance based incentives) [48] | |
Systematic reviews do not necessarily frame the existing evidence in terms of their policy applications [44] | The health agency staff had not been taught “to continue to use research to inform their decisions, to inform their practice. They therefore made decisions based on “common sense”, “gut level”, “standards of practice”, and comparative convenience and awareness of available data, rather than based on systematic reviews of research.” [44, 52] | Accessibility: within a systematic review, have difficulty identifying key messages [44] | Attacks on an evidence-based approach. Several officials also discussed instances in which the whole notion of evidence-based health care had come under direct attack, usually by pharmaceutical companies, sometimes in collaboration with advocacy groups, some of which hid their involvement with industry [44]. | |||
Lack of or unknown credibility of authors of the research [21] | No policy makers mentioned having utilised information from systematic reviews, and most seemed unaware of their existence [38] | Lack features that would make them easier for government officials to evaluate. For example, the quality of studies is often difficult for non-experts to interpret because the explanation of research methods is long and complicated [44] | Concepts presented in tables, including those that showed the GRADE assessment and different levels of risk, were not clear [47] | Accessibility. Even when evidence is available, policymakers may have problems obtaining it [44] | ||
Ethical disagreement [21] | Appraising and synthesising the evidence was seen as an even bigger challenge [40] | Tables running over 2 pages were cumbersome to read [47] | Lack of availability of research results [21] | |||
Research information not valued at community level [21] | Abbreviations caused confusion [47] | Lack of resources to implement research [21] | ||||
Policy decisions are made based on other factors like cost and equity considerations, particularly if evidence base is frail [52] | Words like “sample size” and “relative risk” would be difficult to interpret [47] | Use of jargon and/or unfamiliar vocabulary [47] | Policy climate—provincial/regional not conducive to use [21] | |||
Mismatch between the type of content offered and their information needs [47] | Lack of expertise in evaluating SRs [44] | Numbers in the text and those in the tables do not correspond precisely [47] | Lack of timely completion of review [21] | |||
Translating evidence to the local context (including sub groups of patients): individuals frequently had to make independent decisions about how to relate evidence to the needs of their local context, discuss and debate the evidence with local stakeholders and take decisions about its usein practice [48] | Appraising and synthesising the evidence was seen as an even bigger challenge [40] | Current practice patterns lead policy makers and managers reluctant to use reviews [21] | ||||
Policymakers expected content lying outside the scope of a review: recommendations, outcome measurements not usually included in a review, detailed information about local applicability or costs, and a broader framing of the research enquiry [47] | Wanted a shorter, clearer presentation [47] | Cost of retrieving information prohibitive [21] | ||||
Reviews covered issues at a more complex level than required [52] | Insufficient authority to implement research results [21] | |||||
Lack of detail on how to use strategies, tools, processes that would lead to successful integration (i.e. guidance on breaking down systems barriers or how to achieve integration in the context of big, complex system) [49] | Policy makers had difficulty finding brief research summaries and systematic reviews when they were needed (i.e. difficulty accessing SRs) [42] | |||||
Tended not to use the full report instead referring to the less dense, more accessible articles [49] | ||||||
Wanted a shorter, clearer presentation [47] | Research often published in academic sources, poorly accessible to policy makers, LMIC policy makers have limited access to subscription-based K or the internet [47] | |||||
Lack of indexing local journals in international databases, harmonised reporting criteria, editorial processes and presentation of local journals, minimum standards for reporting of research for all local journals. Coverage and searching quality of databases of papers published in local language needs improvement, single national database for research registration, technical and professional support for current databases [51] | ||||||
If department within commissioning organisation is not in a position of strength, unlikely that evidence will be used for decision-making [52] | ||||||
Lack of time to find or discuss evidence, usually need an answer to a problem on the same day [52] | ||||||
Finding the evidence was described as problematic. Several fellows called for greater access to systematic reviews; this was a resource they wanted to see augmented through the desktop [40] | ||||||
Policy makers’ belief that searching, accessing, and reviewing research findings are highly time consuming is perhaps a good argument for the increased production, promotion, and dissemination of systematic reviews [38] | ||||||
Limited time to read full study reports (of a SR) [43] |
Attitudes
Knowledge
Skills
Behaviour
Facilitators to use of systematic reviews
Attitudes | Knowledge | Skills | Behaviour | ||
---|---|---|---|---|---|
Agreement/usefulness | Motivation | Awareness/familiarity | Expertise/experience/training | SRs/intrinsic factors | Extrinsic factors |
Expecting to use the systematic reviews in the future [23] | Recognition of relative importance of SR compared to other sources of information such as single studies (culture of evidence-based decision-making) [31] | One’s age [23]—younger, more likely to use | Delineating the effects for a particular group with more focused subgroup analyses in SRs [44] enhanced their usefulness | Making decisions in collaboration with other community organisations increased likelihood of using reviews [23] | |
SRs to provide guidance and suggestions for implementation of findings, not just reporting facts [43] | Willingness of health care providers to use systematic reviews [51] | Number of years since graduation [23]—more recent graduates more likely to use | Providing information about the benefits, harms (or risks), and costs [45] | Increasing the opportunities for interaction and exchange between policy makers and researchers is key to promoting the use of research evidence in policy [42] | |
Most policy makers reported having needed the data and reviews in the past 12 months, having commissioned research or reviews during this period, and having used evidence to contribute to the content of policy [42] | Perception that reviews facilitate critical appraisal of evidence and are easy to use, information about what works and clearly articulated implications for policy (costs, applicability, impacts on equity) [47] | Providing training in basic search skills may increase use [52] | One-to-one interaction with the researcher to discuss research findings [43] | ||
Respondents who expected to use the reviews in the future were more likely to have used a review than those who did not expect to use the reviews [22] | Presenting selected important systematic reviews to policy makers may change their attitudes towards evidence-based decision-making, presenting successful/unsuccessful policies [51] | Perception that systematic reviews could overcome the barrier of limited critical appraisal skills [22] | Identify attributes of the context in which the research included in a systematicreview was conducted to inform assessments of the applicability of the review in other contexts [45]. Concise statements about lives or money can infuse the political discussion with a tone of rationality, framing the trade-offs as technical and straightforward [44]. Providing information about the benefits, harms (or risks), and costs [45] | ||
Coming from credible sources [47] | Ongoing training in critical appraisal of research literature [23] | Add additional local value toany type of systematic review by using language that is locally applicable and by engaging in discussions about the implications of reviews with the health care managers and policy makers who could potentially act on the reviews’ take-home messages [45]. Identify attributes of thecontext in which the research included in a systematic review was conducted to inform assessments of the applicability of the review in other contexts [45]. Concise statements about lives or money can infuse the political discussion with a tone of rationality, framing the trade-offs as technical and straightforward [44] | Fund production and updating of SRs with additional resources for health care managers and policy makers to interact and fund local adaptation process for SR [45] | ||
Ensure SRs are included in a one-stop-shop that provides quality-appraised reviews [45]. Add additional local value to any type of systematic review by using language that is locally applicable and by engaging in discussions about the implications of reviews with the health care managers and policy makers who could potentially act on the reviews’ take-home messages [45]. Identify attributes of the context in which the research included in a systematic review was conducted to inform assessments of the applicability of the review in other contexts [45] | Collaborative creation of knowledge in a format that is easy to view and comprehensible and allows fast and easy referencing [49] | ||||
Opportunities for training and education on systematic reviews (definition, significance, appraisal) [30] | Replacing unfamiliar terms or adding definitions to the re view [47]. Ensure SRs are included in a one-stop-shop that provides quality-appraised reviews [45]. Add additional local value to any type of systematic review by using language that is locally applicable and by engaging in discussions about the implications of reviews with the health care managers and policy-makers who could potentially act on the reviews’ take-home messages [45] | Involvement of librarians and health information specialists as a solution to lack of database access, establishment of a national portal for expanding access [51] | |||
Coming from credible sources [47] | Teaching about systematic reviews, integration of this course into postgraduate educational curricula, mandatory education of research methods to researchers, consultation support in methodology and scientific writing, professional methodologists on research teams [51] | Involvement in an advisory role by policy makers on research teams (i.e. involved with the development of research questions, assisted with dissemination) [42] | |||
Relevance to policy decisions [31] | Perception that systematic reviews would overcome the barrier of not having enough time to use research evidence [23] | Make the user-friendly “front end” of systematic reviews available through an online database that could be searched using keywords that make sense to health care managers and policy-makers and that is linked to the full reviews when they are available through other sources, such as The Cochrane Library [45]. Replacing unfamiliar terms or adding definitions to the review [47] | Position of end user within organisation/system: programme manager vs. director vs. medical officer differed in uptake of SRs [22] | ||
Reassurance that no reviews have been missed [52] | Use of stories to help integration come alive for participants so they could see how it lives operationally [49]. Provide section on the relevance of the evidence and the intervention for low and middle income countries (LMICs) [47]. Replacing unfamiliar terms or adding definitions to the re view [47] | Value the organisation placed on using research evidence for decision-making [23] | |||
Make the user-friendly “front end” of systematic reviews available through an online database that could be searched using keywords that make sense to health care managers and policy makers and that is linked to the full reviews when they are available through other sources, such as The Cochrane Library [45]. Provide section on the relevance of the evidence and the intervention for low and middle income countries (LMICs) [47] | Having direct access to online database searching [23] | ||||
Use of stories to help integration come alive for participants so they could see how it lives operationally [49]. Make the user-friendly “front end” of systematic reviews available through an online database that could be searched using keywords that make sense to health care managers and policy makers and that is linked to the full reviews when they are available through other sources, such as The Cochrane Library [45] | Existence of mechanisms to facilitate transfer of new information in health unit [23] | ||||
Reallocate funding away from single study knowledge transfer strategies, fund rapid reviews, more proactive knowledge transfer, health care manager [45] | |||||
Removing jargon and using language that is locally applicable, engage in discussion about the potential implications of the review [45]. It must be packaged to incite and persuade, “to translate the evidence into something that is understandable by the average legislator, average citizen” [44]. Concrete recommendations for practice [31] | Priority of and support for systematic reviews [51] | ||||
It must be packaged to incite and persuade, “to translate the evidence into something that is understandable by the average legislator, average citizen” [44] | Announce priorities to be addressed using SR [51] | ||||
Meeting requestors time constraints [52] | |||||
Consistency in follow-up of individuals using on-demand service to appraise and interpret reviews of research [52] | |||||
Easy to use [31]. Framing the evidence in terms of how they can implement it (specifically as a list of questions to be considered when developing and implementing an integrated health system, information about how to engage stakeholders, build relationships and communicate appropriately across target audiences) [49]. Reassurance that no reviews have been missed [52] | Researchers and policy makers generally found reviews commissioned through evidence check to accurately reflect the state of the evidence, implying that the requirement for rigour and comprehensiveness was not unnecessarily compromised by the rapid timeframe in which the reviews were conducted. It is likely that this is due to both knowledge brokers’ attempts to assist in precisely defining the focus and scope of reviews early in the commissioning process, and researchers’ depth of content knowledge and methodological expertise [41] | ||||
Easy to use [31]. Framing the evidence in terms of how they can implement it (specifically as a list of questions to be considered when developing and implementing an integrated health system, information about how to engage stakeholders, build relationships and communicate appropriately across target audiences) [49] | |||||
Using consistent language and standard phrases to describe effect sizes and the quality of the evidence [47] |
Attitudes
Knowledge
Skills
Behaviour
Format features to facilitate use of systematic reviews
Summary | Dissemination of SRs | Layout, presentation, setup |
---|---|---|
Summary statement [30] | Graded format with key messages [47] | |
1-page summaries in plain language [49] | Provide tailored, targeted messages for relevant audiences [24] | Recipe type guidance, the information indicates this, this, and this [52] |
Abbreviated format of research evidence, such as an executive summary, would be preferable (1 to 2 pages long ) [43] | Electronic communication channels are generally preferred [43] | Title framed as a question [47] |
Expectations of short, clear summary [47] | Newsletters containing summaries of current research developed and directly emailed to managers [43] | Reformatting the text to make it easier to pick out important parts [47] |
Boxes placed throughout the summaries [47] | Chart on first page describing what review is about [47] | |
Summary of findings tables [47] | Reports could be either distributed through professional organisations or through a clearinghouse [43] | A modified academic abstract (relevance and description of review characteristics including the impact, applicability to setting, costs, or other considerations and need for no further evaluation) [47] |
1-page summaries with references, so the reader is able to investigate further, and case studies [49] | Active delivery of information (as opposed to access to online registry) [24] | Preference for less dense, more accessible literature [49] |
Wanted a shorter, clearer presentation [47] | ||
A bullet point evaluation or rating system of study design quality so that for those of us who do not make our living doing that, we do not have to read a half dozen pages to ferret it out [44] | ||
Develop a more user-friendly “front end” for potentially relevant systematic reviews (e.g. 1 page of take-home messages and a 3-page executive summary) to facilitate rapid assessments of the relevance of a review by health care managers and policy makers and, when the review is deemed highly relevant, more graded entry into the full details of the review [45] | ||
Well written and concise [47] | ||
Limiting the number of tables and not letting them break across pages [47] | ||
Simplifying the text and tables and ensuring that the results in the text matched those in the tables [47] | ||
Moving partner logos and the summary publication date to the front page [47] |
Content features to facilitate use of systematic reviews
Decision-making focus | Easy to understand | Details on included studies |
---|---|---|
Address relevant policy questions not academic or business focused questions [44] | Information about the information or meta-information that tells you what to expect [47] | Provide rating scale for quality of study design [44] |
Clearly articulate the implications of the findings to public health practice and policy [43] | Include content that was focused on key findings or the “bottom line” from the study [43] | Include section on the relevance of the evidence and the intervention for low and middle income countries (LMICs) [47] |
Provide potential short- and long-term outcomes expected as a result of implementing the research findings into practice [43] | Provide references to more detailed findings so the reader is able to investigate further if needed [49] | Include table describing the characteristics of the reviews [47] |
Policy makers expect content lying outside the scope of a review: recommendations, outcome measurements not usually included in a review, detailed information about local applicability or costs, and a broader framing of the research enquiry [47] | Lack features that would make them easier for government officials to evaluate. For example, the quality of studies is often difficult for non-experts to interpret because the explanation of research methods is long and complicated [44] | Include critical appraisal of included studies [52] |
Frame the evidence in terms of how they can implement it (specifically as a list of questions to be considered when developing and implementing an integrated health system (which was topic of the review in this study), information about how to engage stakeholders, build relationships, and communicate appropriately across target audiences) [49] | Replace the section for references with a section for “additional information”: information that was helpful for understanding the problem, that provided details about the interventions, or that put the results of the review in a broader context [47] | Include bullet point evaluation or rating system of study design quality so that “for those of us who don’t make our living doing that, we do not have to read a half dozen pages to ferret it out” [44] |
It must be packaged to incite and persuade, “to translate the evidence into something that is understandable by the average legislator, average citizen” [44] | Provide table describing the characteristics of the reviews: makes clear what the review was looking for [47] | |
References are clear [47] | ||
Use familiar, non-jargon language [47] | ||
Use consistent language and standard phrases to describe effect sizes and the quality of the evidence [47] | ||
Limit the discussion of methods [43] |