What factors affect evidence-informed policymaking in public health? Protocol for a systematic review of qualitative evidence using thematic synthesis

This review will only include qualitative studies that report primary data. For the purposes of this review, we will use the following definition of ‘qualitative study’: a study that uses qualitative methods both for data collection and data analysis. This definition is consistent with that used in several recent qualitative syntheses [15‐17] and was cited as one useful definition in the Cochrane Qualitative and Implementation Methods Group supplementary guidance on qualitative evidence synthesis [18]. Methods of qualitative data collection include (but are not limited to) interviews, focus groups and participant observation methods, while methods of qualitative data analysis include, for example, thematic analysis, content analysis, discourse analysis and grounded theory approaches. This definition of qualitative study excludes, for example, studies in which data were collected through interviews or focus groups, but were analysed exclusively through quantitative methods (e.g. tallies, descriptive statistics, etc.). We will include mixed methods studies that used both qualitative and quantitative methods, provided that it is possible extract data derived only from the qualitative methods.

We will not exclude studies according to the epistemological assumption(s) and/or theoretical tradition(s) on which they were based. That is, we will include all work within the broad ‘qualitative paradigm’.

Studies will not be excluded from the review on the basis of any hierarchy of qualitative evidence or criteria related to study quality. However, we recognise the possibility that including all eligible qualitative studies regardless of methodological quality may risk biasing the review’s findings [19]. As described below, a quality appraisal tool will be applied to all included studies prior to data analysis. Using sensitivity analysis (also discussed below), we will assess the potential undue influence of low scoring studies on our findings [20].

Studies will not be excluded on the basis of publication status (e.g. type of publication), publication date or the language in which the study was reported.

This review will include studies reporting on policy activities, decisions, and/or processes with an explicit (though not necessarily exclusive) focus on public health issues. Studies reporting data derived from human participants (i.e. via interviews, focus groups or observation, as opposed to documents) will be included in the review if their participants included policymakers engaged in such activities. There is no universally accepted definition of the population ‘policymakers’ [21, 22]. For the purposes of this review, the population ‘policymakers’ includes government officials of any rank who are either elected (i.e. politicians) or appointed (i.e. civil servants, policy advisors and/or bureaucrats), working at the sub-national (e.g. local, state, provincial), national or international/global (i.e. intergovernmental) levels. Consistent with previous reviews [23, 24], we distinguish policymakers from managers (i.e. program managers, healthcare executives, management consultants, with supervisory and management responsibilities in healthcare and public health organisations) and service providers (i.e. front-line practitioners who typically make decisions about individual patient care, including physicians and nurses). Studies that exclusively include managers and/or service providers will be excluded.

If there is any lack of clarity within study reports about the role(s) of study participants and, as a consequence, uncertainty about whether they fit this definition of ‘policymakers’, we will contact original study authors for clarification.

Unlike in previous reviews of evidence use in public health [10] and non-health sectors of policy [25], studies will not be excluded on the basis of the country in which the study was conducted. The rationale for this inclusiveness is to capture the potential influence on evidence use of factors related to the political, economic and social context in which policymaking activities take place.

To be eligible for inclusion, studies must explore the use or reported use of some type of research evidence in policy processes and/or decisions related at least in part to public health.

For the purposes of this review, public health policy decisions are those taken with the explicit goal of promoting the health of the population (whether at the sub-national, national or international level). This excludes policy decisions related to the provision of individual-level clinical interventions, unless these have an explicitly stated public health goal (as in the case of most immunisation policies). However, recognising the now widely appreciated importance of the social determinants of health [26] and understanding that policy decisions made outside of ministries and departments of public health, across a variety of sectors (e.g. transport, housing, crime), can have meaningful impacts on health [27], studies of policymakers with non-health portfolios will be included if population health, or the relationship between their decisions and public health outcome(s), is an explicit focus of the research.

For this review, research evidence will be defined as research produced by academic researchers and/or published in academic journals. This definition is similar to that used in a recent systematic review of evidence use in non-health settings [25], whose authors found that their original attempt to use a broader definition of research evidence produced results so conceptually heterogeneous that a meaningful synthesis was unfeasible. Included studies may examine the use of research evidence in general, or a specific category or ‘form’ of research evidence, including, for example, reports of randomised controlled trials, systematic reviews and/or meta-analyses and evidence summaries or overviews of reviews.

Study screening and selection will be conducted according to standard systematic review methods [30] using EndNote X7 software (Thomson Reuters). Two reviewers will independently screen all titles and abstracts. Full-text versions of records deemed potentially relevant by at least one reviewer will be obtained for further review. Both reviewers will then independently screen the full-text versions of all potentially relevant articles for inclusion in the review. Any disagreements will be resolved through conference and, if necessary, deferral to a third reviewer. Reasons for the exclusion of studies at the full-text review stage will be recorded, tabulated and provided in an appendix to the final review article.

There is no universally accepted approach to extracting data for the purposes of qualitative metasynthesis. Strategies vary from the very selective to the very inclusive. In extremely inclusive approaches, the entire texts of included papers are essentially treated as data, while in more selective approaches, findings are only extracted from included studies when they are explicitly supported by direct quotations from study participants in the text of the article [18]. This latter approach risks missing findings that on the individual study level are perhaps of secondary importance, but collectively may emerge as important recurrent themes across studies. We will therefore adopt an approach emphasising inclusiveness, since this review is interested in the full breadth of factors affecting evidence use in public health policy.

Since the ‘informants’ in qualitative reviews are the original study authors (not the studies’ participants), all author interpretations of study results (in the form of themes, categories, diagrams, tables, etc.) qualify as data for this review [15, 18]. Adopting a version of Thomas and Harden’s [13] approach, we will extract all data labelled by study authors as results/findings, etc. and discussion/conclusion(s)/interpretation(s), etc. Data will be extracted verbatim from study papers directly into NVivo-11 software (QSR International).

In addition to extracting results and discussion sections into NVivo, the following information will be recorded for each included study:

BV will extract these data from all included studies using a tabular data extraction form in Microsoft Excel; a second reviewer will assess the extracted data for accuracy against original study reports. Discordant interpretations will be discussed and resolved. Persistent disagreements will be resolved in consultation with PM. These descriptive study-level data will be reported in a summary table included in the final report and will be used in exploratory sub-group analysis (discussed below).

Study quality in this review will be assessed according to an adapted version of the Critical Appraisal Skills Programme’s (CASP) tool for appraising qualitative research [31]. CASP is arguably the most ‘user-friendly’ of the widely used tools; however, in a comparative assessment of three popular critical appraisal tools, CASP was found wanting in terms of sensitivity to descriptive, interpretive and theoretical validity [32]. To address these limitations, we have augmented the CASP tool with four items adapted from the Joanna Briggs Institute Qualitative Assessment and Review Instrument [33]. The adapted CASP instrument (Additional file 2) contains 12 items, all of which can be answered with either ‘Yes’, ‘No’, or ‘Unclear’. For the purpose of aiding interpretation of the review findings, included studies will be assigned an overall score of ‘high’, ‘moderate’ or ‘low’ overall methodological quality according to a holistic reading of included papers, guided by consideration of these 12 items. It is important to note that these items are not designed to yield an overall numerical ‘score’ for the methodological quality of studies. Rather, the questions are designed as prompts to guide the reviewers in a critical reading of the studies. Indeed, as there is no consensus on the relative weight that should be ascribed to any individual characteristic of study quality, the presentation of a simple summed score of the tool’s items would risk being more misleading than informative.

BV and a second reviewer will independently pilot the adapted CASP instrument on a random sample of five included studies. The results of the two reviewers’ assessments will be compared and any discordant interpretations of items will be discussed with the broader review team to establish a consensus moving forward. Following the pilot stage, BV and the second reviewer will independently assess the quality of all included studies. Disagreements will be discussed and resolved. Persistent disagreements will be resolved by consulting PM. Quality ratings assigned to each study, and the rationale for these decisions (including any disputes and how they were resolved), will be explicitly detailed in an appendix to the final report.

As mentioned above, the results of the quality assessment will not be used to exclude studies from the review, but quality ratings will be used to inform interpretation of the data [34] and will be used in concert with the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) tool to assess the confidence in the review’s findings (discussed below).

As well, post hoc sensitivity analysis will be undertaken to assess the potential influence of study quality on the review’s findings. This approach is analogous to the sensitivity analyses commonly conducted in quantitative reviews [20, 35]. We will estimate the robustness of the review’s findings to the inclusion/exclusion of studies receiving a ‘low quality’ rating to test for the potential undue influence of severely methodologically flawed studies [36].

Data will be analysed using NVivo-11 software according to Thomas and Harden’s [13] method of qualitative thematic synthesis. Thematic synthesis involves the line-by-line coding of the text of included studies to produce so-called descriptive themes within studies, followed by the re-interpretation and synthesis of these newly organised data across studies to produce higher-order ‘analytical themes’ that ‘go beyond’ the findings of the individual primary studies. This potential generation of new theory via the production of higher order themes represents a synthesis step hitherto not attempted in reviews of evidence use in public health policymaking. No themes will be specified a priori to guide the review. Rather, themes will be allowed to emerge inductively as we interpret individual study data and synthesise data across studies.

The synthesis procedure will follow the three stages of thematic synthesis described by Thomas and Harden [13], the first two of which take place concurrently: (1) coding text, (2) developing descriptive themes and (3) developing analytical themes.

During the first and second stages, the data from each included study will be considered in isolation. BV will read and re-read the text of each study in NVivo-11 and will develop codes to describe the meaning and content of the text line by line. Sections of text (e.g. sentences) may be assigned a single code or multiple codes. As codes emerge and accumulate during this process, BV will organise them hierarchically, as appropriate, to develop descriptive themes to explain the data. During this stage, all of the text assigned to each code will periodically be checked for interpretive consistency to determine whether new codes are necessary and to contemplate whether some codes should be collapsed. Throughout this process, other team members will be consulted regularly to discuss coding decisions and the validity of the emerging list of descriptive themes, and amendments will be made as necessary.

During the third stage of thematic synthesis, ‘higher order’ analytical themes are developed from critical examination of the aggregated descriptive themes developed in stage two [13]. At this point in the analysis, BV will reconsider the set of data, now coded across studies and organised into descriptive themes, and interrogate it for newly emerging cross-study themes. As in stage two, other team members will be consulted regularly to discuss the appropriateness of decisions related to the assignment of analytical themes. Following consultations, amendments to themes will be made as necessary, and the data will be revisited and considered in the context of any newly emergent themes. This iterative process will continue until examination of the data ceases to yield new analytical themes.

As the synthesis process progresses, interpretations of the emerging themes will be summarised as a taxonomy of ‘factors’ affecting evidence use in public health policymaking, categorised according to the analytical themes. As new findings emerge from the interrogation of the data, the list of factors will be amended as appropriate.

An auditable ‘decision trail’ tracking the supporting text for each theme, and the study/ies from which this text was drawn, will be recorded in NVivo throughout the process of data synthesis. Tracking (and transparently reporting) the studies that contribute to each ‘finding’ of the review will facilitate confidence assessments and subgroup analysis (both discussed below) and will enhance the ease with which the review can be critiqued and reproduced.

The CERQual approach will be used to provide an assessment of the level of confidence (certainty) in each of the review’s findings [37]. In the context of qualitative evidence synthesis, ‘confidence’ refers to the extent to which a review finding can be considered a reasonable representation of the phenomenon of interest; for example, an assessment of high confidence would signify that the phenomenon of interest is unlikely to differ substantially from the review finding. The CERQual tool, which is currently under development, draws on the principles used to develop the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach [38]. It emerged in 2010 to assist a World Health Organization guideline development panel in using synthesised qualitative evidence to inform their recommendations. At least six other reviews [15, 39‐43] have utilised the CERQual approach or an earlier version of it.

Confidence is assessed at the level of the individual ‘review finding’; that is, through the process described below, each ‘finding’ from the synthesis will be assigned a confidence rating. The current version of the CERQual tool assesses confidence in the evidence on the basis of (1) the methodological limitations of the studies that contributed to each review finding, (2) the relevance of the included studies to the review question, (3) the coherence of each review finding and (4) the adequacy of the data contributing to each review finding. The methodological limitations of included studies refer to the extent to which there are problems in the design or conduct of the primary studies that contributed evidence to a review finding. The relevance of the included studies to the review question refers to the extent to which the review finding is applicable to the context (i.e., the population, setting and phenomenon) in which the review is interested. The coherence of a review finding refers to the extent to which the ‘pattern’ that constitutes a review finding is consistent across multiple individual studies (or, alternatively, incorporates explanations for variations in the finding across individual studies). The adequacy of data refers to the richness (and ‘thickness’) and quantity of data supporting a given review finding.

CERQual requires that review authors form what are ultimately subjective judgements based on criteria that are somewhat open to interpretation: there are no straightforward, algorithmic criteria upon which to rate studies in these four domains. For the purposes of this review, assessments will be made as follows:

After assessing each of the four components, we will make a judgement about the overall confidence in each review finding. Findings will be assessed as having ‘high’, ‘moderate’, ‘low’ or ‘very low’ confidence. The developers of CERQual define these four levels of confidence as follows:

BV and a second reviewer will independently apply the CERQual tool to the review findings. The results of both assessments will be compared and discordant assessments discussed until consensus is achieved for each finding. Any persistent disagreements on confidence ratings will be resolved in consultation with PM. The final results of the CERQual assessment will be reported in tabular form (discussed below) and will be incorporated into the narrative explanation of review’s results.

As appropriate, we will perform subgroup analysis to explore possible variation in the review’s findings according to three categories of study-level characteristic:

Examining differences in findings according to the qualitative methods used in included studies may help to inform the design of future work in this area. For instance, it is possible that more political and/or less socially desirable characteristics of evidence use or non-use are more difficult to capture using methods of self-report (i.e. individual or group interviews) than through analysis of policy documents or participant observation methods. The content of our findings derived from each of these methods may shed light on such discrepancies.

Exploring differences in findings across contexts could improve our understanding of the nature of public health evidence use at different governance levels and in different economic contexts and could help to inform the specification of interventions to encourage evidence-informed policymaking for different types of policymakers. These results may be also used to generate hypotheses about context-specific evidence use for future quantitative studies in this area. For example, these results could inform subgroup analyses in future reviews of the effectiveness of knowledge translation interventions for policymakers.

The study design features, participant and setting characteristics, data collection and analysis methods and reported findings of all included studies will be summarised in tabular form and described narratively.

Results of the thematic synthesis and of the CERQual assessments will be tabulated in a ‘summary of qualitative findings table’. The review findings (factors affecting evidence use) will be classified into categories, informed by the analysis and listed in a table. A brief description of each factor will be provided, along with each factor’s ‘confidence’ rating. This exercise will produce an empirically derived taxonomy of factors related to evidence use in public health policy that can be interrogated and further refined in future empirical work (both qualitative and quantitative). The review’s findings will also be summarised visually in a proposed conceptual framework explaining the relationship between the key factors affecting evidence use.

This protocol was developed and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)-P Statement for reporting systematic review protocols [44]. See Additional file 3 for a completed PRISMA-P checklist. The review’s methods and results will be reported according to the ENTREQ (enhancing transparency in reporting the synthesis of qualitative research) statement for reporting syntheses of qualitative studies [45]. The final literature searches employed will be reported using the STARLITE (sampling strategy, type of study, approaches, range of years, limits, inclusion and exclusions, terms used, electronic sources) standards for reporting literature searches [46], and search and screening results will be summarised and presented using the flowchart described in the PRISMA statement [47]. Any deviations from this protocol will be recorded by BV and reported in the final review report.