Skip to main content
main-content

01.12.2017 | Research | Ausgabe 1/2017 Open Access

Implementation Science 1/2017

Preliminary testing of the reliability and feasibility of SAGE: a system to measure and score engagement with and use of research in health policies and programs

Zeitschrift:
Implementation Science > Ausgabe 1/2017
Autoren:
Steve R. Makkar, Anna Williamson, Catherine D’Este, Sally Redman
Wichtige Hinweise

Electronic supplementary material

The online version of this article (https://​doi.​org/​10.​1186/​s13012-017-0676-7) contains supplementary material, which is available to authorized users.

Abstract

Background

Few measures of research use in health policymaking are available, and the reliability of such measures has yet to be evaluated. A new measure called the Staff Assessment of Engagement with Evidence (SAGE) incorporates an interview that explores policymakers’ research use within discrete policy documents and a scoring tool that quantifies the extent of policymakers’ research use based on the interview transcript and analysis of the policy document itself. We aimed to conduct a preliminary investigation of the usability, sensitivity, and reliability of the scoring tool in measuring research use by policymakers.

Methods

Nine experts in health policy research and two independent coders were recruited. Each expert used the scoring tool to rate a random selection of 20 interview transcripts, and each independent coder rated 60 transcripts. The distribution of scores among experts was examined, and then, interrater reliability was tested within and between the experts and independent coders. Average- and single-measure reliability coefficients were computed for each SAGE subscales.

Results

Experts’ scores ranged from the limited to extensive scoring bracket for all subscales. Experts as a group also exhibited at least a fair level of interrater agreement across all subscales. Single-measure reliability was at least fair except for three subscales: Relevance Appraisal, Conceptual Use, and Instrumental Use. Average- and single-measure reliability among independent coders was good to excellent for all subscales. Finally, reliability between experts and independent coders was fair to excellent for all subscales.

Conclusions

Among experts, the scoring tool was comprehensible, usable, and sensitive to discriminate between documents with varying degrees of research use. Secondly, the scoring tool yielded scores with good reliability among the independent coders. There was greater variability among experts, although as a group, the tool was fairly reliable. The alignment between experts’ and independent coders’ ratings indicates that the independent coders were scoring in a manner comparable to health policy research experts. If the present findings are replicated in a larger sample, end users (e.g. policy agency staff) could potentially be trained to use SAGE to reliably score research use within their agencies, which would provide a cost-effective and time-efficient approach to utilising this measure in practice.
Zusatzmaterial
Additional file 1: SAGE Interview. (PDF 250 kb)
13012_2017_676_MOESM1_ESM.pdf
Additional file 2: Customised Transcript Template. (DOCX 21 kb)
13012_2017_676_MOESM2_ESM.docx
Additional file 3: SAGE scoring checklist. (XLSX 42 kb)
13012_2017_676_MOESM3_ESM.xlsx
Additional file 4: SAGE scoring instructions. (PDF 709 kb)
13012_2017_676_MOESM4_ESM.pdf
Additional file 5: Mean document scores and standard errors for all nine expert raters on each of the ten SAGE domains. (DOCX 162 kb)
13012_2017_676_MOESM5_ESM.docx
Additional file 6: Histograms for the distribution of policy document scores on each of the ten SAGE domains. (DOCX 52 kb)
13012_2017_676_MOESM6_ESM.docx
Literatur
Über diesen Artikel

Weitere Artikel der Ausgabe 1/2017

Implementation Science 1/2017 Zur Ausgabe