Background
Measuring research utilization
Study purpose and objectives
Methods
Study selection (inclusion and exclusion) criteria
Search strategy for identification of studies
Selection of Studies
Data Extraction
Data Synthesis
Results
Objective 1: Identification and characteristics of self-report research utilization measures used in healthcare
Class | Description | No of Articles [Citations] |
---|---|---|
Nurses Practice Questionnaire (NPQ) | Developed for nurses. The NPQ consists of brief descriptions of 14 specific practice innovations. Seven questions measuring an individual's stage of innovation adoption are posed for each innovation. The first six questions measure adoption of the practice according to Roger's [91] Innovation-Decision Process Theory while the seventh question measures perception of policy existence with respect to the innovation. All items are scored dichotomously (yes/no) except for one item (on 'use'), which is scored as never, sometimes, or always. | |
Research Utilization Questionnaire (RUQ) | Developed for nurses. The RUQ consists of 42 self-descriptive statements comprising four subscales of which research use is one. The research use subscale contains 10 items, which measure the degree to which an individual feels they incorporate research findings into their daily practice. Items are scored on a 5-point Likert scale from strongly disagree to strongly agree. | |
Edmonton Research Orientation Survey (EROS) | Developed in the context of rehabilitation specialties (e.g., physiotherapy,). The EROS has four subscales of which the 'Using Research/Evidence-Based Practice' is one subscale. This subscale is composed of 10 items. Items are scored on a 5-point Likert scale from strongly disagree to strongly agree. | |
Knott and Wildavsky Standards | Developed for leaders based on Knott and Wildavsky's [93]Standards of Research Use. Consists of seven items to measure each of the seven standards of research use: reception, cognition, reference, effort, adoption, implementation, and impact. Items scored on a 5-point frequency scale from never to very often. | 1 article [20] |
Other Specific Practices Indices | Asks respondents to report on their use of a range of specific research-based practices. The number and kind of practices vary by the study. The scales used to measure use of the practices vary by study with some studies measuring use on frequency scales and others dichotomously as use or nonuse. (See Additional File 4) | |
Other General Research Use Indices | Each of these indices combines several items on respondents' general use of research (i.e., not use of specific practices) to derive an index (or overall score) representing their use of research. (See Additional File 4) | |
Past/Present/Future Use | Developed for nurses. Asks respondents to indicate their participation in one or more research activities in the past (> 6 months ago), present (most recent six months), and intention to use research in the future (within the next year). Responses are scored in a dichotomous yes/no format. Each item is considered individually, that is, items are not combined to form an index score. | |
Parahoo Measure | Developed for nurses. Measures research use with three single items. The three items are: frequency of use of research in clinical practice (scored on a 5-point frequency scale from never to all the time), implementation of new research findings in one's own practice in the last two years (scored dichotomously as yes/no), and to list up to three research findings that they have implemented in the last two years (open ended). Each item is considered individually, that is, items are not combined to form an index score. | |
Estabrooks' Kinds of Research Use | Developed for nurses. Measures research use with single items that tap four kinds of research use: instrumental (or direct), conceptual (or indirect), persuasive, and overall. Each item is preceded by a definition of the kind of research use and examples of that kind of research use. For each kind of research use, respondents are asked to indicate, over the past year, how often they have used research in this way. The items are treated individually (i.e., they are not combined to form an index). Items are scored on a 7-point (from never to nearly every shift) or 4-point (from never to nearly every work day) scale. | (8 studies) |
Other Single Item Measures | Developed for different types of healthcare professionals (depending on the target population of the study). Measures research use with a single-item developed for the study, and not used by others in subsequent studies. A variety of scoring methods are used depending on the study using different frequency scales, Likert agreement scales, dichotomous yes/no scales, and/or open-ended responses. | (39 studies) |
Objective 2: Psychometric assessment of the self-report research utilization measures
Acceptability
Reliability
Class (No. of measures) [Citations] | Reliability | ||
---|---|---|---|
Internal consistency Range | Stability Range | Inter-rater | |
Nurses Practice Questionnaire (1) | α = 0.63 to 0.95 | r = 0.83 to 0.99 | |
Other Specific Practice Indices (1) [50] | α = 0.87 | ||
Research Utilization Questionnaire (1) | α = 0.79 to 0.94 | ||
Edmonton Research Orientation Survey (1) | α = 0.83 to 0.89 | ||
Knott and Wildvasky Standards (1) [20] | α = 0.87 | ||
Other General Research Utilization Indices (8) | α = 0.73 to 0.94 | r = 0.88 | |
Other Single Items (1) [37] | NA | r = 0.80 to 0.91 |
Validity
Class | Citation | Participants/Setting/Country | Validity | |||
---|---|---|---|---|---|---|
Content | Response processes | Internal structure | Relations | |||
Other Specific Practices Indices* | [50] | Nurses/Hospitals/Canada | √ | √ | √ | |
Other General Research Utilization Indices* | [24] | Nurses/Hospitals/USA | √ | √ | √ | |
[36] | Nurses/Hospitals/USA | √ | √ | √ | ||
[50] | Nurses/Hospitals/Canada | √ | √ | √ | ||
Other Single Items* | [51] | Leaders/Community/Canada | √ | √ | NA | √ |
[52] | Allied/Variety/Canada | √ | √ | NA | √ |
Class | Citation | Participants/Setting/Country | Validity | |||
---|---|---|---|---|---|---|
Content | Response processes | Internal structure | Relations | |||
Nurses Practice Questionnaire | Nurses/Hospitals/USA | √ | √ | |||
[59] | Nurses/Education/USA | √ | ||||
[32] | Nurses/Variety/Sweden | √ | √ | |||
[60] | Nurses/Hospitals/USA | √ | √ | |||
[61] | Nurses/Hospitals/USA | √ | √ | |||
Nurses/Hospitals/UK | √ | √ | √ | |||
[62] | Nurses/Variety/USA | √ | √ | |||
[63] | Nurses/Hospitals/Canada | √ | √ | |||
[35] | Nurses/Hospitals/USA | √ | ||||
Knott and Wildvasky | [20] | Leaders/Variety/Canada | √ | √ | ||
Other General Research Utilization Indices* | [101] | Nurses/Hospitals/USA | √ | √ | ||
[102] | Allied/Community/USA | √ | √ | |||
[105] | Allied/Variety/Sweden | √ | √ | |||
[104] | Allied/Not Reported/USA | √ | √ | |||
Other Specific Practices Indices* | [21] | Leaders/Community/USA | √ | √ | ||
Multiple Staff/Hospitals/Africa | √ | √ | ||||
Estabrooks' Kinds of Research Use | Nurses/Variety/Canada | √ | √ | NA | √ | |
[74] | Nurses/Hospitals/USA | NA | √ | |||
[69] | Nurses/Hospitals/USA, Canada | NA | √ | |||
Nurses/Hospitals/Canada | √ | NA | ||||
[66] | Allied/Variety/Canada | √ | NA | |||
[67] | Nurses/Long-term Care/USA | √ | NA | √ | ||
[80] | Nurses/Variety/Canada | NA | √ | |||
[86] | Nurse Educators/Variety/Canada | NA | ||||
Parahoo | Nurses/Hospitals/UK | √ | √ | NA | ||
[135] | Nurses/Hospitals/Iran | NA | ||||
[136] | Nurses/Variety/UK | √ | √ | NA | ||
Other Single Items* | [139] | Allied/Variety/UK | √ | NA | √ | |
[144] | Nurses/Hospitals/Nigeria | √ | NA | √ | ||
[83] | Nurses/Variety/USA | √ | √ | NA | ||
[27] | Nurses/Hospitals/Canada | √ | NA | √ | ||
[88] | Nurses/Hospitals/USA | √ | √ | NA | ||
[57] | Nurses/Hospitals/Taiwan | √ | √ | NA |
Class | Citation | Participants/Setting/Country | Validity | |||
---|---|---|---|---|---|---|
Content | Response processes | Internal structure | Relations | |||
Research Utilization Questionnaire | [55] | Nurses/Hospitals/USA | √ | √ | ||
Multiple Staff/Long-Term Care/Sweden | √ | |||||
[120] | Nurses/Long-Term Care/Sweden | √ | ||||
[71] | Multiple Staff/Variety/USA | √ | ||||
[72] | Nurses/Hospitals/Canada | √ | ||||
[121] | Nurses/Hospitals/UK | √ | ||||
[122] | Nurses/Hospitals/UK | |||||
[75] | Nurses/Hospitals/UK | √ | ||||
Nurses/Hospitals/USA | √ | |||||
[81] | Nurses/Variety/USA | √ | ||||
[82] | Allied/Variety/Sweden | √ | ||||
[85] | Nurses/Hospitals/USA | √ | √ | |||
[124] | Nurses/Hospitals/Canada | √ | ||||
[89] | Nurses/Variety/Sweden | |||||
Edmonton Research Orientation Survey | [125] | Allied/Hospitals/Canada | √ | |||
[126] | Nurses/Variety/Australia | √ | √ | |||
[127] | Multiple Staff/Hospitals/Australia | |||||
Nurses/Hospitals/Canada | √ | |||||
[76] | Allied/Hospitals/Canada | |||||
[37] | Allied/Variety/Canada | √ | ||||
[128] | Allied/Variety/Canada | |||||
Other General Research Utilization Indices* | [73] | Allied/Variety/Sweden | √ | |||
[103] | Allied/Variety/USA | √ | ||||
[84] | Nurses/Hospitals/USA | √ | ||||
Other Specific Practices Indices* | [129] | Allied/Variety/USA | √ | |||
Past, Present, Future Use | [130] | Nurses/Variety/USA | NA | √ | ||
[65] | Nurses/Hospitals/Canada | NA | √ | |||
[90] | Nurses/Hospitals/USA | NA | √ | |||
Other Single Items* | [23] | Leaders/Community/Canada | √ | NA | ||
[137] | Physicians/Variety/Australia | √ | NA | |||
[138] | Allied/Variety/USA | √ | NA | |||
[140] | Nurses/Flight Team/USA | NA | √ | |||
[28] | Allied/Variety/USA | NA | √ | |||
[141] | Nurses/Variety/USA | √ | NA | |||
[22] | Leaders/Variety/USA | √ | NA | |||
[142] | Allied/Community/USA | √ | NA | |||
[25] | Allied/Hospitals/Europe | √ | NA | |||
[143] | Multiple Staff/Variety/USA | NA | √ | |||
[145] | Physicians/Hospitals/Denmark | NA | √ | |||
[37] | Allied/Variety/Canada | √ | NA | |||
[54] | Nurses/Hospitals/USA | √ | NA | |||
[87] | Allied/Variety/USA | √ | NA | |||
[146] | Allied/Variety/UK | √ | NA | |||
[58] | Nurses/Hospitals/Taiwan | √ | NA | |||
[147] | Nurses/Variety/UK | √ | NA | |||
[148] | Physicians/Variety/USA | √ | NA | |||
[149] | Nurses/Variety/Australia | √ | NA |
Measures reporting three sources of validity evidence (level one)
Class [Citation] | Research Utilization Measure Details | Sample and Setting | Validity Assessment | |
---|---|---|---|---|
Supporting Evidence | Comments | |||
Specific Practice Indices [50] | Use of 10 specific research practices. Sample practices include: • IM injection • Catheter removal • Sensory information/diagnostic Each practice was scored on a 3 pt scale: never (1), sometimes (2), always (3) or 'not applicable.' A mean score based on the ten practices was then calculated. | Population: Nurses Country: Canada Setting: Hospitals | Content: Measure assessed by an expert panel Response processes: a pilot test was conducted within a larger survey (of which the research utilization index was one component). Relations to other variables: correlations with other variables were reported that support theory and prior empirical research (e.g., with supportive climate and infrastructure) | Content: Unknown whether content assessment was on the specific research-based practices, the question pertaining to use that followed each practice, or both. A high quality assessment of content evidence should include both. |
General Research Utilization Indices [50] | Research use index contains 10 general statements on research use. Sample items include: • Communicating concerns about the effectiveness of practices to colleagues • Use of research articles to support questioning practice • Identification of hospital policies based on research Each item is scored on a 4-point scale from not at all to always. Item scores are then summed for an index score (10 to 40). | Population: Nurses Country: Canada Setting: Hospitals | Content: Measure assessed by a peer panel Response processes: a pilot test was conducted with the larger survey (of which the research utilization index was one component). Relations to other variables: Non-significant correlations (as predicted) with other variables (education and valuing research) which support past empirical reviews. | Content: Process or findings of the content assessment not reported. |
General Research Utilization Indices [24] | Research use index consists of five items focusing on the extent to which respondents participate in research activities. Sample items include: • Reviewed research literature in an effort to identify new knowledge for use in your practice • Evaluated a research study to determine its value for practice Each item is asked with respect to the past year and is scored on a 4-point scale: 0, 1, 2-4, 5 or more times. Mean of the items are then taken as a measure of research utilization. | Population: Nurses Country: USA Setting: Hospitals | Content: Development of the research utilization index was based on a set of five rules (See Additional File 4). Response processes: a small pretest was conducted with the larger survey (of which the research utilization index was one component). Relations to other variables: Covariance analysis reported. Several variables were shown to be nonsignificant as predicted, for example, professionalism. | |
General Research Utilization Indices [36] | Research use index consists of 18 items measuring respondents' reported participation in nursing research utilization activities. Sample items include: • I read nursing research articles and learn about research-based nursing interventions. • I attend conferences/educational programs and learn about research-based nursing interventions Each item is scored on a 5-point scale from never to always. Item scores are then summed for an index score (18 to 90). | Population: Nurses Country: USA Setting: Hospitals | Content: A panel of four experts on research use by nurses assessed the index. Reasons for selecting each panel member were reported, illustrating the appropriateness of the panel selection. Internal structure: Factor analysis was conducted; findings revealed a 3-factor solution. Relations to other variables: A significant association between specialty (working in critical care settings) and research use was reported (as predicted). | Content: Findings from the content assessment were not reported. Internal structure: The 18 items were combined to compute one derived research utilization score (but factor analysis revealed three factors and thus supported deriving three scores and not one score). |
Other Single-Items [51] | Five single items asking respondents (decision-makers) whether they have used five specific systematic reviews in the past two years to make a program-related decision. All five items are scored as yes or no. Each item is analyzed as an individual item. | Population: Decision-Makers Country: Canada Setting: Community | Content: The research utilization item, which was a component of a larger survey, was developed based on a review of research utilization literature, suggesting content validity evidence. Response processes: a pilot test was conducted with the larger survey (of which the research utilization item was one component). Relations to other variables: correlations with other variables, for example, perception that the systematic reviews are easy to use. | All applicable sources of validity evidence reported |
Other Single-Items [52] | A single item asking respondents whether they have applied research to their practice. Scored on a 4-point Likert scale: never, rarely, sometimes, always | Population: Allied Health Professionals Country: Canada Setting: Variety of settings | Content: An expert panel assessed the research utilization item, which was a component of a larger survey. Response processes: A pilot test was conducted with the larger survey (of which the research utilization item was one component). Relations to other variables: a significant association with attitude towards research (as predicted). | All applicable sources of validity evidence reported. Content: The composition of the panel, process undertaken, or related findings were not reported. |