Background
The importance of measuring research engagement actions and existing measures
SAGE: A new measure of research engagement actions
Term | Definition | Example |
---|---|---|
Research engagement actions | Actions undertaken by policymakers to acquire, appraise, and generate relevant and high-quality research evidence or information to inform policymaking. | Examples of research engagement actions include 1) searching for and 2) obtaining research, 3) appraising the relevance and 4) quality of research, 5) generating new research or data analyses, and 6) interacting with researchers to acquire research-related information. The SAGE scoring tool addresses these six research engagement actions. |
Subactions | Subactions† are the essential features or main actions of each research engagement action. They often refer to broad, concrete example actions of undertaking each research engagement action. Each research engagement has a number of subactions that were identified through examination of literature on evidence-informed policymaking and interviews with policymakers. | Examples of subactions of searching for research include a) searching academic literature databases or libraries; b) searching sources of grey literature; c) identifying research by chance, using on-hand research, or research being provided by colleagues; d) seeking out experts to search for relevant research; e) searching for research in search engines or social media sharing sites; and f) examining reference lists, citation indices or databases of references. |
†In order to enhance clarity and comprehension throughout the paper, we used the term subaction instead of attribute, which is most commonly used in choice studies and conjoint analysis. | ||
Level | Levels in conjoint analysis refer to all the possible values of a subaction and are often described in concrete terms. To undertake a conjoint analysis, each subaction must be divided into concrete, perceptible levels. In the present study, the majority of subactions were divided into two levels: i) yes, the subaction was performed by the policymaker, or ii) no, it was not performed by the policymaker. Different levels of subactions are combined in various combinations to form ‘profiles’. | As above, one of the subactions of searching for research was ‘searching academic literature databases’. This subaction has two levels: i) yes, the policymaker searched for research in academic literature databases, or ii) no, the policymaker did not search academic literature databases. |
Profile | A research engagement action profile is made up of a combination of subaction levels. Specifically, a profile consists of one level of each subaction within that research engagement action. | Using the research engagement action – searching for research, an example profile would be: a.ii) yes, research was searched for in academic literature databases (e.g., MEDLINE) or libraries; b.i) no, research was not searched for in sources of grey literature (e.g., OpenGREY); c.ii) yes, research was identified by chance or by colleagues; d.i) no, research was not identified by experts (researchers, working groups, librarians, or other research experts); e.ii) yes, research was searched for in search engines (e.g., Google) or social media sharing sites (e.g., Research Gate); f.i) no, reference lists, citation indices (e.g., Web of Science), or databases of references were not examined (e.g., EndNote). |
Developing a system to score research engagement actions
Using conjoint analysis to develop a system to score research engagement actions
Aim of the present study
Methods
Ethics
Respondents
Procedure
Defining the subactions and levels
Research engagement action | Subaction | Levels of each subaction | Raw utility coefficient | Standard error | Rescaled utility coefficient† |
---|---|---|---|---|---|
1. Searching for research | a. Policymaker searched academic literature databases and/or physical libraries | i) No | −3.59*** | 0.27 | 0 |
ii) Yes | 0 | – | 2.83 | ||
b. Policymaker searched grey literature sources | i) No | −1.81*** | 0.16 | 0 | |
ii) Yes | 0 | – | 1.42 | ||
c. Policymaker obtained research by chance, research that was on-hand, or provided by colleagues | i) No | −1.37*** | 0.11 | 0 | |
ii) Yes | 0 | – | 1.08 | ||
d. Policymaker requested experts (researchers, working groups, librarians, or other research experts) to identify research | i) No | −1.98*** | 0.16 | 0 | |
ii) Yes | 0 | – | 1.56 | ||
e. Policymaker searched generic databases or search engines | i) No | −1.12*** | 0.11 | 0 | |
ii) Yes | 0 | – | 0.88 | ||
f. Policymaker examined reference lists, citation indices, or databases of references | i) No | −1.55*** | 0.14 | 0 | |
ii) Yes | 0 | – | 1.22 | ||
2. Research obtained and used | a. Policymaker found systematic reviews and/or meta-analyses | i) No | −1.94*** | 0.24 | 0 |
ii) Yes | 0 | – | 3.29 | ||
b. Policymaker found books and/or technical monographs | i) No | −0.34*** | 0.12 | 0 | |
ii) Yes | 0 | – | 0.58 | ||
c. Policymaker found primary research and/or theoretical research | i) No | −0.99*** | 0.19 | 0 | |
ii) Yes | 0 | – | 1.67 | ||
d. Policymaker found unpublished research and/or conference resources | i) No | −0.49*** | 0.13 | 0 | |
ii) Yes | 0 | – | 0.82 | ||
e. Policymaker found internal policies, evaluations, or data | i) No | −0.23* | 0.10 | 0 | |
ii) Yes | 0 | – | 0.39 | ||
f. Policymaker found policies, evaluations, or data from external organisations or registries | i) No | −0.60*** | 0.11 | 0 | |
ii) Yes | 0 | – | 1.01 | ||
g. Policymaker obtained recent (up-to-date) research from the above categories | i) No (Older research) | −0.73*** | 0.15 | 0 | |
ii) Yes (Recent) | 0 | – | 1.24 | ||
3. Appraising relevance | a. Policymaker assessed whether the research was applicable to the policy context or policy issue | i) No | −1.70*** | 0.17 | 0 |
ii) Yes | 0 | – | 2.06 | ||
b. Policymaker assessed whether research recommendations were actionable and/or feasible? | i) No | −1.53*** | 0.18 | 0 | |
ii) Yes | 0 | – | 1.84 | ||
c. Policymaker assessed whether the research was consistent with previous research on the issue | i) No | −1.02*** | 0.15 | 0 | |
ii) Yes | 0 | – | 1.23 | ||
d. Policymaker assessed if research was compatible with his/her OR the organisation’s values, knowledge, or experience | i) No | −0.97*** | 0.14 | 0 | |
ii) Yes | 0 | – | 1.17 | ||
e. Policymaker consulted experts to assess the relevance of research | i) No | −1.07*** | 0.12 | 0 | |
ii) Yes | 0 | – | 1.29 | ||
f. Policymaker undertook these actions as part of a pre-specified strategy | i) No (ad-hoc, unplanned) | −1.16*** | 0.18 | 0 | |
ii) Yes | 0 | – | 1.40 | ||
4. Appraising quality | a. Policymaker assessed whether the design or conclusions of the research were valid | i) No | −1.16*** | 0.22 | 0 |
ii)Yes | 0 | – | 2.00 | ||
b. Policymaker evaluated whether the design or conclusions of the research were described clearly and comprehensively | i) No | −0.68*** | 0.16 | 0 | |
ii) Yes | 0 | – | 1.17 | ||
c. Policymaker assessed whether the source of the research was credible | i) No | −0.64*** | 0.12 | 0 | |
ii) Yes | 0 | – | 1.10 | ||
d. Checked if the research cited, or was referenced in other high-quality research or policy documents | i) No | −0.45** | 0.18 | 0 | |
ii) Yes | 0 | – | 0.77 | ||
e. Policymaker consulted experts to assess quality | i) No | −0.76*** | 0.19 | 0 | |
ii) Yes | 0 | – | 1.31 | ||
f. Policymaker assessed the level of evidence of the research | i) No | −0.88*** | 0.17 | 0 | |
ii) Yes | 0 | – | 1.51 | ||
g. Policymaker undertook these actions as part of a pre-specified strategy | i) No (ad-hoc, unplanned) | −0.67*** | 0.22 | 0 | |
ii) Yes | 0 | – | 1.15 | ||
5. Generating new researchers | a. Policymaker expressed explicit intentions to generate or commission new research (to follow-up the current policy) OR stated that he/she had already undertaken this research | i) No (no intentions to generate new research) | −2.08*** | 0.28 | 0 |
ii) No (uncertain intentions only) | −1.97*** | 0.27 | 0.18 | ||
iii) Yes | 0 | – | 3.42 | ||
b. Policymaker mentioned thorough research generation activities | i) No | −1.72*** | 0.19 | 0 | |
ii) Yes | 0 | – | 2.84 | ||
c. Policymaker mentioned less intensive research activities | i) No | −0.96*** | 0.14 | 0 | |
ii) Yes | 0 | – | 1.58 | ||
d. Policymaker advocated for future research to be undertaken | i) No | −0.60*** | 0.15 | 0 | |
ii) Yes | 0 | – | 0.99 | ||
6. Interacting with researchers | a. Policymaker engaged in thorough collaborative activities with researchers | i) No | −2.56*** | 0.25 | 0 |
ii) Yes | 0 | – | 3.75 | ||
a. Policymaker engaged in less intensive interactions with (other) researchers | i) No | −0.91*** | 0.11 | 0 | |
ii) Yes | 0 | – | 1.33 | ||
b. Policymaker engaged in sporadic contact with (other) researchers? | i) No | −0.67*** | 0.11 | 0 | |
ii) Yes | 0 | – | 0.98 | ||
c. Policymaker actively initiated these interaction activities | i) No | −2.01*** | 0.22 | 0 | |
ii) Yes | 0 | – | 2.94 |
The experimental design
Eliciting preferences
Data analyses
Results
Respondent characteristics
Working role | Total | ||||||
---|---|---|---|---|---|---|---|
Policymaker | Researcher | Both researcher and policymaker | Other | ||||
Sex | Male | Count | 6 | 9 | 6 | 4 | 25 |
% of total | 8.7% | 13.0% | 8.7% | 5.8% | 36.2% | ||
Female | Count | 10 | 15 | 11 | 8 | 44 | |
% of total | 14.5% | 21.7% | 15.9% | 11.6% | 63.8% | ||
Total | Count | 16 | 24 | 17 | 12 | 69 | |
% of total | 23.2% | 34.8% | 24.6% | 17.4% | 100.0% |
Conjoint analysis findings for each research engagement action
Research engagement action | Subaction | Importance† (%) |
---|---|---|
1. Searching for research | a. Policymaker searched academic literature databases and/or physical libraries | 31.45 |
b. Policymaker searched grey literature sources | 15.82 | |
c. Policymaker obtained research by chance, research that was on-hand, or provided by colleagues | 12.04 | |
d. Policymaker requested experts (researchers, working groups, librarians, or other research experts) to identify research | 17.32 | |
e. Policymaker searched generic databases or search engines | 9.80 | |
f. Policymaker examined reference lists, citation indices, or databases of references | 13.56 | |
2. Research obtained | a. Policymaker found systematic reviews and/or meta-analyses | 36.50 |
b. Policymaker found books and/or technical monographs | 6.42 | |
c. Policymaker found primary research and/or theoretical research | 18.61 | |
d. Policymaker found unpublished research and/or conference resources | 9.16 | |
e. Policymaker found internal policies, evaluations, or data | 4.28 | |
f. Policymaker found policies, evaluations, or data from external organisations or registries | 11.27 | |
g. Policymaker obtained recent (up-to-date) research from the above categories | 13.75 | |
3. Appraising relevance | a. Policymaker assessed whether the research was applicable to the policy context or policy issue | 22.84 |
b. Policymaker assessed whether research recommendations were actionable and/or feasible | 20.50 | |
c. Policymaker assessed whether the research was consistent with previous research on the issue | 13.70 | |
d. Policymaker assessed if research was compatible with his/her OR the organisation's values, knowledge, or experience | 13.05 | |
e. Policymaker consulted experts to assess the relevance of research | 14.35 | |
f. Policymaker undertook these actions as part of a pre-specified strategy | 15.56 | |
4. Appraising quality | a. Policymaker assessed whether the design or conclusions of the research were valid | 22.17 |
b. Policymaker evaluated whether the design or conclusions of the research were described clearly and comprehensively | 12.98 | |
c. Policymaker assessed whether the source of the research was credible | 12.17 | |
d. Checked if the research cited, or was referenced in other high-quality research or policy documents | 8.60 | |
e. Policymaker consulted experts to assess quality | 14.51 | |
f. Policymaker assessed the level of evidence of the research | 16.77 | |
g. Policymaker undertook these actions as part of a pre-specified strategy | 12.79 | |
5. Generating new researcher | a. The level of intention of the policymaker to generate or commission new research (to follow-up the current policy) | 38.79 |
b. Policymaker mentioned thorough research generation activities | 32.17 | |
c. Policymaker mentioned less intensive research activities | 17.85 | |
d. Policymaker advocated for future research to be undertaken | 11.19 | |
6. Interacting with researchers | a. Policymaker engaged in thorough collaborative activities with researchers | 41.78 |
b. Policymaker engaged in less intensive interactions with (other) researchers | 14.09 | |
c. Policymaker engaged in sporadic contact with (other) researchers? | 11.03 | |
d. Policymaker actively initiated these interaction activities | 33.10 |