Development of measurable indicators to enhance public health evidence-informed policy-making

We defined a measurable indicator as an observable trait that is an objective measure of some other phenomenon difficult to estimate directly. Our focus was on indicators that could be used to assess if and to what extent a certain health policy is informed by evidence; we intended evidence in a wide sense, including research evidence, experiential evidence, and knowledge from stakeholders and target groups.

The initial set of indicators for EIPM was developed based on literature describing existing frameworks of EIPM processes, influences on these processes and constructs that were pertinent to indicator selection, and on previous REPOPA findings [26, 27].

As for the first input, we focused on two types of published frameworks. The first type described knowledge production and translation in policy-making environments. These frameworks, which concerned science-policy relationships and the use of science in policy-making [6, 35‐37], explicated EIPM processes and stages, key actors and influences, and strategies to foster research knowledge use. The second type of framework examined EIPM processes within service delivery organisations [38‐44], or knowledge translation processes between academics and knowledge users. This second set of frameworks specifically described stages of knowledge use in policy-making [41, 45, 46] and organisational factors that influenced users’ acquisition of research [24]. We also examined literature highlighting facilitators of EIPM [21, 22, 47‐52] to select candidate indicators of these enablers. The European Responsible Research and Innovation framework [53] led us to include equity and inclusiveness elements in the indicators. Moreover, literature specifically focused on indicator development and/or validation in health and other policy sectors provided insights on the principles, criteria and processes to be considered while developing indicators [23‐25, 54‐58].

Our second input consisted of results from previous REPOPA research steps [26, 50, 58‐66]. These findings informed the identification, selection and framing of some indicators. In particular, results highlighted the need for indicators that (1) were pertinent to a wide range of stakeholders working in different sectors and at different levels of government; (2) reflected how policy-makers mobilise internal and external networks to inform decisions about physical activity policies; and (3) took into account considerations about the diversity of target groups (including vulnerable populations). The process of building measurable indicators also involved converting tacit knowledge¹ of the researchers of REPOPA Consortium into explicit knowledge, namely an ‘externalisation’ [67] that can be considered as a further input to the initial set of indicators for EIPM. This objective was achieved by means of both online and face-to-face meetings. In particular, the researchers were provided with a specific template for translating their findings into measurable indicators. To define the template structure, we considered previously reported findings [57]. After the first formulation, the proposed indicators were translated in terms of measurable indicators to infer the presence and the extent of EIPM in an objective way, applying the dimensions of ‘SMART’ indicators – specific, measurable, achievable/applicable, relevant, time-bound [33, 55].

Following the steps described above, we developed an initial set of 23 measurable indicators for EIPM to be used as the starting point for the two-round internet-based Delphi study.

To prepare the two internet-based Delphi rounds, the initial set of indicators was organised in thematic domains and criteria were defined both for the type and number of panellists to be involved and the evaluation of the indicators.

The 23 indicators were grouped into four thematic domains related to specific key aspects of EIPM [14, 24, 33, 40, 68‐70]. These domains were as follows:

We aimed to involve an international group of panellists from the fields of health, physical activity and across sectors, with the roles of researchers, policy-makers (both civil servants and politicians) and other relevant stakeholders (e.g. non-governmental organisations). To ensure that different policy-making contexts in Europe were represented and to reach a wide perspective in the Delphi, we planned to have 12 panellists from each of the six REPOPA countries (termed ‘national panels’) and 10 additional panellists working at international level, for a total number of 82 panellists.

While composing each national panel, we aimed to get a balanced distribution of participants in terms of profession (researchers and policy-makers²), sectors (mainly public health, health policy, physical activity and sports, also with reference to disciplines like epidemiology, health economics, political science and social science) levels of policy-making (local, regional and/or national administrative levels), and gender.

We also aimed to include, in each national panel, at least one researcher with experience in science policy and at least one politician among the policy-makers.

Each country team of REPOPA researchers identified and informally contacted more than the scheduled 12 experts, starting from those satisfying the required criteria. We used a snowball sampling approach, asking these experts to suggest the names of other experts suitable for the research. A full list of potential participants was then generated for each country. Subsequently, each country team ranked the contacted experts according to their areas of competence and gradually built the planned panel, reaching the final number of 12 per country.

The panellists working in the international context were chosen among researchers and policy-makers from international organisations related to physical activity, public health, health promotion and policy innovation, e.g. WHO, the European Public Health Association, the Joint Research Centre, the AGE Platform Europe, the European Public Health Association.

The whole panel was thus assembled to include 82 panellists who agreed to take part in the Delphi study (for details see Additional file 1).

We asked Delphi participants to assess the relevance and feasibility of the indicators. Relevance was defined as to the extent to which an indicator inferred the use of EIPM; feasibility was defined as the extent to which an indicator was applicable in EIPM assessment processes. Panellists scored each indicator using a four-point Likert scale (4 – very relevant, 3 - relevant, 2 – slightly relevant, 1 – not relevant; 4 – definitely feasible, 3 – probably feasible, 2 – slightly feasible, 1 – definitely not feasible).

The algorithm developed for ‘accepting’ and ‘rejecting’ indicators, described in Fig. 1, was based on the calculation of medians and first quartiles of both relevance and feasibility.

To be included in the final set of indicators for EIPM, an indicator had to gather consensus on both high relevance and feasibility. Figure 1 shows the cut-off points for consensus we set for an indicator to be accepted or rejected on the left and right side, respectively. These conditions were valid for both the Delphi rounds, so that the indicators satisfying them already in the first round were either directly accepted or rejected and not listed in the second round. The central part of Fig. 1 shows the intermediate cases. Indicators that fell under this condition as a result of the first round were sent to the second round to be reconsidered. Indicators that fell under this condition as a result of the second round were finally rejected.

The indicators that were accepted, in either round, comprise the international set of REPOPA indicators for EIPM.

A total of 82 panellists, as planned, initially agreed to participate in the study, including 12 panellists per country (6 researchers and 6 policy-makers each from Denmark, Finland, Italy, Romania, the Netherlands and the United Kingdom) plus 10 international panellists, including 4 researchers and 6 policy-makers.

A total of 76 (92.7%) panellists answered the first round and 72 (87.8%) answered the second round, always keeping a balanced distribution between researchers and policy-makers.

Following the first round and using the initial set of 23 indicators proposed by the REPOPA team (Additional file 3), 14 indicators were accepted, 9 were sent to the second round for re-consideration and no indicators were discarded, according to the algorithm in Fig. 1 and based on panellists’ ratings.

The suggestions provided by panellists led to the development of 8 new indicators for EIPM to be rated by the panellists in round two (Additional file 4 lists the panellists’ comments that led to new indicators).

The second round led to the acceptance of another 11 indicators (in addition to the 14 previously accepted in the first round), including 5 indicators out of 9 from the initial set (that were neither accepted nor rejected in round 1), plus 6 new indicators out of the 8 proposed on the basis of suggestions given by the panellists in the first Delphi round. These 11 indicators were added to the 14 indicators already accepted in the first round to compose the final set of 25 REPOPA international indicators for EIPM (Fig. 2 and Additional file 5).

Table 1 shows the final set of indicators for evidence-informed policy-making (EIPM) organised in the four thematic domains. 

On the other hand, six indicators were deemed neither relevant nor feasible to be included in the final set of indicators, consisting of 4 indicators from the initial set and 2 new indicators proposed by panellists in the first Delphi round (Additional file 6). Most of these indicators (5 out of 6 indicators; indicators b–f in Additional file 6) were rejected on the basis of relevance, while only one (indicator a: Internships/fellowships provided by research institutions during the policy, Additional file 6) was rejected on the basis of both relevance and feasibility.

Table 1 and Additional file 6 show that all the indicators from the initial set of 23 indicators for EIPM related to acquiring⁴, citing⁵ and producing⁶ evidence in terms of documentation were included in the final set, and two more indicators⁷ attaining to the documentation thematic domain were proposed by panellists to specify the role of evidence briefs and reports on policy results from policy-making organisations at different territorial levels as relevant sources of knowledge.

All the indicators from the initial set related to the involvement of researchers in EIPM – from one-way and bidirectional exchange of knowledge with policy-makers⁸ to a more active role in the development of the policy and in the policy evaluation⁹ – were included in the final set. Moreover, the need for active involvement of policy-competent researchers was stressed by panellists with the proposal of a new indicator (Table 1, indicator 5: researchers with policy-making experience involved in the policy); this is complementary to the indicator related to the involvement of ‘staff with research experience’. On the other hand, indicator a ‘Internships/fellowships provided by research institutions during the policy’ (Additional file 6) was not considered relevant and feasible enough to be accepted. From panellists’ comments, it can be argued that the reason for discarding this indicator – which would be in line with WHO Regional Office for Europe’s recommendations [71] – might be their limited time duration, which does not meet the need of continuity in the relationship between researchers and policy-makers to foster EIPM.

The indicators implying a bidirectional knowledge exchange with stakeholders¹⁰ and their contribution to the policy (Table 1, indicator 2. Stakeholders working on the policy) were included in the final set and panellists further stressed the importance of communication with stakeholders by proposing three new indicators related to consulting target groups to get their perspective, to acquiring communication competences to interact with stakeholders and to fostering knowledge sharing also among different groups of stakeholders¹¹. On the other hand, indicator e ‘Stakeholders working on the policy evaluation’ (Additional file 6) was not accepted by panellists, differently from the equivalent indicator referring to researchers (Indicator 25. Researchers working on the policy evaluation). In this case, what can be argued looking at panellists’ comments is that the prudence in attributing to stakeholders an evaluation role in policy can be linked to the risk of conflict of interests, together with the problem of establishing criteria to select the stakeholders to be involved.

The other remaining indicators discarded from the final set¹² in Additional file 6 concern aspects that can be considered ‘procedural’ rather than ‘substantial’; most of them were related to budget issues. Based on panellists’ comments, it seems that some indicators were deemed not feasible due to the lack of dedicated budgets for EIPM.

The validated indicators for EIPM are intended to be used by decision-makers, researchers and other stakeholders at various stages of a policy-making process. Measurable indicators, by giving objective data, could help inform the design, implementation, and monitoring and evaluation of interventions to foster EIPM.

The indicators are particularly useful for evaluating public health and physical activity policies, either by the organisation responsible for the policy or by other stakeholders such as external evaluators or research institutes. They can support EIPM already during the agenda-setting phase, helping to identify crucial elements to infer the presence and the extent of EIPM to be considered. During the development of a policy, the indicators can be used to monitor enablers of or barriers to EIPM in the policy process, giving the measure of their occurrence, making it possible not only to assess whether, and to what degree, a policy is or is not being informed by evidence, but also to discover why and how, possibly allowing adjustments. The indicators can also be used to evaluate the extent of EIPM of an already implemented policy by the organisation responsible for the policy or other administrative or research bodies. Moreover, policy evaluations using the indicators can also provide valuable insights for future policy processes, also helping to infer if the policy has created new evidence.

Besides evaluation purposes, the indicators can form the basis for EIPM recommendations, implying actions that, if accomplished, would foster EIPM. The indicators may also be the basis for an intervention and for active, critical reflection on how and why EIPM might be addressed, as already shown in literature for other validated knowledge translation tools [14]. Therefore, the use of international REPOPA indicators for EIPM may support EIPM processes, ensuring not only that the policy is informed by evidence, but also that evidence is used instrumentally to support the selection of activities to be implemented [36, 65, 72, 73], and not selectively to justify an already made decision [1, 74].

The availability and use of the indicators proposed in this study may contribute to an organisational culture where extended value is given to the use of evidence for decisions. Others have shown that awareness of an indicator may lead policy-makers to perceive that a problem exists, to change the way they view the problem or to potentially focus the options they see as suitable solutions [75]. In this way, the indicators could also impact on stakeholders’ frameworks of thinking [56], and generate new norms for EIPM within governmentally broad social norms [54].

Furthermore, international REPOPA indicators are a valuable resource for EIPM beyond physical activity and the health field as they attain to transversal approaches to policy-making, enhancing their use for EIPM in other sectors. This is firstly due to the circumstance that all sectors use policy-making cycles with common elements. Moreover, this potential transferability of the indicators was enhanced by the variety of areas of competence and roles among Delphi panellists and the cross-sector approach that was followed and examined during the REPOPA project [26, 50, 63, 76].

A first step towards the practical application of the international REPOPA indicators for EIPM has already been performed by testing them within national conferences held in the six REPOPA countries (to be presented in a later manuscript); based on these national conferences, evidence briefs and guidance resources for the use of the international REPOPA indicators were developed.

According to WHO Regional Office for Europe [71], many tools to support EIPM are already available but are not widely used, and more research and development should continue, including evaluation of new and existing tools [77]. Therefore, institutional support and incentives [78, 79] such as funding or other stimuli for the individuals to foster EIPM could be considered [80]. Health systems that provide strong incentives for dialogues between policy-makers and researchers through formalised processes and enabling structures and environments are actively facilitating knowledge generation. Formalised processes should include explicit incentives to demand and use evidence, as well as time and space for inter-linkages between policy-makers and researchers [43].

Specifically, we think that new approaches for institutionalisation of the indicators would be required, including what employees are rewarded for. A proposal would be to build in a requirement for an assessment of indicators on EIPM into routine job performance. Our suggestion related to the international REPOPA indicators, validated by this study, is to foster their joint use by policy-makers and researchers, as a way to encourage joint researcher–policy-maker teams – a possibility given by the fact that the indicators were jointly developed with the contribution of both researchers and policy-makers, also in line with the WHO recommendations of involving both researchers and policy-makers while developing tools [71]. Indeed, strengthening the interactions between researchers and policy-makers has been described as a potential solution to foster EIPM [22, 24, 81], to such an extent that, according to the WHO Regional Office for Europe, it should be required among the actions to foster EIPM for policy development by the establishment of a legal framework to support the use of evidence [71]. This issue is also reflected in several indicators retained in the final set of international REPOPA indicators that imply a relationship between researchers and policy-makers, also addressing the well-characterised communication gap between them [5, 14, 22, 24, 48, 69, 82]. Current views, which are reflected in the final set of indicators, suggest that EIPM-oriented communication between research and policy-makers should be systematic and continuous, consisting of a collaborative approach towards using knowledge in real-world settings, adapting research questions to policy needs and helping policy-makers to interpret research findings [6, 42, 43, 61, 72, 74, 78, 83‐85].

Moreover, the future use of indicators is facilitated by the availability of a reliable version of the indicators in six country languages (in addition to English, Danish, Dutch, Finnish, Italian and Romanian). Although we did not provide back translation from the six national languages to English, the methodology adopted, involving two researchers external to REPOPA project per country for feedback regarding the comprehension and intelligibility of the questionnaires and indicators, can be considered as an initial step toward validation of the six versions of the set of indicators. This process of validation continued within the national conferences held in the six REPOPA countries and with the analysis and comparison of their results.

According to WHO Regional Office for Europe [71], existing evidence and tools for EIPM should be available in local languages and sharing lessons and learning from country experiences is important as an action to build EIPM capacities, in particular in assessing and comparing EIPM practices across countries.

Finally, according to the literature [86, 87], processes of interaction, discussion and exchange are more effective to promote learning than those based on summarising research, disseminating papers and commissioning reports. In this sense, as the REPOPA international Delphi process has the added value of being a research work and a first dissemination action at the same time, the REPOPA indicators have already started spreading.

Two main strengths of the study are the quality of the panel, including experts coming from different areas of competence and different geographical contexts, and the unusually high response rate obtained in both the first and second round of the Delphi (92.7% and 87.8%, respectively) [32, 88, 89]. Reaching this goal was supported by a coordination strategy that involved local management of country panellists by leads in each of the participating countries and Delphi coordinators supporting the local managing process. A possible limitation is that, in order to make the indicators adaptable to various contexts, we did not define specific units of measurement (e.g. Boolean, numerical, percentage values) and baselines (e.g. specific values to be reached to assess the presence of EIPM) to be assigned to each indicator – these should be established by the users with reference to the context of a specific health organisation or policy in a given territory. At the same time, psychometric assessment of the indicators could be performed in order to deeply understand latent factors in the indicators in view of improving their implementation in various health and research organisations, as reported in the literature for other tools [90].

Although the process of contextualising the indicators in different countries has already started by means of the national conferences held in the six European countries within the REPOPA project, further adaptations might be needed to enlarge the environments where this set of indicators can be applied, especially with reference to the specific contexts of resource scarcity and high burdens of disease in low- and middle-income countries, where evidence uptake to support effective and efficient health systems interventions is crucial to reduce health inequities [43] and EIPM might face specific barriers to be considered. In low-resource settings, among the variety of specificities to be kept in mind while dealing with EIPM, a further issue may concern the interface between national policies and the policies of international agencies.

At the same time, the implementation of the indicators within a specific health policy or organisation is still to be tested. Future empirical studies should test the proposed indicators in actual policy processes to further assess their usability and help to understand how to integrate them in the regular business of an organisation. This testing should also involve policies not strictly related to the health field in order to verify the transferability of the indicators to other sectors.

Finally, further implementation research would be required to examine processes necessary to stimulate the use of the indicators by researchers, policy-makers and other stakeholders.