Reporting guideline for priority setting of health research (REPRISE)

The purpose of REPRISE is to facilitate comprehensive and transparent reporting of health research priority setting exercises, in which there is direct involvement of stakeholders setting research priorities. The REPRISE guideline is flexible so that it may be used for a range of approaches. The scope of REPRISE does not cover approaches without direct involvement of stakeholders such as documentary analysis (e.g. evidence mapping), and econometrics methods (e.g. value of information). REPRISE is not intended for use to appraise the quality of priority setting studies, establish or evaluate criteria for research priorities (e.g. evidence gaps, prevalence of disease, economic considerations), and does not recommend a preferred approach.

Systematic reviews have consistently shown the reporting of the process of research priority setting with stakeholders is highly variable and limited with many details omitted [21, 23, 25, 26, 32]. The need for a reporting guideline has also been identified through our workshops and forums at national and international meetings (e.g. Cochrane Colloquia and Symposia [12, 33‐35]; James Lind Alliance [5]), and through our collective experiences of conducting, publishing, reviewing, and using research priority setting studies.

We conducted a comprehensive search for frameworks, guidelines or models for conducting, appraising, reporting and evaluating health research priority setting, reviews (including systematic reviews) of research priority setting studies, and primary research priority setting studies. We searched electronic databases including MEDLINE, Embase, CINAHL, PsycINFO from inception to 23rd July 2019 using sensitive search strategies provided in Additional file 1. We used Medical Subject Heading (MeSH) terms and text words for research priorities and combined this with terms related to reporting, conduct and evaluation. We also searched Google Scholar, relevant organizational websites (e.g. WHO, EQUATOR, Cochrane, James Lind Alliance and PCORI), and reference lists of articles. The search results are shown in Additional file 2. From the 21,556 records retrieved, we identified 13 frameworks or guidelines for conducting or evaluating research priority guidelines [5, 13, 18‐20, 28, 36‐42] (none designed for reporting research priority setting), and 13 reviews of research priority setting [15‐17, 21‐27, 32, 43, 44], of which four were systematic reviews [23‐26]. (Additional file 3) We also identified 455 primary research priority setting studies.

We extracted items related to the process of priority-setting from the frameworks and systematic reviews included (the sources are listed in Additional file 2). We translated these into reporting items for the candidate REPRISE reporting guidelines by grouping similar items and rephrasing the statements as a reporting item. We inductively developed the initial list of reporting items. This was reviewed by two other investigators (AB, AM) to ensure all relevant items were included in the list. The reporting items were compiled into 10 domains: context and scope, governance and team, framework for priority setting, stakeholders/participants, identification and collection of priorities, prioritization of research topics, output, evaluation and feedback, translation and implementation, and funding and conflict of interest. (Additional file 4) We imported all sources (frameworks and systematic reviews) into HyperRESEARCH software for coding textual data, and AT conducted line-by-line coding of each source to the initial items. We generated a report of each reporting item and the corresponding coded text (content). We developed descriptors and examples based on the content of the sources and input from all the investigators. The sources that contributed to each reporting item, and examples of the original extracted items are shown in Additional file 5.

The preliminary REPRISE guideline was presented at the Australasian Cochrane Symposium, in which participants used the guideline to assess the reporting of a research priority exercise, and provided feedback on the guideline [34]. We subsequently used the REPRISE guideline in two systematic reviews of research priority setting studies in childhood chronic conditions [23] and organ transplantation [26]. We also applied the preliminary guideline to report a research priority setting exercise in health communication and participation [45].

In the final stage, we piloted the guideline with research priority setting studies. We used a purposive sampling strategy to select 30 priority setting exercises from the total of 455 studies retrieved from the search to ensure a diverse range of health topics, regions, stakeholders involved, framework or methods used, and type of output. Using a standardized data extraction template with the reporting items from the REPRISE guideline, the investigators (two per study) independently assessed if the study reported on each item (yes/no), added comments, and suggested new reporting items that were not yet captured. The results are provided in Additional file 6. After completion, the investigators discussed the relevance and applicability of the items, clarity of the items, comprehensiveness of the descriptor and examples, and any new reporting items proposed. These were integrated into the final reporting guideline, which was reviewed and approved by all investigators.

Establishing the context and scope is recommended as these “underpin the process of research priority setting,” [13] including the selection of relevant stakeholders and methods used. In terms of geographical scope, priority setting may be done at an institutional, local, national, or international level [13, 19, 21, 39‐41]; recognizing that each will have its own “sense of mandate, capacity, culture, and resources.” [39] Research priorities may address a specific condition, disease or risk factor (e.g. cancer, mental health), population (e.g. elderly, adolescents), health system, research design, or interventions (e.g. vaccination) [5, 13, 19, 21, 24, 27, 37, 38, 40, 41, 44]. These can be decided upon based on the evidence [5, 38] and initial deliberations with stakeholders [38].

The intended beneficiaries [13] may include patients, caregivers, or the general community who could benefit from the priority setting exercise, and the target audience are those who have the potential to implement or fund the research priorities identified [13, 17, 39‐41]. The focus, content and type of research to be considered can determine the scope of the priority setting exercise. In terms of the broad research areas, these generally span public health, health services, clinical research and basic science [24, 37, 41, 44]. The types of research questions that may be included can range from etiology, diagnosis, prognosis, treatment, to behavioural; and social science, economic evaluation and implementation [23, 24, 27, 39]. It is not necessary for the type of research questions to be determined a priori.

Providing an estimated time frame that the priorities are expected to be valid or relevant may be relevant. This is because research priorities may evolve due to the development of new technology or interventions, emerging evidence, or changes to the health system or socio-political contexts [13, 17, 19, 37‐39, 41, 42]. If there are plans to update the priority setting or to monitor the priorities for the need to update, these could be described. There has been suggestion of 3–5 year cycles of prioritization if the priority setting exercise is to be repeated [37, 38].

It has been argued that priority setting requires “credible” [39] leadership to support acceptability and uptake. This may require leaders who are trusted by stakeholders and who have the necessary expertise, knowledge, decision-making skills, and ability and deliver the project. The leadership and management team is usually responsible for overseeing, developing and implementing the process for priority setting [5, 13, 19]. The leadership group may take the form of, for example, an Executive Committee, Advisory Group, Technical Expert Group [13]. Members of the leadership team would generally be expected to contribute broad and relevant collective insights, harness their networks for engagement and partnership; and include a diversity of members to offer legitimacy to wider stakeholder networks (e.g. patients, caregivers, researchers, policy makers, clinicians, representatives from other non-government or government organizations) [5, 21], and those with technical expertise [5]. The membership and selection of stakeholders may need to take into consideration the need for equity [12, 36]. Also, it has been suggested that the involvement of individuals or organizations with experience in priority setting and relevant research skills can ensure a “high quality process.” [5, 13, 40] With regard to facilitators, neutrality and facilitations skills may be important to elicit input from diverse and mixed stakeholders [5, 36].

Some priority setting studies use or adapt frameworks to guide the process [17, 22, 23, 25, 44]. Common frameworks include the James Lind Alliance [5], Council on Health Research for Development Essential National Health Research (COHRED/ENHR) [19], Essential National Health Research (EHNR), Child Health and Nutrition Research Initiative (CHMRI) [41], and the Dialogue Model [36]. A summary of these frameworks is provided in Table 2. Some priority setting exercises may develop and use a different approach, and not necessarily follow an existing or established framework.

Stakeholder involvement in priority setting can vary across the priority setting exercises. In some cases, they are involved in all key stages of the process and in others, they are consulted in specific steps and existing data or documents are used instead of consultation. Relevant stakeholders whose “values and interests should be respected in setting health research priorities” [41] can include patients, caregivers, clinicians, policy makers, representatives from non-governmental organizations [5, 18, 21, 39, 44]; and diverse groups, for example based on demographic or clinical characteristics, may need to be included in research priority setting [5, 18, 20, 36, 37, 39, 40, 42]. It has been emphasized that patients/caregivers (and if relevant the public) need to be directly involved in the priority setting process [5, 18, 38‐40, 42], as they have direct experience of the health condition or context and often have different priorities to researchers and clinicians. There is also recognition of the need to involve individuals from vulnerable or marginalized groups, particularly in equity-focused research priority setting exercises [5, 12, 21, 36, 39].

Multiple strategies may be used to engage stakeholders in the priority setting process, and this is namely through partnership with relevant stakeholder organizations [5, 39]. The number and characteristics of the participants involved enables assessment of the degree of inclusivity, diversity and equity [5, 13, 39] in priority setting processes. The characteristics to specify may include role and expertise, discipline, organizational affiliations, demographics (e.g. age, sex, socio-economics status, ethnicity), and clinical factors [13, 21‐23, 36, 42, 44]. Support for patients/caregivers involved in priority setting may include reimbursement for travel, arranging care for dependents, and time [5]. This may indicate to readers the degree to which the team was able to ensure inclusivity across the different groups. Of note, there is recognition that attention must be given to power dynamics, otherwise the engagement of disadvantaged and marginalized groups may lead to “presence without voice and voice without influence.” [2] Therefore, it may be relevant to acknowledge and discuss how hierarchies and “asymmetries between stakeholders” [36] are addressed to maximize constructive and balanced interaction. For example, some groups, such as patients, may require additional time, training, resources, or other strategies to be able to engage; to have the opportunity to contribute meaningfully [36].

Different methods and approaches are available for collecting and selecting initial research priorities from stakeholders and developing the first list of priorities. This can be one or a combination of methods including interviews, focus groups, workshops, and surveys; and consensus methods (e.g. Delphi survey, nominal group technique); and these may be conducted through various modes such as face-to-face or online [5, 13, 23, 25‐28, 37‐39]. Documents such as systematic reviews, technical data, and other relevant reports may be used to identify the initial list of priorities [5, 13, 18, 21, 24, 39, 40]. In some priority setting exercises, the initial list of research priorities is derived from literature or existing data rather than consultation or engagement of stakeholders [47‐49].

If a wide range of different initial research priorities are submitted or identified, it can be challenging to manage and synthesise to capture the diversity of views in a concise manner, whilst also retaining the context and nuances of the submissions. They may need to be organized, usually by collating and categorizing them into themes, topics or other relevant taxonomy [5, 22, 36, 38, 39, 41]; and by removing those that are “out-of-scope,” [5] or duplicative [39]. They may then be translated into “indicative, researchable questions” [5] and edited for clarity [21, 27, 35, 36, 38]. Some priority setting exercises conduct cross checking of the priorities against the evidence (i.e. systematic reviews [5]) and evidence mapping [48, 50‐52].

The number of research priorities identified at each stage vary widely [24]. Generally, 10 to 20 questions/topics are included in the final set of priorities [5, 23‐26, 38, 40].

Prioritization techniques can include scoring, ranking, voting, and ordering, and these are usually embedded in similar methods and modes used for collecting priorities as outlined in Section E. Some frameworks and priority setting exercises use explicit criteria to prioritize questions [13, 21, 22, 38, 39]. Examples include condition-related criteria (burden of disease, variation in care and outcome, evidence gaps), and research-related criteria (resources required, likelihood of success and impact) [37, 42]. The CHNRI method proposes criteria including: answerability, attractiveness (likely to be published in high-impact journals), novelty, potential for translation, effectiveness (likely to identify better interventions), affordability, deliverability, sustainability, public opinion (acceptability to the general public), equity (leads to interventions that will be accessible to marginalized or vulnerable populations), and cost and feasibility [27, 41]. Using specific criteria can facilitate a deliberative and rational process, particularly when there is limited information [21]. It may be relevant to report the processes for selecting, defining and changing the criteria. Of note, the use of criteria can add complexity to the process, and strategies may be needed to avoid inadvertent exclusion of other stakeholder values that influence prioritization. Whilst assigning scores based on such criteria may be rational, there are concerns that it may give a false sense of objectivity. The method for excluding priorities at this stage i.e. based on a quantified threshold or other criteria should be provided. Any processes to appeal or challenge the results may be specified.

The output should be “clear and of value to the research community.” [5] The final priorities generated can range from having a specific structure i.e. the Population, Intervention, Comparator, Outcome (PICO) format [5, 37], to broader outputs such as topics or themes/areas [38]. It is possible that components of PICO are not specified in original submission of priorities, or that it cannot be applied to some types of research questions [5]. In some circumstances, attempting to produce very technical research questions can potentially place non-researchers, who may include community members, patients, caregivers, at a disadvantage, as they may feel unable to articulate or consider the specific technical components. Also, consideration may need to be given to ensure that the contextual data and values around the questions are not missed. Some priority setting exercises seek to identify broader themes or areas and translate these into research questions after prioritization.

While there is no “gold standard” [37] approach for evaluating the process of research priority setting, process evaluation can provide information about the acceptability, “reliability and usefulness” [37] of the process and results [5, 13, 37]. Stakeholder satisfaction with the process in terms of being able to engage and express opinions, and whether the priorities are considered meaningful and valid may be evaluated [13, 18]. Participants and stakeholders could have an opportunity to review and provide feedback on the prioritized questions [22, 36, 39]; and having “revision or appeal” [18] mechanisms available to identify and address disagreements in a constructive manner [18] have been suggested.

Strategies to implement the research priorities could involve informing and garnering support from government, policy makers, and funding agencies to allocate funding and resources toward the priorities identified [13, 19, 22, 39], and working with researchers to develop proposals [5, 19, 39]. Assessing the impact of research priority setting is challenging but needs to be considered [5, 22]. This may include the impact on decision-making, allocation of funding and resources, and research output [18, 19, 38].

There are different sources of funding that can affect a priority setting process. The funding and resources used to conduct the priority setting exercise and support the stakeholders (directly or indirectly) and whether the priority setting exercise is connected to a funding source to support the identified research priorities. Reporting the sources of funding and support is usually required. The resources required for research priority setting will depend on the size, scope, timeline, methods used, and personnel required [5, 19], and providing information about the budget may be useful for others who are planning on conducting research priority setting. It is recommended that any relevant disclosures be stated for transparency, to allow assessment of potential political or commercial influences or undue bias [5, 13, 24]. For example, pharmaceutical companies may have close ties with patient organisations and clinicians, and the potential influence this may have on the priority setting process would need to be addressed explicitly. This may be declared at an individual level, or at a process or contextual level, for example, providing a narrative of any political issues, conflict or controversies that may affect the process, output or implementation of the priority setting exercise [20].