Scope
The specific area of health or healthcare to which the COS is to apply needs to be described, with details of health condition, population and types of interventions. The COS may be developed to encompass all stages or severity of a health condition or it may be focused on a particular disease category. For example, in colorectal cancer, a COS may be developed for all patients or it may focus on patients with metastatic disease. Similarly, the core set may be developed for all treatment types or for a particular intervention (for example, COS may be created for use in all trials of interventions to treat morbid obesity or for bariatric surgery alone).
Identifying existing knowledge
One of the difficulties in this area of research is how to identify studies that have already been done (or are underway) to develop COS. As part of the COMET Initiative, a searchable database has been developed [
17]. This enables researchers to check for existing or ongoing work before embarking on a new project, thus minimizing duplication of effort.
A review of previous trials [
20] or systematic reviews in the area can provide evidence of need for a COS and also identify a potential list of outcomes. Systematic reviewers are starting to use the outcome matrix recommended by the ORBIT project [
3] to display the outcomes reported in the eligible studies. This matrix may demonstrate the inconsistency of outcomes measured to date in addition to potential outcome reporting bias. A review of studies other than clinical trials (for example, observational research into harms) may also identify additional outcomes, such as rare endpoints, that would be worthy of consideration for inclusion in the COS.
Stakeholder involvement
Key stakeholders may include patients and the public, health care practitioners, regulators, industry representatives, and researchers. Bringing diverse stakeholders together to try to reach a consensus is increasingly well-accepted as the future of collaborative, influential research. An important example of this is the work of the James Lind Alliance in determining important questions about treatments where uncertainty remains [
21]. Decisions regarding the stakeholder groups to be involved and the target number from each group will be dependent upon the particular scope of the COS as well as upon existing knowledge and practical feasibility considerations. These decisions should be documented and explained in the study protocol. Consideration should be given to potential conflicts of interest within the group developing the COS (for example, the developers of measurement instruments in the area of interest or those whose work is focused on a specific outcome) [
22].
Few COS studies to date have involved patients or the public, yet those that have done so have identified outcomes that were not previously identified by the other stakeholders [
4,
5,
23‐
26]. Methods for identifying patients or their representatives include clinics, patient societies, advocacy groups, and care giver support groups. To achieve representativeness, it may be helpful to consider approaching practitioners with the support of professional bodies.
Consensus methods
The first step in the process is typically to develop consensus about ‘what’ to measure. The ‘how’ and ‘when’ to measure are usually later in the process but may be determined by consensus methods as well. Groups that first establish consensus about what concepts to measure may subsequently conclude that there is a single measurement instrument for an outcome in the COS that is supported by sufficient evidence to recommend its use. However they may identify that gaps in outcome measurement exist, either because there is no ideal instrument for a particular concept or because the evidence base for existing instruments is of limited quality. These limitations may make it difficult to identify which of several potential measures may be preferable for use in the relevant context. The COSMIN (COnsensus-based Standards for the selection of health Measurement Instruments,
http://www.cosmin.nl) checklist [
27] can be used as a tool for developing studies of the validity and reliability of measurement instruments because it describes the necessary design requirements for the assessment of those measurement properties. In addition, feasibility of measurement is a further consideration [
13].
Methods used in previous studies to elicit opinions and to develop consensus about important outcomes include expert panel meetings [
28], Delphi surveys [
22], Nominal Group Techniques [
13], focus groups [
24], individual interviews [
25] and individual questionnaires [
5]. Considerations concerning the choice of method include the need to build a true consensus with methodological rigor, strategies to ensure that a diverse range of opinions are heard, and factors such as financial and carbon costs that might limit the practicality of face-to-face meetings. It is important to ensure that views from all key stakeholder groups are included when making the final decision regarding the COS. This can be achieved through anonymous voting facilitated through email or keypads at meetings.
It is necessary to decide what information about possible outcomes should be given to stakeholder participants before a consensus exercise begins. A literature review of relevant studies showing the outcomes clinicians value and report most frequently, which may include information about methods to measure the identified outcomes and when to measure them, has been proposed [
18‐
20]. This should be combined with outcomes deemed to be important to health service users if such work has been undertaken previously. If consensus participants are shown a list of potential outcomes, it is generally recommended that they be given the opportunity to propose the inclusion of additional items, especially as the literature may not include outcomes associated with the most recent treatments available. Because this has the potential to result in a long list of items, criteria for determining inclusion of items to be considered in the consensus exercise may be needed. For example, Devane
et al.[
15] required new items to have been suggested by at least two participants. If it is felt that the sharing of a list of outcomes at the outset of the consensus process may bias responses, open questioning may be preferred. Techniques to do this include administering questionnaires [
5], focus groups [
24] and in-depth interviews [
25], to determine outcomes important to patients. However this may lead to stakeholders not considering areas previously deemed important, and subsequent questions to prompt consideration of specific outcomes may be warranted.
Researchers should consider the potential impact of the following methodological decisions on the final results: group composition, questioning technique, the information participants receive to inform their answers, whether or not responses are anonymous, how the group participants interacted with or influenced each other, the medium of the interaction, attrition bias, analysis which can miss or overstate the importance of certain outcomes, and the way in which consensus is reached. A single heterogeneous consensus panel comprising the various stakeholders may be deemed appropriate for particular areas of healthcare whereas separate panels for different stakeholder groups followed by work to integrate the multiple perspectives may be more appropriate for others. A variety of methods have been used to date in published and ongoing studies, particularly related to the inclusion of the patient perspective. The proportion of patients and health service users chosen in a Delphi survey may depend upon the clinical setting; for example, for breast reconstructive surgery, which is an optional procedure undertaken for cosmetic purposes, the involvement of patients is more critical than in other settings. A research project has recently started within the COMET Initiative to assess the effectiveness of these different methods.
Consideration should be given in advance to the criteria that will be used to determine when consensus has been achieved. A review of the reporting of Delphi studies to develop COS demonstrated poor reporting of the methods used [
29]. As an example definition, in a Delphi survey participants may be asked to score each outcome from a long list using the scale proposed by the GRADE group
http://www.gradeworkinggroup.org, in which 1 to 3 signifies an outcome of limited importance, 4 to 6 important but not critical, and 7 to 9 critical. A number of rounds may be held in which responses are summarized and fed back to individuals, allowing them to change their score in light of the group’s opinion. Consensus regarding whether an outcome should be in the COS could be defined as 70% or more of the respondents scoring it 7 to 9 and fewer than 15% scoring it as 1 to 3. Consensus that an outcome should not be included in the COS could, for example, be defined as 70% or more scoring it as 1 to 3 and fewer than 15% scoring it as 7 to 9. All other score distributions would be taken to indicate lack of agreement for inclusion of a given outcome in the COS. The rationale for these thresholds is that acceptance that consensus has been reached for an outcome to be included in the COS requires agreement by the majority regarding the critical importance of the outcome, with only a small minority considering it to have little or no importance. Likewise, for consensus to have been reached that an outcome should not be in the COS requires agreement by the majority that the outcome is of little or no importance, with only a small minority considering it to be critically important. Whereas choice of thresholds is somewhat subjective, specification of the definition in the study protocol should reduce the risk that researchers will define consensus post-hoc in a way that would bias the conclusions toward their own beliefs.
Consideration should be given in advance to the possibility that consensus may not be achieved. For example, different stakeholders may disagree about the inclusion of specific outcomes in the COS. This might lead to a decision to recommend the smaller COS, about which there is consensus.
Implementation of core outcome set
To increase COS uptake, it is recommended that developers consider engagement with the relevant Cochrane Review Groups, clinical guideline developers, research funders, journal editors, regulators such as research ethics committees, and trial registries. For example, the NIHR Health Technology Assessment funding body in the UK has recently added the following statement to its application form:
"‘Details should include justification of the use of outcome measures where a legitimate choice exists between alternatives."
"- Where established Core Outcomes exist they should be included amongst the list of outcomes unless there is good reason to do otherwise. Please see The COMET Initiative website at
http://www.comet-initiative.org to identify whether Core Outcomes have been established.’"
Potential barriers and cost implications of implementing the COS should be considered.
Clear and transparent presentation
Reporting standards for studies specifically using Delphi methods to achieve consensus about a COS have been recommended [
29]. We propose the checklist in Table
1 to improve the reporting quality of studies to develop consensus around domain concepts or what to measure more generally.
Table 1
Checklist of the items that groups should consider when reporting the development of a COS of domain concepts (that is, ‘what’ to measure)
Title | 1 | Identify the report as a study to develop a COS. |
Structured summary | 2 | Provide a structured summary including, as applicable: background, objectives, data sources, participant eligibility criteria, study methods, results, limitations, conclusions, and implications of key findings. |
Rationale | 3 | Describe the rationale for the development of a COS in the context of what is already known. This may include a review of outcomes in previous trials or systematic reviews. |
Objectives | 4 | Provide an explicit statement of questions being addressed with reference, as applicable, to: health condition, population, and types of intervention(s). |
Protocol and registration | 5 | Indicate if a study protocol exists, and where it can be accessed (for example, web address) |
Eligibility criteria | 6 | Specify participant eligibility criteria, including stakeholder group, the rationale for involving them, and how participants were identified and sampled. |
Information sources | 7 | Describe all information sources (for example, systematic review, databases with dates of coverage, contact with study authors) provided to participants before the start of and during the consensus process. If no information on previously measured outcomes is provided, this should be clearly stated together with details of the method for obtaining information on outcomes of importance from the participants. |
Consensus process | 8 | Describe method to determine consensus and the rationale. A checklist for reporting Delphi methods applied to the development of COS has previously been recommended [ 29]. |
Outcome scoring | 9 | Describe how outcomes will be scored during the consensus exercise, and how scores will be summarized across participants during each stage of the consensus process. |
Definition of consensus | 10 | Clearly describe any pre-determined definition of consensus. Describe procedure for determining how outcomes will be included or excluded from consideration at each stage of the consensus process. |
Participants | 11 | Give the total number of participants invited and the number involved in each aspect of the study. Give the proportion of each type of participant from the various stakeholder groups involved. Present any data collected on participant characteristics. |
Results of the consensus process | 12 | As a minimum, provide a comprehensive list of all the outcomes that participants agreed should be included in the core set. Describe a measure of group response and distribution of response for each outcome considered during the process. |
Summary of evidence | 13 | Summarize the main findings regarding the level of consensus and the content of the COS. Consider its relevance to key groups e.g. patients and the public, healthcare providers, and policy makers, and any potential barriers to implementation. |
Limitations | 14 | Discuss limitations in terms of stakeholder and geographical coverage. Describe methods used for assessing risk of bias, in relation to information provided to participants beforehand, attrition, any lack of anonymity, etc. |
Conclusions | 15 | Provide a general interpretation of the results in the context of other evidence, and implications for future research. |
Funding | 16 | Describe sources of funding and the role of the funder in the study. |
Conflicts of interest | 17 | Describe any conflicts of interest within the study team, for example researchers who have developed an outcome measurement instrument applicable to the scope of the COS. |
This includes declaration of potential conflicts of interests within the COS development study team to avoid concerns that individuals with vested interests; for example, the developers of measurement instruments in the area of interest may have overtly or covertly manipulated the process of consensus development.