Chapter 1: Background
1.1 Outcomes in clinical trials
1.2 Problems with outcomes
1.3 Standardising outcomes
1.3.1 Core outcome sets (COS)
1.3.2 Core outcome set initiatives
1.3.3 The COMET Initiative
-
Information about COMET, including aims and objectives and a description of the COMET Management Group
-
Information about upcoming and past COMET events, including workshops and meetings organised by the COMET Initiative
-
Resources for COS developers, including relevant publications, examples of grant-funded projects, examples of COS development protocols, plain language summaries and PPI resources
-
Relevant web links, including core outcome networks and collaborations, patient involvement, how to measure, and research funding
1.3.4 Other relevant initiatives
1.4 The aim of this Handbook
Chapter 2: Developing a core outcome set
2.1 Background
2.2 Scope of a core outcome set
2.2.1 Health condition and target population
2.2.2 Interventions
2.2.3 Setting
2.3 Establishing the need for a core outcome set
2.3.1 Does a relevant core outcome set already exist?
2.3.2 Is a core outcome set needed?
2.3.3 Avoiding unnecessary duplication of effort
2.4 Study protocol
2.5 Project registration
2.6 Stakeholder involvement
2.6.1 Patient and public involvement and participation
-
Provide advice on the best ways of identifying and accessing particular patient populations
-
Inform discussions about ethical aspects of the study
-
Facilitate the design of more appropriate study information
-
Promote the development of more relevant materials to promote the study
-
Enable ongoing troubleshooting opportunities for patient participation issues during the study, e.g. recruitment and retention issues of study participants
-
Inform the development of a dissemination strategy of COS study results for patient participants and the wider patient population
-
Ensure that your COS is relevant to patients and, crucially, that patients see it to be relevant and can trust that the development process has genuinely taken account of the patient perspective.
2.7 Determining ‘what’ to measure – the outcomes in a core outcome set
2.7.1 Identifying existing knowledge about outcomes
Number of databases |
n
| Databases searched |
---|---|---|
1 | 18 | Medline (n = 9) |
PubMed (n = 7) | ||
Central Register of Controlled Trials (n = 2) | ||
2 | 8 | Medline and Embase (n = 4) |
PubMed and Central Register of Controlled Trials (n = 1) | ||
Medline and CancerLit (n = 1) | ||
Medline and Central Register of Controlled Trials (n = 1) | ||
Medline and PubMed (n = 1) | ||
3 | 4 | Medline, CINAHL and Embase (n = 2) |
Medline, Embase and Cochrane Central Register of Controlled Trials (n = 1) | ||
PubMed, CINAHL, PsychINFO (n = 1) | ||
4 | 1 | PubMed, Medline, Embase and the Cochrane Collaboration |
5 | 3 | Medline, PreMedline, CancerLit, PubMed (National Library of Medicine) and Cochrane Library (n = 1) |
Medline, Embase, PsycINFO, Cochrane Library and CINAHL (n = 1) | ||
Cochrane Wounds Group Specialised Register, The Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library), Medline, Embase and CINAHL (n = 1) | ||
6 | 0 | |
7 | 2 | Cochrane Oral Health Group’s Trials Register (CENTRAL), Medline, Embase, Science Citation Index Expanded, Social Science Citation Index, Index to Scientific and Technical Proceedings, System for Information on Grey Literature in Europe (n = 1) |
Cochrane Skin Group Specialised Register, the Cochrane Central Register of Controlled Trials in The Cochrane Library (Issue 4, 2009), Medline, Embase, AMED, PsycINFO, LILACS (n = 1) | ||
8 | 2 | CINAHL (Cumulative Index to Nursing and Allied Health Literature), Embase, Medline, National Criminal Justice Reference Service (NCJRS), PsycINFO, Sociological Abstracts, The Cochrane Database, The Patient-reported Health Instruments (PHI) website (n = 1) |
Medline, PubMed, Embase, PsycINFO, CINAHL, Web of Sciences, Cochrane Central Register of Controlled Trials, and Cochrane Database of Systematic Reviews (n = 1) |
Number of years searched | Frequency |
---|---|
Less than 5 | 1 |
5 to 9 | 3 |
10 to 14 | 12 |
15 to 19 | 2 |
20 to 24 | 6 |
25 to 29 | 1 |
30 to 34 | 1 |
35 to 39 | 4 |
More than 40 | 3 |
2.7.2 Identifying and filling the gaps in existing knowledge
2.7.3 Ontologies for grouping individual outcomes into outcome domains
Reference | Method for classifying/grouping outcomes |
---|---|
Duncan (2000) [225] | Each outcome measure was classified into one of the following categories: death or, at the level of pathophysiological parameters (blood pressure, laboratory values, and recanalisation), impairment, activity, or participation. Measures were classified according to the system used by Roberts and Counsell [226], which includes the Rankin/modified Rankin scale as a measure of activity rather than participation |
Sinha (2012) [60] | Each outcome was grouped into one of the following six outcome domains, some of which were further divided into subdomains: disease activity, physical consequence of disease, functional status, social outcomes and quality of life, side effects of therapy and health resource utilisation. Where it was unclear which domain was appropriate, this was resolved by discussion between the authors. Reference given in support of this approach: Sinha et al. 2008 [227] |
Broder (2000) [129] | List developed by staff at institution (but no further detail) |
Distler (2008)[228] | The results of this literature search were discussed at the first meeting of the Steering Committee. Based on this discussion, a list of 17 domains and 86 tools was set up for the first stage of the Delphi exercise to define outcome measures for a clinical trial in PAH-SSc. Domains were defined as a grouping of highly related features that describe an organ, disease, function, or physiology (e.g. cardiac function, pulmonary function, and quality of life) |
Devane (2007) [173] | Outcome measures addressing similar dimensions or events were discussed by the team and collapsed where possible. For example, various modes of delivery/birth were presented as ‘mode of birth (e.g. spontaneous vaginal, forceps, vaginal breech, caesarean section, vacuum extraction)’. This pilot tool was tested for clarity, with a sample of 12 participants, including 3 maternity care consumers, and subsequently refined |
Smaïl-Faugeron (2013) [111] | Because we expected a large diversity in reported outcomes, we grouped similar outcomes into overarching outcome categories by a small-group consensus process. The group of experts consisted of 6 dental surgeons specialising in paediatric dentistry, including 3 clinical research investigators. First, the group identified outcomes that were identical despite different terms used across trials. Second, different but close outcomes (i.e. outcomes that could be compared across studies or combined in a meta-analysis) were grouped together into outcome domains. Finally, the group, with consensus, determined several outcome categories and produced a reduced-outcome inventory |
Merkies (2006) [229] | In advance of the workshop, a list of outcome measures applied in treatment trials was prepared including their scientific soundness, WHO and quality of life classification (WHO classification reference is ICF) |
Rahn (2011) [230] | From this outcome inventory, the outcomes were organised and grouped into eight proposed overarching outcome domains: 1. Bleeding; 2. Quality of life; 3. Pain; 4. Sexual health; 5. Patient satisfaction; 6. Bulk-related complaints; 7. Need for subsequent surgical treatment and 8. Adverse events. Categories were determined based on their applicability to all potential interventions for abnormal uterine bleeding and the physician expert group’s consensus of their relevance for informing patient choices. Outcomes related to cost, resource use, or those determined by the review group to have limited relevance for assessing clinical effectiveness were excluded from categorisation and further analyses |
Chow (2002) [231] | Some detail but process not described – the endpoints employed in previous bone metastases trials of fractionation schedules were identified and listed in the first consensus survey under the following headings: 1. Pain assessments; 2. Analgesic assessments and primary endpoint; 3. Endpoint definitions; 4. Timing, frequency and duration of follow-up assessment; 5. When to determine a response; 6. Progression and duration of response; 7. Radiotherapy techniques; 8. Co-interventions following radiotherapy; 9. Re-irradiation. 10. Non-evaluable patients (lost follow-up) and statistics; 11. Other endpoints; 12. Other new issues and suggestions and 13. Patient selection issues |
Van Der Heijde (1997) [232] | Grouped into patient-assessed, physician-assessed or physician-ordered measures |
Chiu (2014) [233] | A wide variety of different outcomes measures were reported [in the studies included in the systematic review]. We classified these into 4 categories: postoperative alignment, sensory status, control measures and long-term change |
Fong (2014) [139] | Twenty-one maternal and 24 neonatal outcomes were identified by our systematic reviews, one randomised controlled trial and two surveys. The maternal components included complications associated with pre-eclampsia and were broadly classified under neurological, respiratory, haematological, cardiovascular, gastrointestinal, renal and other categories. The neonatal components were prematurity-associated complications involving respiratory, neurological, gastrointestinal, cardiovascular systems and management-based outcomes such as admission to the neonatal unit, inotropic support and use of assisted ventilation |
Fraser (2013) [234] | Based on an overall evaluation of intra-arterial head and neck chemotherapy, there are several outcome variables that should be monitored and reported when designing future trials. These outcome variables can be categorised into ‘Procedure-related’, ‘Disease control’ and ‘Survival’ |
Goldhahn (2014) [235] | The group reached an agreement to use the ICF thereby identifying key domains within this framework using the nominal group technique. Recently, the WHO has established core sets for hand conditions [236]. A Comprehensive Core Set of 117 ICF categories were selected appropriate for conducting a comprehensive, multidisciplinary assessment. A brief core set of 23 ICF categories were selected, and considered more appropriate for individual health care professionals. The body functions contained in this core sets include emotional function, touch function sensory functions related to temperature and other stimuli, sensation of pain, mobility of joint functions, stability of joint functions, muscle power functions, control voluntary movement functions, and protective functions of the skin. The group agreed that the ICF categories would be consistent with clinicians’ current practice patterns of focussing on pain, joint range of motion and hand strength [237] |
Saketkoo (2014) [238] | The Delphi process began with an ‘item-collection’ stage called tier 0, wherein participants nominated an unrestricted number of potential domains (qualities to measure) perceived as relevant for inclusion in a hypothetical 1-year RCT. This exercise produced a list of >6700 items—reduced only for redundancy, organised into 23 domains and 616 instruments and supplemented by expert advisory teams of pathologists and radiologists |
Smelt (2014) [159] | We then grouped the answers [from round 1 of the Delphi questionnaires] according to the presence of strong similarity. During this process, we followed an inductive method, i.e. answers were examined and those considered to be more or less the same were grouped as one item. No fixed number of items was set beforehand in order to accommodate all new opinions. The answers were grouped by two of the authors (AS and VdG) separately, to ensure independence of assessments. Any discrepancies were resolved through a discussion with two other authors (ML and DK) who also checked whether they agreed with the items as formulated by AS and VdG |
Wylde (2014) [137] | The pain features that were retained after the three rounds of the Delphi study were reviewed and systematically categorised into core outcome domains by members of the research team [86]. The IMMPACT recommendations [239] were used as a broad framework for this process. Each individual feature was reviewed to determine whether it was appropriate to group it into an IMMPACT-recommended pain outcome (pain intensity, the use of rescue treatments, pain quality, temporal components of pain) or a new pain outcome domain. The developed outcomes domains were then reviewed to ensure that the features that they encompassed adequately reflected the domain and that the features were conceptually similar. These core outcome domains were subsequently discussed and refined by the Project Steering Committee and the PEP-R group |
-
Includes subsets all, cause-specific, quality of death, etc.
-
Disease activity (e.g. cancer recurrence, asthma exacerbation, includes ‘physical consequence of disease’, etc.)
-
Blood pressure, laboratory values, recanalisation
-
New, recurrent
-
Includes Health-related Quality of Life (HRQoL)
-
Does this cover activities? Participation? (Read the Roberts and Counsell paper referenced in the review of stroke outcomes)
-
Reported by patient, health professional, etc.
-
Includes subset hospital, community, additional treatment, etc.
-
Be clear that this could include things like death, pain, etc. when they are unanticipated harmful effects of an intervention
2.7.4 Determining inclusion and wording of items to be considered in the initial round of the consensus exercise
2.7.5 Short- or longer-term outcome assessment
2.7.6 Eliciting views about important outcomes
2.8 Determining the core outcome set
2.9 Qualitative methods in core outcome set development
2.9.1 Why use qualitative methods in core outcome set development?
2.9.2 In what circumstances might core outcome set developers consider using qualitative methods?
-
The specific purpose for which qualitative evidence is needed.Whilst qualitative methods can be used to access the perspectives of patients, carers and other stakeholders [133, 146] developers will find it helpful to specifically consider what it is that they are hoping to achieve through the qualitative work, and whether a qualitative approach is the most suitable way to achieve these aims. For example:
-
о A COS developer may be concerned that not all potentially important outcomes have been identified to go forward to a consensus process and will use qualitative work to help ensure that no potentially important outcomes are missed
-
о Retention of Delphi participants over several rounds can be problematic and qualitative studies may help to minimise the number of Delphi rounds needed. For example, qualitative research may enable developers to omit the initial ‘blank page’ or open-ended round of a Delphi survey
-
о COS developers may be working with stakeholder groups, such as patients with dementia or learning disabilities, for whom other COS development activities are unsuitable
-
о As we note above, qualitative research findings may also help developers to define the scope of a COS by informing the choice of population and interventions to be covered by a particular COS
-
о As also noted above, qualitative research findings may help to ensure the consensus process (language, explanations, etc.) is accessible to patients
-
о COS developers may need insights on why particular outcomes are important to patients; qualitative research is widely regarded as being useful in addressing ‘why’ questions
-
-
Whether existing qualitative studies have been conducted that could address some of these aims.Many qualitative studies have now been published describing patients’ and carers’ experiences of specific conditions and treatments. It may be possible to use this existing work to identify how well outcomes from systematic reviews of trials map to those of patients. Similarly, qualitative datasets may be available for secondary qualitative analysis that could serve the same aims. If previous qualitative evidence or data are available, COS developers will want to be clear about why additional primary qualitative research is needed.
-
Qualitative research requires specialist methodological expertise and COS developers will need to ensure that their team includes this expertise. Similarly, the collection and analysis of interview or focus group data requires time and resource and COS development teams will also need sufficient funds to support the qualitative work.
2.9.3 Issues to consider in designing primary qualitative research to inform core outcome set consensus processes
-
Interviews might begin by asking participants to talk about their actual experiences, including how the illness and treatment has affected their lives, about treatment decisions, how these decisions were made and what influenced them.
-
Over the course of an interview, questions may become more focussed on outcomes and opinions; for example, asking patients what they had hoped for from their previous treatments or what they would want from a new treatment if one became available. Understandably, many patients with chronic conditions might hope for curative treatments and interviewers may need to be prepared to prompt patients to describe expectations that are more immediately achievable.
-
Towards the end of interviewing, questions might become more research- or COS-specific. For example, the interviewer might summarise outcomes that patients have discussed in previous interviews and explore what words or phrases patients think researchers should use to label or explain outcomes for future patients.
-
Where the qualitative work has a very specific focus; for example, informing an online Delphi survey for a particular stakeholder group, interviewing might focus on a prototype of the survey and use think-aloud techniques [103, 104] to explore how stakeholders interpret the outcomes and how the phrasing might be refined.
2.10 Considerations to enhance patient participation in a core outcome set
2.10.1 Accessing patients
2.10.2 Information for patients
2.10.3 What questions to ask when involving patients as participants in a core outcome set study?
-
The MoMENT study [55] – this was a study to develop a COS for the management of otitis media with effusion in children with cleft palate. A consensus survey was used in this study. Parents were presented with a list of outcomes to score but could add in any missing outcomes that they considered relevant. The question used was: ‘Think about when your child had glue ear and how you might decide if their treatment for glue ear had worked. We would like you to look at the list below and tell us how important each thing on this list is in deciding if treatment has worked’ [55].
-
An online Delphi consensus study was conducted with patients who suffered from migraine. The patients were asked two open-ended questions in round 1 of the Delphi survey and these were:
-
о What do you find most bothersome about having a migraine attack?
-
о If a new medicine was developed against migraine attacks, what would you wish the effect of this medication to be [159]
-
-
A COS was developed for children with asthma [60]. In an online Delphi survey, parents, young people and clinicians were asked open-ended questions in round 1 to identify outcomes of importance. The following four open-ended questions were used with the parents and young people:
-
о Over the last 12 months, have you generally felt that the regular preventer treatment that your child (you) takes has kept their asthma under control? Yes/No. If you ticked ‘Yes’, please tell us what aspects of your child’s (your) asthma, or their daily life, have made you feel happy that they are on the correct regular medication. If you ticked ‘No’, please leave this question blank
-
о Over the last 12 months, have there been times when you felt that your child’s (your) regular preventer treatment should be increased or changed because their (your) asthma was not under control? Yes/No. If you ticked ‘Yes’, please tell us the reasons why you were not satisfied with the regular preventer treatment that they (you) were taking? If you ticked ‘No’, please leave this space blank
-
о Does anything worry you about the fact that your child (you) has asthma? Yes/No. If you ticked ‘Yes’, please tell us the worries you have about the fact your child (you) has asthma. If you ticked ‘No’, please leave this space blank
-
о Does anything worry you about the regular preventer treatment that your child (you) takes for their asthma? Yes/No. If you ticked ‘Yes’, please tell us what worries you have about the treatment your child takes for their asthma. Please be as specific as you can. If you ticked ‘No’, please leave this question blank
-
2.10.4 Maintaining patient involvement throughout a consensus process
2.10.5 Disseminating survey results to patients/the patient population
2.11 Determining ‘how’ to define and measure an outcome in the core outcome set
2.11.1 Choice of measurement instrument for an outcome
-
Step 1. Conceptual considerations
-
Step 2. Finding existing outcome measurement instruments, by means of a systematic review and/or a literature search
-
Step 3. Quality assessment of outcome measurement instruments, by means of the evaluation of the measurement properties and feasibility aspects of outcome measurement instruments
-
Step 4. Generic recommendations on the selection of outcome measurement instruments for outcomes included in a COS
2.12 Achieve global consensus
Collaborators n (%) | Participants n (%) | |
---|---|---|
North America, Europe | 78 (34) | 62 (27) |
North America | 53 (23) | 50 (22) |
Europe | 49 (22) | 42 (19) |
North America, Europe, Australasia | 13 (6) | 15 (7) |
North America, Europe, Asia | 6 (3) | 12 (5) |
North America, Europe, Australasia, Asia | 6 (3) | 10 (4) |
North America, Europe, Australasia, South America, Asia | 4 (2) | 10 (4) |
Australasia | 3 (1) | 2 (1) |
North America, Australasia | 3 (1) | 2 (1) |
Europe, Australasia | 3 (1) | 1 (<1) |
North America, Europe, South America, Asia | 2 (1) | 3 (1) |
North America, Europe, Australasia, South America | 2 (1) | 1 (<1) |
North America, Europe, Australasia, Africa | 1 (<1) | 3 (1) |
North America, Europe, South America | 1 (<1) | 3 (1) |
North America, Europe, Africa | 1 (<1) | 1 (<1) |
North America, Europe, South America, Africa | 1 (<1) | 1 (<1) |
Asia | 1 (<1) | |
North America, Europe, Australasia, South America, Asia, Africa | 4 (2) | |
North America, Europe, Australasia, South America, Africa | 2 (1) | |
North America, South America, Asia, Africa | 1 (<1) | |
North America, Europe, Australasia, Asia, Africa | 1 (<1) | |
North America, Europe, South America, Asia, Africa | 1 (<1) |
2.13 Study team and study committees
2.13.1 Study Management Group
-
A clinical lead in the area covered by the COS
-
A study coordinator with understanding, and ideally experience of systematic review and/or outcomes and/or Delphi studies
-
A qualitative researcher – if undertaking qualitative components of the study and not covered by the experience of the study coordinator
-
PPI research partners – ideally a minimum of two to provide support and cross cover
-
Other clinical or methodological experts depending on the individual needs of the study
2.13.2 Study Advisory Group (SAG)
2.13.3 Costing the project
-
Co-applicants/members of the Study Management Group – who are the co-applicants of the study, what role will each have and how much time will they need to fulfil that role including contributions to the final report and study dissemination?
-
Statistical support – consider statistical support for study design, analysis and reporting
-
Information systems – will a study database or bespoke software be required? If so include time for an information systems developer to provide these resources
-
Data management – consider any data management requirements of the study. For example, if using paper-based questionnaires, who will enter responses into the study database and who will populate and distribute follow-up questionnaires, if appropriate?
-
Study coordination – the amount of coordination will depend on the activities of an individual study. Generally, study coordination time to schedule study meetings, distribute and follow-up survey responses and to schedule and prepare documentation for focus groups, consensus meetings, etc. should be considered. The coordinator may also take responsibility for systematic reviews and other methodological areas of the study depending on experience, financial management and communication with the funder, ethical or governance approvals needed for the study, monitoring of participant consent, etc. and preparation of the final study report
-
Qualitative research – consider any qualitative aspects of the study and include time for data collection, analysis and write up
-
Study Management Group and Study Advisory Group (SAG) meetings (viii) – consider frequency, whether the meeting is face to face requiring travel costs or by teleconference only. Take into account the location of participants, local, national, international
-
Consensus meetings – you may need to include the cost of venue hire and subsistence; some locations will provide a per person cost that includes all facilities and subsistence. The location of the meeting and subsequent travel costs for participants should be included. If national and/or international participants are attending a location close to a main train station or airport would be most appropriate and the cost of venue hire can be considered against additional travel costs for attending an institutional venue
-
Focus groups – take into account who the participants are. If health care professionals then consider where the meeting will take place, travel costs and subsistence. Focus groups for patients/parents should also include travel and subsistence for each person. Focus groups for patients/parents might take place in an alternative setting that also acts as an incentive for attendance. For example, holding the session at a zoo or an aquarium which allows participants to access the attraction free of charge after the meeting. Childcare/carer needs should be included for patient (see ‘PPI’ section below for cost calculator)
-
PPI membership of study committees/groups
-
PPI co-applicants
-
Other PPI activities. Examples include parent/patient facilitators of workshops or consensus meetings
-
Training – consider whether PPI contributors require any training and who will provide this. This could include attendance at conferences, external training events or internal training. The time to attend the training together with travel, subsistence and childcare/carer costs should be included
-
Childcare/carer time
2.14 Reporting guidance
2.15 Quality assessment/critical appraisal
Chapter 3: Implementation, review and feedback
3.1 Background
3.2 Existing research on the uptake of core outcome sets
3.3 Implementation
3.3.1 Role of the COMET Initiative
3.3.2 Role of core outcome set developers
3.3.3 Role of trialists
3.3.4 Engagement with funders
3.3.5 Engagement with prospective research registries
3.3.6 Engagement with journals
3.3.7 Role of systematic reviewers
3.3.8 Engagement with Cochrane
3.4 Review and feedback
3.5 Future research into the uptake and implementation of core outcome sets
3.5.1 Clinical trials registries
3.5.2 Applications for funding and ethical approval
3.5.3 Citation analysis and surveys of reports of trials and reviews
3.6 Conclusions
Chapter 4: Discussion
4.1 Recommendations for practice
4.2 Recommendations for research
-
Use of existing qualitative research and secondary analysis of qualitative data as part of the foundations for the COS development
-
Use of qualitative research at different stages of COS development, including to update existing COS
-
Methods of qualitative data collection and of eliciting stakeholder perspectives on outcomes
-
Impact of qualitative research on stakeholder participation in subsequent processes
-
Impact of qualitative research on user confidence in COS