Exploring methods the for selection and integration of stakeholder views in the development of core outcome sets: a case study in reconstructive breast surgery

This research was integrated into the BRAVO study which developed a COS for RBS [15]. Full ethical approval was obtained for the study (REC-11/SW/0305).

The BRAVO study is reported in detail elsewhere [15], but in brief the study consisted of three phases: phase 1 – creation of a questionnaire with an exhaustive list of potential outcomes identified from systematic literature reviews and qualitative work with patients and health care professionals; phase 2 – two sequential surveys with 303 key stakeholders (215 patients and 88 health care professionals) using Delphi methods to prioritise outcomes; phase 3 – two consensus meetings, one with patients and one with professionals to agree the COS. This paper is informed by the initial round-1 Delphi survey conducted in phase 2.

One hundred and fifty-six professionals were invited to participate of whom 88 (56.4 %) completed and returned the questionnaire. This included 40 breast surgeons, 21 plastic surgeons, 20 CNS and seven psychologists from centres across the UK with a range of experience. Response rates were 71.4 % (40/56), 46.7 % (21/45), 44.4 % (20/45) and 63.6 % (7/11) for breast surgeons, plastic surgeons, CNS and psychologists, respectively. Fewer plastic surgeons, CNS and psychologists participated than hoped due to difficulty engaging these stakeholders; recruitment continued until the sample pool was exhausted. There was an even mix of men and women and there was a wide spread of years in post (Table 1).

Four hundred and thirty-four patients from three centres were invited to take part in the study of whom 242 (55.8 %) consented to participate and 215 (49.5 %) completed and returned the questionnaire. A good representation of each of the prespecified subgroups was obtained with the exception of women undergoing therapeutic mammaplasty (Table 1). Despite recruiting until the sample pool was exhausted, this group remained relatively under-represented with only 11.6 % (n = 25) of respondents undergoing this procedure type.

Table 2 presents all 34 outcomes included in the questionnaire and indicates the top 10 outcomes prioritised as ‘extremely important’ (score of 9) by: (1) the whole group, ignoring stakeholder status, (2) broad ‘patient’ and ‘professional’ groups ignoring stakeholder subgroups and (3) professional and patient stakeholder subgroups separately. Differences in the items prioritised are summarised in Table 3.

This paper has explored the impact of analysing results from a Delphi survey by broad stakeholder groups (professionals and patients) and specific stakeholder subgroups, on the top 10 items prioritised for a COS. When the participants were considered as two broad stakeholder groups (two heterogeneous panels), two additional outcomes were prioritised compared to when participants were considered as a single heterogeneous group. Six further items were prioritised when participants were treated as multiple heterogeneous panels (professional and patient) (Table 3). Pooling broad stakeholder groups or all the participant data led to oversimplification of views of the study participants, with potentially important outcomes being lost. Decisions as to which stakeholders to include in a Delphi process and how to integrate them (in terms of single or multiple panels) in the presentation of feedback and consensus criteria are likely, therefore, to impact on the final COS. Indeed, donor site complications, an outcome that was only prioritised by one patient subgroup in the initial round of the Delphi process was ultimately included in the final COS for RBS [15].

Breast reconstruction is a particularly complex area, yet this exploratory work demonstrates the importance of a comprehensive sampling of stakeholders to include in a Delphi survey to select a COS. The application of qualitative techniques, such as maximum variation sampling with or without a sampling matrix, may provide a robust framework for this, although other methods may also be appropriate.

Other studies have demonstrated the importance of stakeholder selection in the development of COSs [9]. For example, when the set for rheumatoid arthritis was developed by the OMERACT group it did not include patients’ views [36‐38]. Subsequent patient involvement demonstrated that fatigue was central from their perspective. This was added to the COS, highlighting, on a broad level, the importance of including all relevant stakeholder views [36]. Other Delphi surveys conducted as part of COS development have previously considered participants from professional and patient stakeholder groups and subgroups [14, 39, 40], but we are not aware of any that examined how this may influence selection of outcomes. Recent methodological work suggests that all participants should receive feedback for each stakeholder group [18] or stakeholder subgroup [14] separately to optimise consensus during the process of selecting COSs. This present study highlights differences in opinion across all of the stakeholder subgroups; presenting separate feedback for each subgroup enables participants to reconsider their scores in light of different views, hence enabling better consensus to follow. If stakeholder subgroups or broad groups are pooled, differences in opinion will be lost and potential consensus across stakeholder groups or subgroups reduced.

For the purposes of this methodological study we chose to rank outcomes based on the percentage of participants rating an item ‘extremely important’ (score of 9) and the top 10 outcomes identified. Core outcome set developers use a variety of consensus criteria within Delphi surveys for determining which outcomes should be included in the COS or taken forward to the next stage of the consensus process [17]. Commonly, these relate to a mean or median value for each outcome or a percentage of participants scoring an outcome as ‘important’. The top 10 was selected for this study to make comparisons across the different panels more straightforward; any other approach is likely to have identified similar differences in the outcomes prioritised.

Whilst this study suggests that all potentially relevant stakeholder subgroups should be considered as multiple homogeneous panels in a Delphi survey, it may not be possible to identify all the characteristics which influence outcome prioritisation. It may also not be feasible to adequately sample all identified stakeholder subgroups. Furthermore, different stakeholder characteristics (e.g. age and reconstruction type) may interact to influence which outcomes are valued. Such interactions may be difficult to disentangle in a Delphi process and preliminary qualitative work with stakeholders may be required to allow meaningful subgroups to be selected. In addition, it may be possible to conceive an almost infinite number of subgroups for a given condition. This would almost inevitably lead to under-representation of any single group and, depending on the criteria used to retain items, may lead to the retention of an unwieldy number of outcomes. Large numbers of subgroups may also complicate the provision of feedback in the Delphi process. Type of feedback has been shown to influence the prioritisation of outcomes in COS development [18]. How feedback is provided in the context of multiple stakeholder subgroups would, therefore, require careful consideration and further work is needed to determine how this process may be optimised. There is a need, therefore, to identify a priori the key professional and patient characteristics to consider as separate panels during the Delphi, ideally based on previous evidence. Breadth of experience can then be ensured with maximum variation sampling within each predefined subgroup.

It may also be argued that since the aim of the consensus methods is to determine a minimum ‘core’ outcome set, variation in outcome prioritisation by stakeholder subgroups at early stages in the process may be irrelevant since differences may disappear as the consensus process moves forward. However, if items are dropped early on because they are not deemed important by participants as a whole, or by all professionals or all patients; or if feedback is not presented for each subgroup separately, consensus across all stakeholders cannot be fully achieved.

Finally, this study may have implications beyond the Delphi process. One or more consensus meetings are an important step in agreeing and ratifying the final COS if uncertainty remains following the Delphi. Extrapolating the findings of this study, adequate representation from each identified stakeholder subgroup at these meeting would be vital to allow meaningful consensus to emerge. This may not be feasible if multiple stakeholder subgroups are involved, but should be advocated as best practice in future COS development studies when consensus meetings are considered necessary.

Core outcome sets can improve the quality and consistency of research and, hence, its value to patients, professionals and policy-makers [10]. Careful and appropriate stakeholder selection and integration of stakeholder views, however, are necessary to ensure that the resultant COS is valid and accepted. Further work is required to produce definitive guidance on this important issue, but the formal, robust and transparent recommendations presented here may aid future COS developers in the design and conduct of their studies and may be the first step to improving the validity and value of COSs. Widespread adoption of these recommendations may, therefore, promote the uptake and use of future COSs in practice.