Results
Theme saturation was achieved and three super-ordinate themes related to the four main research questions were identified:
1.
What is the task—how did advisory group members conceptualise what they were charged with doing (related to research question—What beliefs about the purpose and nature of evidence and recommendations may explain strategies to consider evidence and form recommendations)?
2.
Who gets heard—who contributed to discussions and/or influenced decision making (related to research question—who has most influence and does this lead to the dominance of particular approaches)?
3.
What is the process—what were group members’ experiences and evaluations of the advisory group process (related to research questions—What strategies are used in considering evidence and in formulating recommendations? What are GDG members’ views about the quality of GDG process and outcome)?
Views expressed by advisory group members who had been interviewed twice on the guidance development process were broadly consistent over time; there were no significant changes in their perceptions between the two interviews.
What is the task?
There were contrasting views on issues fundamental to the guidance development process—what recommendations should be used for and about the nature of evidence and how it should be considered.
What is the purpose of recommendations?
The range of opinions about the purpose of guidance recommendations highlighted the tension between recommendations being aspirational, serving as a long-term lever of change and being immediately implementable into current practice, considering financial and practical feasibility. Examples of aspirational purpose are:
‘If you only do what’s actually achievable, or easily implemented, um, you’re not putting any pressure on the system to change. And it’s quite useful to use guidelines as a lever of change’. (Healthcare professional 2, MH1c)
Examples of immediate implementability are:
‘I don’t think aspirational, because I think that just gives an excuse not to do anything’. (Service user, MH2)
A content analysis (Table
2) revealed an almost even split of statements supporting guidance as aspirational versus immediately implementable. This theme was discussed mostly in the mental and acute physical health advisory groups with only two statements about this issue in the public health group.
Table 2
Purpose of recommendations
Aspirational | 22 | 8 | 14 | 0 | 2 | 7 | 7 | 6 |
Immediately implementable | 24 | 7 | 15 | 2 | 6 | 3 | 5 | 8 |
What is evidence?
Two themes emerged from responses to questions about the nature of evidence: how group members defined evidence and how they appraised it.
Group members defined evidence in two ways, a ‘scientific’ definition based on peer-review literature presented to the group and a broader definition based on professional and lay experience of members and drawing on a wider range of sources.
There was awareness that different views were held about what constitutes evidence:
‘I would imagine there would be a range of views, from those people who think that the only kind of evidence worth looking at …d is meta-analysis … through to those people who…are sceptical about how well some of those trials defend themselves against threats to external validity’. (Healthcare professional 2, MH1)
Those subscribing to the ‘scientific’ definition viewed EBM as the gold standard of evidence that everyone should agree with:
‘Well, evidence is, in the setting of NICE guidelines really refers to high-quality evidence, which is evidence in randomized control trials’. (Chair, MH1)
Examples of the broad definition of evidence are:
‘I think expert opinion, patient opinion’s all evidence actually, if I’m pushed on it’. (Healthcare professional 1, AH1e)
‘There’s a number of different types of evidence I suppose. … gold standard, as it’s called, being the randomized control trial published in a peer review paper … but … evidence I suppose goes right the way down to individual experience as well’. (Service user, MH1)
Those who held a broad view of evidence thought there was the potential for exaggerating the significance of small findings with a narrow, ‘scientific’ conceptualisation of evidence:
‘People take a single small trial and over-interpret it. … there’s a sort of pretence that the more you systematize it … the more you get rid of these issues. But in fact you don’t, you just hide them. In some ways you make the problem worse’. (Healthcare professional, AH1)
Table
3 contains frequency data indicating that the mental health advisory group spent more time debating what evidence is than did the acute physical or public health groups. The advisory group chairs spoke more about ‘scientific evidence’ whereas service users spent more time talking more about evidence ‘of experience’, which reflects their respective professional backgrounds and experience.
Table 3
‘Scientific’ evidence versus evidence ‘of experience’
‘Scientific’ evidence | 29 | 6 | 14 | 9 | 6 | 15 | 3 | 5 |
Evidence ‘of experience’ | 24 | 8 | 10 | 6 | 1 | 11 | 9 | 3 |
In cases where ’scientific’ evidence was of poor quality, sparse, or contradictory, group members offered different solutions. Difficulties were anticipated in reaching agreement on recommendation based on weak evidence:
‘The bits of the guideline where the data is much less clear … it’s not randomized control trials with clear outcomes, it’s always the hardest bit in a way because you’re … relying on the groups to come to some kind of consensus expert opinion, which is always trickier’. (Systematic reviewer, MH2)
The imperative to provide a recommendation drove members to look for evidence other than randomized controlled trials. Some regarded this very much as ‘second best’:
‘Where we don’t have RCTs…we have to downgrade our standards, and we look for other kind of evidence’. (Health economist, MH2)
‘The whole point about NICE is to try and look at what the best evidence is and make recommendations on that basis. So coming up with a consensus is, I think, a bit of a second best’. (Service User Representative, AH2)
Reliance only on expert opinion was viewed by some as ‘pontification’ and there were concerns about how representative expert opinion would be:
‘There are some areas where … because there’s only one trial, or it’s only a controlled study … sort of were dismissed … because they weren’t sufficiently rigorous, but yet we were all allowed to pontificate and give our views, and get them in there’. (Healthcare professional 1, MH2)
Others considered evidence ‘of experience’ more positively:
‘Lack of evidence doesn’t mean to say that there’s … that’s a reason not to intervene at all. We do have to … use common sense and I think the same is true really to make sure that within the guidelines we’re not … taking lack of evidence as being negative evidence’. (Chair, AH2)
‘Where there was no evidence … so the recommendations were a little bit limited … we kind of had that freedom, as it were, to be able to draw on our own experience without needing this gold standard’. (Service user, MH2)
The role of context and judgement in interpreting evidence
Two factors emerged that influenced the interpretation of evidence: the role of contextual factors (such as the type of intervention) and the role of judgement.
There were different views as to whether evidence should be considered differently according to the context:
‘There were discussions … saying there’s no evidence for X drug being useful, do not use it. Whereas, in the psychological treatments, there was no evidence for Y being a very good treatment, but … it didn’t actually say do not use it’. (Service user, MH2)
‘In the world of science two and two makes four much more simply than in the world of education’. (Professional member 2, PH1f)
The opposing view is illustrated with these examples:
‘You can’t pretend that evidence depends on the discipline. The evidence depends on the evidence’. (Chair, MH2)
Judgment by advisory group members was identified as an important factor in the interpretation of evidence:
‘[Group members]’ve got different areas of expertise; otherwise the NICE team could just get the evidence and write the guidance’. (Chair, PH2)
‘It’s pooling human judgments, isn’t it? This isn’t a process you can do by computer, or algorithmically’. (Healthcare professional 2, MH1)
Technical team members appeared to be slightly more cautious than advisory group members about using judgement in interpreting and using evidence:
‘For health economics … it’s not as clear-cut … if there’s not evidence available, we are allowed to go away and do models. And … if there’s not evidence available … then we’re allowed to make assumptions … which you would never do for any of the clinical questions’. (Health economist, AH1)
Group members considered that interpretation of evidence was dependent on context (Table
4). There did not appear to be differences by advisory group or membership apart from members of the technical team who, possibly related to their role, did not discuss whether interpretation of evidence was dependent on context.
Table 4
Evidence dependent versus not dependent on context
Evidence dependent on context | 14 | 6 | 5 | 3 | 2 | 6 | 6 | 0 |
Evidence not dependent context | 3 | 0 | 3 | 0 | 2 | 1 | 0 | 0 |
Who gets heard?
There were challenges for the group in managing the diverse range of opinions and vested interests of group members and ensuring that group members contributed equally to discussion and decision making. Service users’ input was valued and there were strategies to guard against tokenism. Fostering good relations was seen as important to the functioning of the group.
The challenges of diversity
Effectively accommodating the diverse opinions of group members was a challenge for all three advisory groups. These challenges included minimising professional protectionism:
‘Because they’re … all coming from different backgrounds, so some of them will try to defend what they’ve done the whole of their lives’. (Health Economist, MH1)
Another challenge resulting from the diversity of members was allowing time for them to bond:
‘There was real uncertainty within the group, and quite a wide divergence of opinion … it does take people quite a bit of time to, to sort of settle down in their thoughts … and make decisions’. (Chair, AH2)
Dealing with tensions arising where opinions differed was also seen as a challenge arising from diversity within the advisory group:
‘There were a lot of product champions in this field and … I thought that there may be difficulty in accommodating their views’. (Chair, MH1)
‘If I’m honest, there were tensions between grumpy academics and… like me, and a, sort of, annoyance at the repetition of certain issues, really’. (Professional member 1, PH2)
Having a voice and having influence
Most members thought that discussion was dominated by some to the exclusion of other members, in particular service users:
‘I don’t remember many items we’ve discussed where I’ve heard a patient’s perspective sought or, or expressed’. (Healthcare professional 1, AH1)
‘It did strike me … that there are … some people who are more voluble and find it easier to get space on the floor than others’. (Healthcare professional 1, MH1)
There was also a perceived inequality between members’ contributions when it came to influencing decision making:
‘I think you would have to shout quite loud to say that you had equal influence’. (Service user, MH2)
‘I think there is a hierarchy of opinion-formers within the group… I think the hospital physicians would come nearer the top of it… Probably the … patient representatives [at the bottom]’. (Healthcare professional 2, AH1)
There were twice as many comments about there being unequal contributions of members to discussion and influence on subsequent decision making than comments describing equal contributions of members (Table
5). While this was the case among service users and health professionals, advisory group chairs were more likely to say there were equal contributions to discussion and decision making. This is of note given the responsibility of chairs to promote opportunities for all to contribute.
Table 5
Equality of contribution and influence
Equal contribution/ influence | 24 | 9 | 13 | 2 | 9 | 6 | 6 | 3 |
Unequal contribution/ influence | 56 | 17 | 22 | 17 | 6 | 27 | 19 | 4 |
Unequal influence was seen by some as beneficial for decision making by preventing unfocussed discussion:
‘I think the group has some people … that will take the lead and … have strong arguments and convince the rest of them, because some of them like to be really vague and go back and forth all the time’. (Health economist, MH2)
Managing vested interests
There was a general view that people would bring their own vested interests to the table, influenced by their experiences, their academic or professional speciality, or allegiance to particular ‘products’ (e.g., types of treatment). Service users acknowledged this and considered these to be part of their role in the group:
‘I feel like I’ve gone into it very biased … I suppose that’s something to be expected. I mean, presumably, I mean … patient members, they are going to have a very biased, very solitary view in some respect and I have got one’. (Service user, MH1)
They also expected others in the group would be biased towards their own area of specialism:
‘I think everybody will be going in with their own individual bias, because nobody is impartial in this’. (Service user, MH1)
In view of this, the independent, neutral role of the NICE reviewers was seen as particularly important. The Chair of one advisory group, while acknowledging vested interests, did not consider that they impeded work of the advisory group.
‘There was very little partisanship. It wasn’t as though people said, you’ve got to do things this way, because I’m representing this constituency, as it were, and you can’t ignore them. There didn’t seem any of that happening, really’. (Chair, MH2)
The Chair of one advisory group saw the technical team as using scientific evidence to guard against biases bought to the group by ‘product champions’. When interviewed at the end of the process the Chair thought bias had been controlled in the group.
‘[The] systematic reviewer was trusted to be completely independent and just rely on the data. … [group members] respected that she was a genuinely independent investigator, who had no particular bias towards any of the treatments on offer’. (Chair, MH2)
Some advisory group members were faced with the problem of members evaluating their own research or practice:
‘I think you’ve got a problem actually when you, you’re basing a recommendation made on your own research. You, you’re just too emotionally, um, involved with it’. (Health Professional 1, AH2)
‘[Members of the advisory group] who were very proud, fiercely proud, of what they did and were doing it for the best reasons. Very hard to challenge best practice when somebody is so committed to best reasons’. (Professional Member 2, PH2)
The theme of bias appeared to be more common in the acute physical and mental health advisory groups compared with the public health group (Table
6). This may be related to the nature of the evidence appraised by these two groups, specifically considerations around the need to minimize bias. Utterances were coded as ‘bias’ if they referred to instances of or perceptions that the vested interests of advisory group members were apparent in discussions and decision making; utterances were coded ’no bias’ if advisory group members reported that members were not influenced by any particular allegiances.
Bias | 24 | 9 | 14 | 1 | 6 | 8 | 5 | 5 |
No bias | 4 | 0 | 4 | 0 | 3 | 0 | 1 | 0 |
The role of the service user
The general perception was that the services users did a good job and their contributions of challenging health professionals and adding their perspectives as recipients of healthcare were necessary and valued:
‘Obviously service users know far more about the experience of receiving that service than those who deliver it do’. (Healthcare professional 1, MH1)
‘To be reminded of what the person with [condition x] feels … you don’t get that unless you have a proper service user on the panel’. (Chair, MH2)
Some concern was expressed that the inclusion of service users was tokenistic and that this was evidenced by there being an average of only two service users compared to six to eight healthcare professional members serving on advisory groups:
‘Ah, now then. I think the, um, service-user representation is a bit tokenistic’. (Healthcare professional 2, MH1)
‘I think it says something about how much experts by experience are valued, because if we’re all equally valued, there’d be equal numbers of people’. (Healthcare professional 1, MH1)
Service users thought they carried less influence than other group members leading some to question why they had been invited to participate:
‘If I voiced an opinion what wasn’t evidence based, ah, and I think, let’s say for example, (clinician A) or (clinician B ) had opinion, their opinion, I think those would have had more sway than mine’. (Service User 1, AH2)
There appeared to be strategies in place to protect against the presence of service users as a ‘box-ticking’ exercise and to support them within the team:
‘[The Chair] makes a real effort to make sure that … he directly asks [the service user] things if he hasn’t been saying much and he tries to get, get a view’. (Health Professional 2, AP1)
Group composition
The importance of fostering good social relationships was noted. Opportunities to socialise during the process, such as dinner before the meeting, facilitated more relaxed and productive discussion and led to the development of a more cohesive group:
‘I think there does need to be that team-building bit at the beginning’. (Professional Member 2, PH1)
‘The first few GDGs were a bit stilted as people were finding where, you know, how the group was going to operate, and I think now there’s a bit of banter and, um, you know, well, people just know each other better, so it, it works more, more smoothly’. (Healthcare professional 2, AH2)
This view appeared to be consistent across all groups.
What is the process?
Group members spoke about the value of debate and the need for marshalling discussion to retain focus and avoid conflict. Most members were satisfied with the process and output.
Managing discussion
Discussion was encouraged by some members as it was seen as a valuable part of resolving disagreement and necessary to consider a variety of perspectives in order to improve the quality of recommendations:
‘I do genuinely believe that the more debate the better the findings’. (Professional Member 1, PH1)
‘It would be a terrible shame if you had a group which did have no dissenting voices at all. It would be very dull and probably produce very unchallenging … results at the end’. (Chair, AH2)
Discussion appeared to be curtailed for two reasons: to stop unfocussed time-consuming debate (task-focused chairing) and to minimize the potential for conflict between group members (emotion-focused chairing). One strategy used by the Chair and co-Chair in one advisory group to keep discussion on track was to actively steer the group away from conflict:
‘He [co-Chair] is very skilled indeed at, um, bringing a group round to an opinion, which represents consensus and steering them away, similarly, from areas where they might disagree’. (Chair, MH2)
Another strategy to curtail discussion was to steer members towards consensus:
‘It can be the case that we’d spend a hell of a lot of time discussing, but actually not pushing on with the task … we just needed to be pushed, and when pushed, we came up with the goods’. (Healthcare professional 1, MH2)
But these strategies also created some dissatisfaction about the lack of debate:
‘By the time we’d got to the stage where actually we were comfortable enough as a group to actually have some of those very frank discussions … the NICE guideline machine … took over, and we became … focused on the … nicety of text’. (Healthcare professional 1, MH2)
Members suggested that the lack of debate would have a negative impact on decision making and the quality of guidance developed:
‘It all seemed a little bit tame … which is all right, you know, I mean, who wants to go to work and have fights—but it did just, it surprised me, and maybe it was a little bit ineffectual because of that’. (Service user, MH2)
Satisfaction with process and output
Members expressed satisfaction with the process of developing the guidelines.
‘I think it’s gone remarkably well ah, considering how the, how the groups meet up, and how sort of limited work, is actually done within, within the meetings’. (Service user, MH2)
‘Well, this is my first encounter with a NICE guideline group, and I have to say it’s been a tremendously uplifting experience, because it’s gone extremely well, in my view, and it’s been very smooth by and large’. (Chair, MH2)
Some members thought there was not enough time to complete recommendations, with a sense of being rushed towards the end:
‘The actual wording of the recommendations struck me as a little bit rushed and a bit of a free for all towards the end’. (Healthcare professional 2, MH2)
‘There was a bit of steamrollering going on. We were, more or less, accepting things, because there was no time left to discuss it properly’. (Chair, MH2)
While other members were more pragmatic:
‘There’s a finite time you can spend on each recommendation, arguing over one preposition, isn’t there? You’ve got to be very pragmatic about it, and say, actually that’s good enough’. (Healthcare professional 2, MH2)
Factors that influenced satisfaction with the process included small group work. This was valued as a way of working effectively and allowing equality of input:
‘If you’ve got a small group people interact a lot better’. (Health economist, AH2)
‘These smaller group working sessions … [ARE] … an opportunity for people who … prefer the smaller group to, to be able to get their views across’. (Community Member 2, PH1)
‘They (ADVISORY GROUP MEMBERS) tend to be more productive in their small groups’. (Chair, PH1)
The guidance development process was considered to benefit from a transparency of process:
‘For me it’s transparent, validated, rigorous processes’. (Professional Member 1, PH2)
‘One of the nice things which, I think, is central to the NICE process is transparency. And I think, I think we were very transparent all the way through’. (Chair, MH2).