Methods
The overall design and methods have been described previously in the published protocol [
20]. In brief, the overall design was a mixed methods parallel process evaluation which examined a set of pre-defined processes and their associations with change in high-risk prescribing at practice level. The quantitative element examined how change in prescribing at practice level was associated with practice characteristics and practice implementation of key processes and is reported separately (Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: quantitative examination of representativeness of trial participants and heterogeneity of impact. Submitted). The qualitative element consisted of comparative case studies in 10 of the 33 participating practices purposively sampled using maximum variation sampling [
21] to include a mix of those initially responding and not responding to the intervention by rapidly reducing their high-risk prescribing, as judged by visual inspection of run charts approximately 4 months after practices started the intervention. The case-study analysis of how practices adopted, implemented, and maintained the intervention is described separately (Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: case-study evaluation of adoption and maintenance of a complex intervention to reduce high-risk primary care prescribing. Submitted). This paper examines professional participant perceptions of the multicomponent intervention delivered to practices using qualitative analysis of interview data collected in the case-study practices.
In each practice, all interviews were carried out by AG (a researcher over 10 years of qualitative experience and already known to two of the practices from a previous project examining prescribing behaviour) [
22] with the most involved GP and one other GP, the practice manager and any attached primary care pharmacist approximately 6 months after the practice started the intervention, and the most involved GP again 9 to 12 months after starting the intervention to explore changes over time. Interviews were facilitated by a NPT informed topic guide and lasted approximately 1 h. As part of the intervention in all practices, the AG accompanied the pharmacist (TD) on the educational outreach visit (EOV) and made field notes detailing attendance and the practice’s response. Data was gathered between September 2011 and December 2013 and was in parallel with the trial to capture changes over time.
The analysis was concurrent and iterative with data generation allowing issues and themes identified to inform subsequent data generation and facilitate greater exploration. The analysis continued after data generation until no new themes emerged. The analysis was carried out by AG with BG contributing through discussion of data and interpretation until he became aware of the trial results, and was completed by AG before she knew the outcome of the trial, meaning that qualitative interpretation was blind to trial findings and detailed quantitative process evaluation data. Interview audio-recordings were transcribed verbatim. To preserve anonymity, some identifiable details have been changed and pseudonyms used. A coding frame was developed inductively from field notes and initial interviews and based on our topic guides, framework [
18], and logic model [
20]. The constant comparative method facilitated revision through detailed analysis [
23]. This coding frame was systematically applied to all data, facilitated by NVIVO 8. Analysis utilised the framework technique [
24] and NPT as a conceptual framework [
19]. AG analysed the data collected in the study twice, inductively letting themes emerge from the data and deductively based on normalisation process theory. NPT interpretation and coding reliability was established through a workshop with NPT experts. In this paper, we drew on analysis from the coherence construct which relates to participant’s perceptions of the intervention which was useful in identifying intervention components from the participant’s perspective. In our accompanying papers, we present the qualitative analysis from the remaining NPT constructs: cognitive participation, collective action, and reflective monitoring (Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: case-study evaluation of adoption and maintenance of a complex intervention to reduce high-risk primary care prescribing. Submitted) and the quantitative data analysis from the NPT informed questionnaires (Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: quantitative examination of representativeness of trial participants and heterogeneity of impact. Submitted). The data was explored for negative cases. Practice names have all been anonymised.
Results
The findings presented are from 38 professional interviews (ten lead GPs of whom nine were interviewed twice, seven GPs less involved with DQIP, nine practice managers/administrators and three practice pharmacists and from approximately 11 h of field notes from each case study’s EOV (one practice requested and received the education and training twice because of initial implementation failure).
Tables
1 and
2 provide a detailed description of the intervention components defined in the published protocol [
15,
20] based on the TIDieR checklist [
15] with illustrative study materials provided in the Additional files
1,
2, and
3. The intervention delivered to practices had three main components: (1) financial incentives; (2) education; and (3) a web-based informatics tool which used data extracted from GP electronic medical records. Each of these three main components had a number of subcomponents, included in the intervention for a range of rationales which are detailed in Table
2. Participant’s perceptions of these components and when they were perceived to have an effect are summarised in Table
2 and discussed in detail below.
Table 1
TIDieR description of the intervention (item 2 is shown in Table
2)
Financial incentives |
Up-front payment | £350 (547 USD, 497 EUR) paid to every practice paid by the research team immediately before practice started the intervention, the same for every practice. |
Payment per review | £15 (23 USD, 21 EUR) per review completed, paid by the research team after the end of the 48-week intervention period once the practice submitted an invoice, the same for every practice. |
Education |
Branding intervention ‘patient safety’ | Research team used the term ‘patient safety’ in all communications with all practices. |
Prescribing advice | Written by the research team for all practices (not tailored) and communicated and distributed at EOV on a one page laminated sheet. |
Structured written educational material reinforcing EOV | Written by the research team for all practices (not tailored) and available electronically from the tool and distributed at the EOV. |
Educational outreach visit (EOV) | 1 hour face-to-face meeting held in the practice and delivered by the research team to a common basic structure, but tailored according to practice interests and expressed needs (Additional file 1). The EOV summarised the latest research evidence, provided clear prescribing advice, and included training on the tool. |
Discussion about potential process to do the work | Discussions facilitated by research team during EOV tailored to specific practice characteristics and wishes. |
Newsletters | The research team sent a practice progress report to the lead GP and practice manager 8 weekly both before the practice started the intervention (non-tailored update on the progress of the trial and a reminder of the practice start date) and during the intervention period (tailored to reflect practice progress by providing a run chart, commentary on practice progress including comparison to other practices at the same point in implementing the intervention, and offers of further support) (Additional file 2). |
Informatics tool |
Identification of patients to review | Web-based tool hosted by NHS Tayside, and the same for every practice. Updated weekly using data-extracted from the general practice’s own clinical records. Practice’s had controlled access through personal log in details via the NHS intranet. At log-in, an updated list of patients needing review was available (Additional file 3: Appendix Figures A1 and A3). |
Summarise clinical information to facilitate review | The web-based tool provided a structured summary of relevant patient clinical information to facilitate review (summarised risk factors for relevant adverse drug effects, summary of recent relevant prescribing, Additional file 3: Appendix Figure A4). |
Recording of review decisions |
Run charts of change in prescribing | GPs were required to record their review decisions within the informatics tool in order to receive payment (Additional file 3: Appendix Figures A4 and A5. This information was used to ensure that reviewed patients where prescribing was judged appropriately were not repeatedly flagged for review (to increase efficiency by avoiding pointless re-review) except where prescribing judged inappropriate was restarted (to ensure follow-up of patients where intended care was not delivered). |
Timely visual feedback of progress (also used in 8 weekly newsletters in ‘education’) tailored for reach practice and available to view at any time within the informatics tool (Additional file 3: Appendix Figure A2). |
Table 2
Summary of the active and less active components of the DQIP intervention
Financial incentives |
In general | Attract practices to participate | Important for recruitment as symbolised recognition of the additional work required of GPs and generated extra income. |
Up-front payment | Increase practice commitment to doing the work as already accepted some payment | Had a limited role in mediating effectiveness. |
Payment per completed review | Ensure reach is maximised and work is maintained over trial duration | Practices said the financial incentive did not change what they did but two failing practices said had they known about the financial incentive they may have done more. |
Education |
Branding DQIP patient safety | Motivate GPs by appealing to their professional values | Important for recruitment as most GPs felt they could not ignore this topic. |
Prescribing advice | Avoid inertia | Had an important role in mediating effectiveness because GPs valued clear and concise prescribing advice and were able to action decisions quickly. |
Structured written educational material reinforcing EOV | Support and reinforce the educational messages delivered in the EOV | No perceived role in effectiveness. Two GPs used the one page laminated sheet when communicating with patients. Otherwise, this material was not referred to. |
Educational outreach visit | Persuade the GPs that the prescribing mattered and encourage GPs to perceive this as new and necessary work which required immediate attention | Had a limited role in mediating effectiveness because already persuaded GPs said they did not find the messages ‘new’, and the already less convinced GPs were not always persuaded that this was a problem worthwhile addressing. |
Discussion about potential process to do the work. | Motivate GPs to commence review immediately. | Had an important role in large practices for identifying an appropriate process and defining roles and responsibilities. |
Newsletters | Aimed to encourage continued reviewing activity. | Encouraged non-reviewers to revisit tool. Reviewers liked seeing their high risk prescribing going down. |
Informatics |
Identification of patients to review | Mobilise GPs to review by reducing administrative burden (at the time of the trial this was a labour intensive process primarily conducted by pharmacists and administrative staff). | Important for implementing change as GPs valued the tool’s simple case finding ability and did not question its accuracy. |
Structured clinical information to facilitate review | Facilitate efficient reviews by providing relevant information (reviewing was time consuming as involved reading patient’s notes to identify relevant information). | Important for effectiveness as GPs legitimised and valued the relevant and accurate data; however, all GPs continued to consult patient’s clinical notes. |
Record review decisions | Record data important for the trial and process evaluation. | Some GPs found the requirement to ensure all relevant information was addressed irritating. |
Run charts of change in prescribing | Motivate GPs to continue reviewing by comparison to previous performance | Had a limited role mediating effectiveness because GPs were not generally motivated by this in the web-based tool, although the same run charts were motivating for some when sent in newsletters. |
Financial incentive
GPs perceived that the offer of a financial incentive was important during recruitment since it was recognition of the additional work to be done in the context of already stretched work schedules and offered a means of generating extra income for the practice. The financial incentive was structured with an upfront payment of £350 ($600, €440) paid after the EOV and £15 ($25, €19) per patient reviewed paid after the end of the intervention in the practice. Most GPs said that the per-patient fee did not actually change whether or not they reviewed patients. However, in some practices, sampled for the case study analysis as early implementation failures (Orosay, Hellisay and Boreray), the interviewed GP felt increased awareness of the financial structure or payment could have facilitated greater implementation of changes in care for patients. In contrast, GPs in some practices which implemented the intervention immediately felt that DQIP was
easy money for work they should already be doing. One GP went as far as questioning the legitimacy of paying GPs for safety work:
“… whether we should be earning for doing that particular thing I think I would feel a little bit … mmm, you know, this is something that we should be picking up on probably without someone dangling a carrot really …” (Hirta Practice, GP 3 Interview).
Some practices had to be repeatedly reminded to invoice for the work done, supporting the belief that in at least some practices, financial incentives played a limited role in mediating effectiveness even though the offer of payment helped to get initial engagement during recruitment.
Education
The educational component had several elements; written material summarising the literature and providing prescribing advice, tailored newsletters summarising practice progress and offering support, and an EOV which both targeting knowledge and attitudes, and facilitated discussion of how practices were going to organise to do the DQIP work. All of these and the initial recruitment material were designed so that they clearly branded the work as being about patient safety.
Branding as patient safety and the NSAID and anti-platelet topic
Generally, participants in practices in the process evaluation said they signed up to the DQIP trial because they perceived that prescribing safety was clearly about good patient care and was therefore work they should be doing. GPs said that the NSAID and anti-platelet topics covered by DQIP were well-known safety issues, and the logic of the trial resonated with messages they had received from other sources, such as in Health Board organised ‘protected learning time’ educational sessions and from their practice pharmacist. As a result of this shared understanding, they perceived that their practice required good justification for not signing up for an intervention which targeted a well-known patient safety risk and paid them to do so. This high legitimation of the work also facilitated implementation of the intervention in all but four practices which took part in the trial and facilitated expeditious implementation in some practices.
“I felt this was actually genuinely useful, and of benefit to the patients… we’ve always had a bit of an interest in prescribing actually and quite enjoyed some of the little projects that we’ve done with the pharmacist…he is supporting this because it covers some of the issues we’ve already covered but in less detail.” (Monach Practice, GP 1 interview)
Prescribing advice, structured written educational material and educational outreach visit
The summary of the literature and educational material aimed to improve knowledge and motivate GPs to carry out the reviews. This was valued because it provided clear and up-to-date recommendations, brought the latest evidence and recommendations to the fore of their minds and aided consistent prescribing behaviour.
… it, it was good, there were references that you could go away along and, and read if you wanted … that’s one of the benefits of these things, is if somebody’s done the research and presents it to you and we’re all singing from the same Hymn sheet.” (Hirta Practice, GP 1 interview)
However, these messages were not viewed as ‘new’, and although participants valued the educational material, they did not refer to it while delivering care to patients, perhaps reflecting that its messages were easily internalised. These views were consistent across all participants in the process evaluation. Participants contrasted this summary material with what was normally provided during other improvement activities, which was typically longer, less focused and perceived as being less useful.
Overall, the EOV was viewed by interviewees as useful but not essential (see Additional file
1 for the presentation used). They felt it was useful to have a verbal summary of the literature and recommendations and an overview of the DQIP tool, but these were not felt to be essential to implementing effective reviews. Consistent with this, some GPs who led on delivering their practice’s reviews using the tool did not attend the EOV. Interestingly, one practice (Orosay Practice) which failed to implement the DQIP intervention by not conducting any reviews using the tool still observed a reduction in the targeted prescribing, potentially because of an effect from the educational material.
“Yeah it’s been because of DQIP really yes, aye. Well I suppose we were always anxious prescribing in the elderly… eh, but it’s merely made me sit up and pay a bit more attention…
AG: when you see the patient’s name on the tool or now more generally are you better informed?
It’s more now when I see them, and I didn’t even know if their name’s on the tool …em it’s more of when I see them and I see that they’re on it (NSAID or antiplatelet).”(Orosay Practice, GP 1 interview)
Discussion about potential process to do the work
During the EOV, practices were offered the opportunity discuss the ‘best’ process by which they could manage the work load. For some practices, these discussions played an important role in defining how they would organise the work, including roles and responsibilities. These discussions ensured the work commenced shortly after the EOV in the large initial implementation practices where administrative staff were given a co-ordinating role (Taransay and Hirta Practices), but were of less value in small practices where the intervention was being delivered by fewer people.
“…we started immediately…a list is issued by (administrative staff member named), em to whoever perceived to be the em doctor who sees most of that patient to action it each week, and then it’s, the boxes are ticked and it’s gone back to them to compute.” (Taransay Practice, GP 1 interview)
Newsletters
Newsletters, summarising practice performance using data from the tool, were sent to practices at eight weekly intervals. They were circulated around GPs in most practices, but were not formally discussed in any practice participating in the process evaluation (an example is shown in Additional file
2). In practices which immediately implemented DQIP, GPs felt the newsletters were nice to receive to see their targeted ‘high-risk’ prescribing reducing. In two small practices, with immediate implementation, the GPs responsible for the review work found the run charts showing practice progress motivating. Similarly, in two of the practices which initially did not implement the intervention, GPs felt the newsletters were motivating and pushing them into revisiting the tool and review medication.
“Oh yes! Yes we did … yeah we did, em … yeah it kind of pushed us into doing it (reviewing prescribing)…when we got the monthly newsletter.” (Hellisay Practice, GP 2 Interview)
In order to be paid, GPs had to use the informatics tool to record review decisions. The tool provided feedback on changes over time in targeted prescribing (run charts), identified patients requiring a review, supported review by summarising relevant clinical information extracted from multiple areas of the GP clinical record, and allowed recording of review decisions which then determined whether and when the patient would be identified as needing review again. Although some practices initially intended to use a print out of identified patients to do reviews, all but one GP carried out the medication reviews online using the tool (see Additional file
3, for example, screenshots). All GPs found the tool intuitive, straightforward and well structured, as the quote below illustrates:
“I liked it — I think we both did — it was, it was easy to use, it was intuitive, you didn’t need a half day tutorial on how to use the blessed thing.” (Hellisay Practice, GP 2 Interview)
The tool’s case finding ability was particularly attractive, especially identifying historical risk factors which GPs felt they were likely to overlook when carrying out a medication review using the same data in their own electronic medical record (e.g. previous peptic ulcer). The GPs also valued the focused data presentation, which only displayed information relevant to the prescribing decision.
“…it’s very straightforward in that you have the on-line tool and it gives you a very structured way of doing it which is, once you get into it much quicker, and other stuff that we do for the sort of GMS and the quality, the National Quality indicator things, em all involve us having to go in and do searches and find people and you know a lot of it’s done on bits of paper and things, so it, it’s done but it’s not, it’s not as easy, it’s, it’s actually made doing this really quite simple and that the tools you know, searches for the patients for you and things which is really good.” (Mingulay Practice, GP 2 Interview)
Although GPs were very positive about the tool, they would prefer a tool which was able to write to their clinical records to prevent double entry of review decisions, for example:
“I think it would be great, the only thing that would be really, really good is if it could talk to Vision [the clinical IT system]…” (Mingulay Practice, GP 1 Interview)
The majority of GPs also expressed a desire for a real-time intervention triggered by an alert when the patient consulted:
“…at the time of the prescription being done rather than retrospectively looking back at it, so that the prescription’s not issued in the first place, might be a good idea.” (Monach Practice, GP1 Interview)
However, a number of GPs preferred a DQIP-like intervention which retrospectively corrected prescribing, because they felt consultations were already heavily time constrained and preferred reviewing outside of clinical consultation to allow sufficient time to adequately review medication.
“Correcting past behaviour … that can make it … it’s a bit of a frustration but I can’t see how it would be otherwise really in the sense that, I don’t know how you fancy an alarm going off every time you sort of try to prescribe something…” (Gighay Practice, GP 1 Interview)
There were no differences in opinion about the informatics tool between professionals in practices which initially responded to the intervention delivered to them and those who did not.
Within the informatics tool were run charts of change in high-risk prescribing over time where practices could review their progress. These depicted the trend in their targeted high-risk prescribing 2-years pre-DQIP and since they started receiving the DQIP intervention. The run charts in the tool were not perceived to be of value by most GPs due to the relatively small number of patients identified in most practices, although as described above, the same run charts in the newsletters were motivating in some practices.
Although the component parts of the DQIP intervention were important at different stages of recruitment and delivery of the intervention, they also had interactions which were important for effective delivery and implementation. Consistent branding of the DQIP intervention and the financial component ensured GPs were on-board and engaged, the educational component provided clear messages which helped ensure consistency when making prescribing decisions using the tool, and the newsletters provided feedback and motivation to ensure engagement was maintained.
Discussion
Our findings have shown that all primary components of the intervention were ‘active’, but at different stages. The financial incentive was perceived as most important during recruitment because it acknowledged the additional work required and offered a means of generating extra income. The education including the focus on patient safety reflected in the way the work was branded, motivated the GPs to prioritise this work during initial implementation, and was valued because it provided clear and concise prescribing advice. Notably, many GPs perceived that some or all of the elements of education were useful but not essential, but individuals varied in terms of which elements they valued most. The informatics tool was crucial in case-finding patients, in facilitating efficient medication review decisions, and in implementing the required changes. Many GPs expressed a desire for a real-time alerting element to the informatics as well as support for retrospective review (although it is important to recognise that the majority of the targeted indicators would have triggered an interaction alert at the point of prescribing, consistent with point of care reminders not being a panacea in this area).
Overall, all respondents perceived that some of the components and sub-components were active and synergistic, but there was a variation between participants in which were most valued or perceived to be most active. There was therefore no clearly or consistently identified set of inactive components, and our interpretation is that this suggests that all component parts should be delivered in any further roll-out. However, our prior expectations of how and when different components would be active were not always correct. In particular, financial incentives were perceived by participants as less important than we anticipated, being active in recruitment and initial engagement, but not in promoting sustained delivery to all eligible patients (notwithstanding the caveats described above about how professionals might talk about such incentives in interviews). Although, the run charts and newsletters presented the same data, the run charts providing feedback on progress in the informatics tool were rarely looked at whereas the newsletters were valued in some practices possibly because the latter were accompanied by some individualised interpretation, emphasising that delivery mechanisms matter as well as what is being delivered. This is consistent with the wider audit and feedback literature which emphasises that feedback needs to be optimised to context to maximise effectiveness, although this study was not designed to examine this directly [
25].
It is important to note that professional interview data about financial incentives can be difficult to interpret, since some professionals may find it morally ambiguous to say that they require money to improve safety, making it more likely that they say that the incentives did not alter their actions. Nevertheless, some practices had to be repeatedly reminded to invoice for the work done, supporting the belief that in at least some practices, financial incentives played a limited role in mediating effectiveness even though the offer of payment helped to get initial engagement during recruitment.
It is also worth noting that the targeted prescribing topic and intervention were intertwined, in that participants were already mostly persuaded that the targeted prescribing was risky, and their perceptions of the intervention components have to be understood in this context. One implication is that the same intervention components might not be as effective if targeting a prescribing topic that GPs did not perceive as important, or which they felt was less under their control. This is consistent with the findings of a trial of a much simpler feedback intervention in Scotland where five of the targeted measures were similar to DQIP and reduced, but the sixth measure targeting anti-psychotic prescribing in older people was unaffected by the intervention [
26].
The normalisation process theory construct of coherence was useful in identifying and describing the components and sub-components of the intervention and when they were useful from the perspective of those who had engaged in some from with the DQIP intervention. For the analysis presented in this paper, we found normalisation process theory to be useful in understanding the nuances associated with collective implementation of the DQIP intervention in general practices where much clinical work is shared.
This study has a number of limitations. The GPs and practices which participated in this study are a sample of approximately a third of the whole trial population and were sampled as outliers at both ends of the distribution of initial success in reducing prescribing, so their views may not be truly representative. Data collection and analysis were carried out by a researcher (AG) who had involvement in the development of the intervention although the researcher with primary responsibility (TD) for the intervention development and the trial had no input into analysis and interpretation of this study. AG did have discussions with BG about the data analysis and interpretation. BG is responsible for the research programme grant, meaning that analysis had limited external scrutiny and challenge.
The analysis in this paper has focused on overall perceptions of when and how different components of the intervention were effective, but the accompanying case study paper (Process evaluation of the data-driven quality improvement in primary are (DQIP) trial: case-study evaluation of an complex intervention to reduce high-risk primary care prescribing. Submitted) qualitatively examining variation between practices in more detail indicates that this plays out variably in different practices, although variation in implementation is also significantly driven by variability in the barriers experienced. The quantitative analysis reported in the second accompanying paper (Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: quantitative examination of representativeness of the trial participants and heterogeneity of impact. Submitted). Found that 30/33 practices had at least some reduction in high-risk prescribing, which is consistent with the finding in this paper that all involved professionals perceived that some or all of the intervention components were active.
In the wider literature, the DQIP intervention is most like the intervention evaluated in the pharmacist-led information technology intervention for medication errors (PINCER) trial [
27]. Important differences were that PINCER targeted a broader range of prescribing topics than DQIP, that it was delivered over 12 weeks by pharmacists, and that it was more standardised in terms of how the pharmacists carried out medication reviews. However, the PINCER process evaluation did not explicitly examine the subcomponents of the intervention [
28]. A systematic review of main trial evaluations of pharmaceutical and non-pharmaceutical interventions found evaluations of non-pharmaceutical interventions are significantly less likely to list ‘active ingredients’ (those elements of the intervention intended to lead to change in the outcome, [
29], and we have not been able to identify any other process evaluations of cluster-randomised trials of multicomponent complex interventions which have examined the ingredients of complex interventions (although such studies are not easy to find because of poor labelling and reporting of process evaluations more generally) [
17,
18].
Acknowledgements
We would like to thank all participating practices, the Advisory and Trial Steering Groups, and Debby O’Farrell who provided administrative support.