Collaborating with front-line healthcare professionals: the clinical and cost effectiveness of a theory based approach to the implementation of a national guideline

Medical Directors at 14 NHS hospitals from Yorkshire and the Humber region of the UK were contacted by the project team in April 2011 to inform them of the opportunity for their hospital to be involved in a project which aimed to support the implementation of national patient safety guidelines. In total, four hospital Trusts expressed an interest, and at the initial scoping meeting for each organisation, Medical Directors and senior management staff were asked to prioritise one or two guidelines. Three hospitals (from here on in referred to as H1, H2, H3)^a chose to focus on a common guideline released by the National Patient Safety Agency (NPSA) in March 2011: ‘reducing the harm caused by misplaced nasogastric (NG) feeding tubes’.

Fine bore NG tubes are frequently used in the clinical setting. The delivery of enteral feed through NG tubes that have been inadvertently placed in the respiratory tract is not uncommon and can have serious consequences; between 2005 and 2011, there were 21 deaths and 79 cases of harm in the UK due to feeding into the lungs [17], 50% of which were caused by misinterpretation of X-ray. Although there is no completely reliable method for checking tube placement, the NG tubes guideline provides a total of 17 recommendations for management/clinical staff for preventing feeding into the lung, including: pH testing should be the first line method used to confirm the tube is in the stomach, and that X-ray is used only as a second line test when no aspirate from the stomach can be obtained or the pH indicator paper has failed to confirm the tube position.

The project staff worked with teams from each hospital using the six step TDFI approach to support the implementation of the NG tubes guideline. Full details of how each step was completed are reported elsewhere [15]. In summary the TDFI approach used six steps: 1) Team selection: multidisciplinary implementation teams (ranging from between 4–10 team members including doctors, nurses, and allied health professionals who were approached based on their expertise in, or enthusiasm for, the area of the guideline) were formed in each Trust; 2) Audit of current practice: staff audited NG tubes practice using co-developed audit tools to identify a target behaviour for change: increasing the ‘use of pH as the first line method to check tube position’ was the target behaviour identified by all three hospitals; 3) Identification of barriers to change: the Influences on Patient Safety Behaviours Questionnaire IPSBQ; [16] was distributed to relevant healthcare professionals in each Trust in both a paper copy and online format to assess the barriers to performing the target behaviour according to the domains of the TDF; 4) development of solutions to overcome barriers: following analysis of the questionnaire data, focus groups were held at each hospital with multi-disciplinary groups of staff from relevant wards/departments to discuss views regarding the specific target behaviour, and to devise interventions to overcome any barriers faced to performing this behaviour using guidance from evidence based behaviour change literature; 5) Implementation of solutions: a report outlining the process, findings, and suggested interventions was produced for each project and submitted to senior management in each Trust. Following authorisation for implementation, strategies were implemented; 6) Evaluation: post-intervention audits were undertaken by the implementation teams to determine the impact of the intervention on the performance of the target behaviour by staff within each organisation.

To support the above implementation methods, three implementation tools were required: an audit tool to assess performance – co-developed with each implementation team, the validated IPSBQ to assess barriers to change according to domains of the TDF, and a focus group schedule to develop further understanding about TDF barriers and to generate tailored implementation strategy ideas using TDF domain-mapped behaviour change techniques [18]. Details of how each of these tools have been developed are available elsewhere [15],[16].

Ethical approval was sought from the Bradford Research Ethics Committee. In view of the lack of involvement of patients, use of routine data and quality improvement focus of the project, it was classified as a service evaluation. Case note reviews were undertaken by trained healthcare professionals as part of the implementation approach. Focus group interviews with staff were undertaken following receipt of written informed consent; staff were made aware that all data collected would be anonymous and remain confidential.

This project is one of a portfolio of projects contributing to the objectives of the Yorkshire and Humber Academic Health Science Network, in particular translating research and into practice (AHSN). NHS England, which licenses the AHSN, asked it to provide case studies showing the potential rate of return on investment (ROI) on these. This project was such selected as a case study and thus its methodology adopts a ROI approach for the Yorkshire and Humberside region.

The return on investment (ROI) was estimated using the formula [20]:

Benefits and costs were estimated for the Yorkshire and Humberside region with a population of 5.3 million. The analysis was based on a total of 34 acute hospitals within the region. Estimated benefits arose from:

The main additional cost was for the intervention itself (see Additional file 2), which included: development and delivery of training and e-learning resources, poster and screensaver design and implementation, development of care pathway documentation, organisation of an awareness day/week, and the time devoted to the project by implementation team members. Estimates of resource use were obtained from hospital staff familiar with the intervention and current practice in order to minimise use of assumptions. The two main sources of unit cost information are “Unit cost of health and social care” [21] for cost of staff time and NHS Schedules of Reference Costs for examination and disease specific costs [22].

Baseline data involving 43 to 53 patients and post-intervention data involving 40 to 46 patients are presented in Table 2 and graphically in Figure 1. The use of pH as a first line method at baseline was lower in the intervention hospitals (H1 = 18.4%, H2 = 11.6%, H3 = 13.6%) compared to the control hospital (H4 = 45.3%), whilst the use of X-rays was higher in the intervention hospitals (H1 = 49%, H2 = 76.7%, H3 = 40.9%) compared to the control hospital (H4 = 24.5%). The risk ratio for pH use in the baseline period for intervention hospitals ranged from 0.26 (95% CI = 0.11 to 0.62) to 0.41 (95% CI = 0.21 to 0.79) relative to the control (Table 3 and Additional file 3: Table S3.1)^b. In the post intervention period, the use of pH testing increased significantly in the intervention hospitals (Table 2 and Figure 1). The relative risk of pH use after controlling for differences at baseline ranged from 3.1 (95% CI = 1.14 to-8.43, p < .05), to 8.14 (95% CI = 3.06 to 21.67, p < .001) compared to the control hospital, which remained unchanged (risk ratio (95% CI) = .77 (.47-1.26) p = .296).

For H1 and H2 the use of X-ray first line decreased (H1: from 49% to 23%; H2: from 76.7% to 9.1%), and for H3 the use of radiological insertion first line decreased between pre-post intervention implementation (from 36.4% to 10%). There was a 5% reduction in the use of X-ray first line in the control hospital (from 24.5% to 19.6%). H1 had an improvement in documented practice (from 32.6% missing documentation to 14.6%), whereas poorer documentation recording was found in H2 (from 9.3% missing documentation to 18.2%) and H3 (from 9.1% missing documentation to 17.5%) and the control hospital (from 30.2% missing documentation to 45.7%).

Table 4 presents the pre-post intervention use and unit cost of diagnostic tests, with the main change being greater use of pH tests and fewer X-rays first line. Staff requirements to undertake the tests were advised by a nurse and radiographer at one site, whilst the cost of an X-ray was advised by a finance staff member. Applying the unit costs to the change in the usage of tests provides estimated savings across the Yorkshire and the Humber region of £0.51 million.

The error rate in reading X-rays was taken from an audit of a similar intervention [23] and applied in this case. At baseline 192 X-rays were used to confirm placement of tubes, with 43 errors of interpretation found. Of these, seven placements were interpreted as unsafe when actually safe and five were in the lungs. After one year of implementing a project to improve the checking process, misinterpretation was found in one case across the three intervention hospitals. Avoided costs for each event include £43 for each tube re-inserted and £2,391 for the cost of pneumothorax injury [22]. The cost of a near-miss was estimated at £4,791, assuming a consultant-led team investigated the event (with staff members involved and duration of investigation advised by a clinician in H2). Total savings from fewer errors in reading X-rays were estimated at £60,427 per hospital, equivalent to £2.05 million across the region. Estimated savings of 10 minutes per year from no longer addressing the topic at team meetings were £14,970 per hospital, and £0.51 million for the region. Total estimated savings were £75,397 per hospital and £2.56 million across the region. The costs to deliver the intervention were estimated at £45,824 per hospital and £1.56 million across the region (see Additional file 3). Each hospital is assumed to hold an annual refresher at a cost per hospital of £18,192, equivalent to £0.62 million across the region. The estimated net savings per hospital in the first year are £29,573 rising to £57,205 in subsequent years and £1.00 m and £1.94 m respectively across the region.

The base case ROI, presented for 34 hospitals in the region, has a five-year time horizon, with costs and savings discounted at a rate of 3.5% per year. Sensitivity analyses are conducted assuming a 20% change in costs and savings. The ROI by year for the base case and various sensitivity analyses can be found in Additional file 4. The base case results show an excellent return even in year 1; the return improves each year with the improvement in diagnostic practice being maintained over time, but costs are lower because only a refresher course would be delivered in later years. The result is sensitive to both costs and savings.

Benefits of the TDFI approach include the ability to identify different barriers across organisations associated with the implementation of the same guideline, demonstrating the validity of the approach. These differences in key barriers indicate the need for tailoring of interventions to local contexts. This is another facet that the TDFI approach can offer using barrier-matched theoretically underpinned behaviour change techniques [9] alongside principles of implementation, such as incorporating co-designed interventions into established structures [13],[28],[29]. At the same time, where there is some overlap between organisations for the presence of specific barriers, this allows for sharing of interventions and avoids unnecessary duplication. This occurred on an informal basis in the current project, and implementation team members indicated this was useful because it saved time and promoted inter-organisation working [15].

The limitations of using the TDFI approach itself include the complexities associated with working with healthcare professionals to identify a specific target behaviour for change. The TDF indicates that the barriers and levers affecting the performance of a single target behaviour should be identified. However, clinical guidelines frequently include a range of recommendations that may require behaviour change, so identifying the most important behaviour to focus upon can be challenging [16]. Additionally, the time, skills, and resources required by a nominated person to use this approach with teams to elicit behaviour change within a healthcare setting, in comparison to other methods of implementation, are as yet undefined. Investment in training those tasked with guideline implementation (e.g., quality managers) to use the TDFI approach, and undertaking longitudinal follow up with attendees to understand their experiences, and the extent to which behaviour change occurs, may be useful in this regard. Such activity would also fulfil the much needed translational gaps that currently exist by mobilising knowledge (i.e., clinical guidelines) into practice using an evidence based and pragmatic method.

This is a pragmatic safety improvement project with an evaluation component. Evaluation of safety improvement initiatives is notoriously challenging [30], and pre-post intervention comparisons with a suitable control is frequently the most realistic approach [31]. Furthermore, this is the first time that ROI results have been presented, together with clinical outcomes from a TDFI project. To enhance ROI accuracy, resource use estimates were provided by clinical experts and unit costs come from national cost databases [22].

Despite our encouraging findings, there are methodological limitations of this study. Primarily, the hospitals were volunteers and were therefore not randomised to an intervention or control condition. The audit team were not blinded to case note reviews, and pragmatic factors meant that sometimes a mixture of retrospective and prospective reviews were undertaken, increasing the potential for bias. Furthermore, the control data we collected was from a single Trust on a retrospective basis, which encapsulated the timeframes covered in the pre- and post-intervention periods for which the data was collected in the intervention hospitals. This limits the extent to which we can be confident that the statistically significant differences we have found are reliable. However, these shortcomings are consistent with the well documented challenges associated with evaluating complex interventions in the healthcare setting, such as having to accept the restrictions that prevent the adoption of an ideal evaluation design, and the need to undertake evaluation alongside large scale implementation [32],[33]. Nonetheless, the collection of additional data to provide a comparison to the intervention hospital data offers further support of the potential value of the TDFI approach and therefore warrants progression onto a more sophisticated research design. It would also be useful to test the effectiveness of the TDFI approach for the implementation of a range of guidelines to ensure it is not only generalizable across organisations, but also across different areas of healthcare professional behaviour change. Finally, although we have previously reported implementation team member perceptions of the feasibility, acceptability, and sustainability of this approach through post-implementation interviews [15], this was not undertaken as part of a formal process evaluation. A qualitative process evaluation e.g., [34],[35] would have been valuable for providing some clarity around specific intervention effects.

The ROI approach also has limitations, including failing to capture other benefits particularly improved quality of life and the potential to reduce mortality. Moreover, some hospitals may elect to deliver the intervention and conduct refresher courses using different numbers and staff functions, and may not achieve and sustain the change in clinical outcomes observed in this study.

This work has revealed promise in the use of a theoretically informed framework for bottom-up implementation of guidelines, which provides a practical alternative to top-down implementation. It would be useful to further understand more about the transferability of the TDFI approach by assessing whether those tasked with eliciting behaviour change to improve the uptake of guidelines in healthcare organisations are able to apply the model in practice.

Implications for research include the need to test the TDFI approach under more robust research design conditions, to investigate the sustainability of the results (i.e., whether behaviour changes are maintained), and the generalizability of the methods for the implementation of different guidelines and across a range of settings. A crucial next step in the progress of this research will be to establish whether using a theoretically underpinned framework of implementation is more effective than providing support using implementation principles alone (e.g., adopting the perspective of the target group, obtaining management approval, mapping guidelines onto local problems). Furthermore, it would be useful to identify the extent to which the specific components of the approach (e.g., the identification of barriers and levers, co-designing interventions, using evidence based behaviour change techniques, etc.) contribute to success. Increasing understanding of exactly what works will enable evidence based refinements to be made to improve both the clinical and cost-effectiveness of this approach.