Background
Educational outreach visits
| A personal visit by a trained person to a health care provider in his or her own setting |
Reminders (manual or computerised]
| Prompts performance of a patient specific clinical action |
Interactive educational meetings
| Participation of health care providers in workshops that include discussion or practice |
Audit and feedback
| Any summary of clinical performance over a specified period of time |
Local opinion leaders
| Health professionals nominated by their colleagues as being educationally influential |
Local consensus process
| Inclusion of professionals in discussions to agreed the approach to managing a clinical problem that they have selected as important |
Patient mediated interventions
| Specific information sought from or given to patients |
Educational materials
| Distribution of recommendations for clinical care (such as clinical practice guidelines, audio-visual materials, electronic publications). |
Didactic educational meetings
| Lectures with minimal participant interaction |
Financial incentives
| payments directly rewarding health care providers for specified behaviours |
Multifaceted interventions
| A combination of two or more interventions |
Methods
Topic selection
How best to conduct audit and feedback?
Definition of audit endorsed by the National Institute for Clinical Excellence [4] |
---|
A quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change. Aspects of the structure, processes, and outcomes of care are selected and systematically evaluated against explicit criteria. Where indicated, changes are implemented at an individual, team, or service level and further monitoring is used to confirm improvement in healthcare delivery. |
Definition of audit used by the Cochrane systematic review [5] |
The provision of any summary of clinical performance over a specified period of time. The summary may include data on processes of care (e.g. number of diagnostic tests ordered), clinical endpoints (e.g. blood pressure readings), and clinical practice recommendations (proportion of patients managed in line with a recommendation). |
Stage
|
Recommendations
|
Addressed within Cochrane Review?
|
---|---|---|
Preparing for audit
| Securing stake-holder interest and involvement (e.g. professionals, patients or carers) | No |
Selection of appropriate topic, according to whether: | ||
• Topic concerned is of high cost, volume, or risk to staff or users | No | |
• Evidence of a serious quality problem | Yes: effects greater if low baseline | |
• Good evidence available to inform quality standards | No | |
• Amenability of problem to change | No | |
• Potential for involvement in a national audit project | No | |
• Topic is pertinent to national policy initiatives | No | |
• Topic is a priority for the organisation | No | |
Clear definition of purpose of audit, e.g. to improve or ensure the quality of care | No | |
Provision of necessary support structures, i.e. | ||
• Structured audit programme (committee structure, feedback mechanisms, and regular audit meetings) | No | |
• Sufficient funding (audit staff, time of clinical staff, data collection, feedback) | No | |
Identification of skills and people needed to carry out the audit | No | |
Selecting criteria
| Definition of criteria (structure, process and outcome) | No |
Validity and potential to lead to improvements in care | ||
• Evidence based | No | |
• Related to important aspects of care | No | |
• Measurable | Yes (implicitly) | |
Measuring level of performance
| Planning data collection | |
• Definition of user group (and exceptions) | Can't tell | |
• Definition of healthcare professionals involved | Yes (implicitly) | |
• Definition of time period over which criteria apply | Yes (implicitly) | |
Making improvements
| Identification of barriers to change | No |
Implementing change | ||
• Establishing the right environment (at individual, team and organisational levels) | No | |
• Considering external relationships (e.g. with patients or other agencies) | No | |
• Use of other supporting interventions (e.g. educational outreach, reminders) and / or multifaceted interventions | Yes: not supported by evidence | |
Sustaining improvement
| Monitoring and evaluating changes, e.g. continuing audit cycle, use of performance indicators | No |
• Appropriate organisational development (e.g. cultural change, adequate training) | No | |
• Use of existing strategic, organisational or clinical frameworks | No | |
• Leadership | No |
-
Does audit and feedback work for this condition and setting, specifically improving the care of patients with a chronic disease – diabetes mellitus – in primary care?
-
Does it work equally across all dimensions of care – from simple recording of cardiovascular risk factors to more complex areas of care such as glycaemic control? The latter requires a greater number of actions to achieve which include measuring blood glucose levels, reviewing the patient, checking compliance with drug and dietary therapies and checking patients' understanding of the condition.
-
How should it be prepared? Should data be comparative and if so, what should the comparator group be? Should data be anonymised?
-
How intensive should feedback be? Intuitively, providing more and personalised feedback on a recurrent and regular basis should have a greater impact on practice than a one-off report of (say) PCT-level aggregated data. However, it is uncertain whether the extra time and costs of ongoing data collection and preparing more frequent feedback would be matched by additional benefits.
-
How should it be delivered – by post or by a messenger in person? And if by a messenger who should this be? Professionals might be more convinced by a message delivered by a colleague with a recognised interest in diabetes care rather than a non-clinical facilitator.
-
What activities, if any, should accompany feedback? The likely costs and possible benefits of (say) educational meetings or outreach visits need to be weighed up against providing feedback via paper or computerised formats alone.
-
What should be done about the poorest performers detected by the audit? Targeting such practices may help close the gap between the poorest and best performers. Alternatively, spreading effort to improve quality more equally amongst all practices may improve average performance for the whole PCT.
Results
The evidence from the systematic review
-
audit and feedback can improve professional practice, although the effects are generally small to moderate
-
effectiveness varies substantially among different studies
-
variation may be related to different methods of providing feedback or contextual factors, such as targeted behaviours and professionals
Questions | Most relevant analyses from Cochrane Review | Evidence from all trials reviewed (n = 85) | Evidence from chronic disease management trials (n = 15) | Evidence from trials of diabetes care (n = 4) |
---|---|---|---|---|
Does audit and feedback work?
| Any intervention involving audit and feedback versus no intervention +/- educational materials | 83 comparisons: for dichotomous outcomes, median adjusted relative risk (RR) of non-compliance was 0.85 [Interquartile range (IQR) 0.74 to 0.96]* | Small to moderate effects in 11 of 19 comparisons | Moderate to large effects in two comparisons [12;13] |
Audit and feedback versus other interventions | Five comparisons: two show audit and feedback more effective than reminders; one that local opinion leaders more effective; one no effect over patient education; one no effect of audit and feedback with educational meetings over educational meetings alone | Small effect of audit and feedback over reminders from one comparison | None | |
Does it work equally across all dimensions of care?
| No direct comparisons; exploration of heterogeneity | No heterogeneity explained by complexity of the targeted behaviour | None | None |
How should it be prepared? Should data be comparative and if so, what should the comparator group be? Should data be anonymised?
|
Content. Patient information, such as blood pressure or test results, compliance with a standard or guideline, or peer comparison; versus information about costs or numbers of tests ordered or prescriptions | Two comparisons: no difference between peer comparison and individual feedback without peer comparison; nor between feedback on medication and feedback on performance | No difference between feedback on medication versus feedback on performance in one comparison | None |
How intensive should feedback be?
|
Recipients. Individual or group | No difference between individual versus group feedback in one comparison | None | None |
Frequency. Once only or more frequent feedback | None | None | None | |
Length. Once only feedback versus audit and feedback over a period of time | None | None | None | |
Short term effects compared to longer term effects after audit and feedback stops | Mixed results from 11 comparisons | No difference from one comparison [14] | No difference from one comparison [14] | |
Exploration of heterogeneity | No heterogeneity explained by intensity of audit and feedback | |||
Questions
|
Most relevant analyses from Cochrane Review
|
Evidence from all trials reviewed (n = 85)
|
Evidence from chronic disease management trials (n = 15)
|
Evidence from trials of diabetes care (n = 4)
|
How should it be delivered – by post or by a messenger in person? And if by a messenger who should this be?
|
Format. Verbal, written or both | None | None | None |
Source. Influential source [seen to be credible and trustworthy by the professional] or feedback from any other source | Two comparisons: peer feedback better than non-physician observer feedback; no difference between peer physician versus nurse feedback | No difference between peer physician versus nurse feedback in one comparison [11] | No difference between peer physician versus nurse feedback in one comparison [11] | |
What activities, if any, should accompany feedback?
| Audit and feedback with complementary interventions versus audit and feedback alone | No clear effect of complementary interventions from 14 studies including various comparisons except for small effect of audit and feedback combined with educational outreach. Lower baseline compliance associated with larger effect sizes. | Small or mixed effects in two out of four comparisons | Outreach by peer or nurse more effective than feedback alone [11] |
What should be done about the poorest performers detected by the audit?
| None | None | None | None |