No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention

In the process of developing guidance for A&F intervention development, three linked concepts were identified: identifying known best practices; applying relevant theory when operationalizing best practices; and considering intervention components as factors to be manipulated.

Although the ‘ideal’ design for A&F interventions depends on the recipient, their context, and the targeted behaviour, ‘best practices’ can be guided by theory and evidence. The consensus judgment of ‘best practices’ for designing A&F interventions are summarized in Table 1, and evidence in support of these practices is reviewed below.

The meta-regression in the Cochrane review of A&F indicated that feedback is more effective when it is presented both verbally and in writing than when using only one modality and when the source (i.e., person delivering the feedback) is a supervisor or senior colleague rather than unknown investigators. Conversely, feedback is less effective when it comes from a regulatory body, agreeing with qualitative work suggesting that feedback perceived to be potentially punitive appeared less effective than a supportive approach [15, 20]. Feedback intervention theory [10] suggests that feedback that captures the recipient’s attention and directs it toward a specific behaviour is more likely to be successful, but when A&F comes from a regulatory body, the recipient may be more likely to activate affective processes (e.g., distress) which distract attention from the specific task requiring change. Diffusion of innovation theory [21] would suggest that feedback from opinion leaders is likely to be more effective, but this was not investigated by the Cochrane reviews of opinion leaders [22] or the A&F review. Aside from exerting social pressure, in-person delivery of feedback could allow for tailoring of the message according to recipient characteristics. This type of process appears to offer a potentially important and readily testable hypothesis. Finally, the A&F review suggested that repeated feedback cycles led to greater improvements in performance than once-only feedback. In part, this may be because recipients are more likely to perceive the data as relevant and accurate when it is delivered closer in time to their own performance [9].

Although the Cochrane review of A&F did not make a specific recommendation, there is a theoretical basis for including comparison data in feedback as a means of motivating recipients to change behavior. Specifically, the theory of planned behaviour emphasizes the potential for social pressure to influence intentions to change professional practice [23]; this could help explain why feedback delivered by senior colleagues tends to be more effective than feedback delivered by unknown investigators. When feedback is delivered via written reports, a preferred approach to the inclusion of normative data for comparison has been identified through a large head-to-head trial [24]. In particular, recipients who received feedback with ‘achievable benchmarks of care’ [25] (i.e., comparing to the top 10% of peers) had greater improvements in processes of care than those receiving feedback where the median performance of peers was highlighted.

The Cochrane review indicated that feedback is most effective when targeting performance with large room for improvement. This may be attributed to larger discrepancy with the comparator, resulting in greater intention to take action, or may reflect the absence of ceiling effects, or both. In addition, the likelihood of feedback identifying a discrepancy between actual and ideal care leading to changed behaviour may depend upon the degree to which recipients believe they have control over that behavior and are committed to change [11]. Feedback should describe achievable behaviors that are ideally desired by the recipient and implementable by the member(s) of the healthcare team responsible for enacting the desired change [15]. This may, in part, explain the tentative findings in the Cochrane review that A&F was less effective when the targeted behaviour changes were more complex (e.g., when special skills are required to implement the desired change).

An observational study showed that clinical practice recommendations rated as incompatible with clinician values and norms were associated with lower compliance at baseline but greater behavior change following feedback than recommendations perceived as highly compatible [26]. This illustrates the potential for A&F to have an effect even upon targeted behaviours reported by clinicians not to be compatible with their values [20, 24].

The Cochrane review findings that feedback interventions were more effective when they included explicit goals and an action plan are not surprising [3]. This is consistent with theories [11, 13] positing that goals can make feedback more salient and action plans specify and facilitate the steps required to achieve goals. When goals are more specific, this process is more effective [9]; ideal goals are commonly considered to be specific, measurable, achievable, relevant and time-bound [27]. Feedback intervention theory proposes that action plans help focus recipients’ attention more productively on the task [10]. In healthcare, providers often manage multiple competing goals with their patients [28]; when A&F successfully directs attention toward specific tasks, it may influence prioritization of these goals.

From a cognitive perspective, receipt of feedback that indicates a discrepancy between expected and actual performance may generate a response described as cognitive dissonance [12]. In this case, participants may attempt to reduce the discrepancy by planning to improve [13] or by undermining the saliency of the discrepancy by discounting the data (or the goals targeted) as invalid [16]. Facilitating the former and reducing the latter is an important aspect of designing effective A&F interventions.

While the optimal design of A&F will vary to some extent for each purpose or setting, the strategies chosen for each component (Table 1) should be justified both by empirical evidence and theory [29]. If theory is used, it allows the evaluation of mechanisms of action, thereby contributing to an understanding of why any observed effects occurred. Although a single, all-encompassing, highly predictive theory of A&F is unlikely in the foreseeable future, a number of theories and theoretical constructs are informative [12]. For example, at the individual level, increasing self-efficacy [30‐32] and goal commitment [11, 33] is likely to increase performance. Organizational and system-level factors that may affect recipients’ ability to act upon a discrepancy highlighted by feedback could also be considered, including competing priorities for the recipients, teams and organizations [34]. This has implications not only with respect to potential contextual effect modifiers, but for the design of the intervention itself: the ideal number and nature of goal-directed behaviours targeted by the feedback may vary by setting and by the organizational level at which the feedback is directed. For example, managers may be able to assimilate and act on a greater number of topics than frontline clinicians, but may lack direct control over specific behaviors.

A&F interventions consist of multiple components, each requiring attention during the design stage; ensuring the inclusion of all desirable components and designing each component in the optimal fashion is rarely possible. We recommend considering individual intervention components as ‘levers’ to be manipulated when working within setting-specific or study design constraints. For example, if circumstances dictate that the delivery of feedback cannot be repeated in a reasonable timeframe, extra attention should be paid to other aspects of the intervention to ensure that discrepancies between actual and desired practice are salient and actionable.

In addition, co-interventions, tailored to overcome identified barriers and boost facilitators, may help if feedback alone seems unlikely to activate the desired response [35]. Ideally, co-interventions that are likely to enhance the process and outcome of A&F interventions should be selected, based on careful consideration of factors that are likely to influence the target behaviours. This gives the opportunity to investigate potential synergy between multiple implementation strategies. For example, while it may seem intuitive to combine payment incentives with A&F, in recipients who are already highly motivated to achieve the goals but have limited perceived control over the outcomes, this combination may not be effective [33]. Alternatively, co-interventions that increase recipients’ skills in quality improvement may be useful because skill development and resultant self-efficacy may increase the likelihood that feedback will lead to improved performance [11, 36]. Since multifaceted interventions featuring A&F may offer little benefit over A&F alone [3], cost-effectiveness of co-interventions should also be considered.

To summarize our guidance for developing A&F interventions, the best practices outlined in Table 1 should be considered and applied when possible. In particular, prior to conducting A&F, the nature of the behaviour change required to improve performance should be elucidated in order to develop clear action plans. Theory and evidence should be made explicit with respect to proposed design choices when manipulating the design of individual intervention components, and the expected mechanisms of action should be justified to aid interpretation.

There is a need for future research to turn its focus from evaluating A&F interventions against usual care to explicitly evaluating different types of A&F. In particular, confirmatory head-to-head trials examining different strategies for implementing the tentative recommendations in Table 1 would provide a valuable contribution to knowledge and practice. However, even if the components featured in Table 1 were optimized, additional components of A&F (that might be currently known or unknown) or co-interventions would be worthy of testing. Additionally, further knowledge regarding potential effect modifiers is needed.

Examples of potentially fruitful research topics that could lead to enhancing the effectiveness of A&F are listed in Table 2. These topics are organized by factors related to: a) contextual and recipient characteristics that may moderate A&F effectiveness, b) the design of the A&F intervention itself, and c) the characteristics of the targeted behaviour change. Knowledge about organizational resources and the nature of the team responsible for achieving the outcome of interest could help explain variability in the effects of A&F interventions [37]. The complexity of clinical practice recommendations and the degree to which benefits can readily be observed may play an important role [21, 38‐40]. More information is needed to understand whether some behaviours are more amenable to change through A&F and/or how to adapt the design of A&F accordingly.

There are opportunities to embed research within ongoing organizational initiatives at low cost in settings where routine data are available and where mechanisms for audit are in place, and this should be considered in the planning and conduct of A&F studies [41]. In particular, A&F researchers could partner with those providing routine feedback to healthcare providers to test variations of intervention components that might increase effectiveness. Administrators of quality improvement initiatives involving A&F may be keen to participate in head-to-head trials comparing different A&F designs because, in this case, all providers receive an intervention, while management learns about how their programs may be improved. Thus, settings where valid data for audit and mechanisms for feedback are already established are potential implementation research laboratories, enabling relatively rapid testing of multiple intervention components.

There is evidence suggesting that physicians respond differently than other health professionals to implementation interventions [42], and it is possible that this is also true for A&F. Even within the same broad professional discipline, there may be differential responses related to values and contexts of different groupings (e.g., primary and secondary care providers). The Cochrane review found little difference in effect size between A&F targeting physicians and non-physicians, but 86% of the trials focused on physicians. It is plausible that the effectiveness of A&F and key effect modifiers may vary in less-studied settings where non-physicians are the primary providers, such as home care or long-term care.

Opportunities for advancing the science related to A&F may be found through consideration of different study designs. When evaluating the addition of a single component, such as adding action plans to feedback, a two-arm, head-to-head trial is a logical option [43]. However, where there is an opportunity to consider many A&F-related factors at once, a multi-phase optimization strategy including a fractional factorial trial offers an efficient approach [44]. In such a study design, an investigator could simultaneously test various components of A&F content and delivery of the intervention. Existing A&F-relevant theories should be used to select the intervention components to be studied and to generate hypotheses regarding the interaction between such components and contextual characteristics, including those of the recipient and of the targeted behaviour. In addition, a process evaluation within a trial can be used to investigate and develop program-specific theory [45]. Examining mechanisms of action and qualitative studies prior to trials may help identify barriers and facilitators to change that can inform intervention design [15].

To summarize, we believe that future A&F research should prioritize studies that aim to determine how the effect of the intervention may be optimized rather than assessing whether the intervention can increase the implementation of evidence-based practices. This justification is similar to the arguments made in support of the shift toward comparative effectiveness research elsewhere in healthcare. Table 2 provides example topics in which greater understanding is needed to determine how best to utilize A&F. On the basis of the considerations reviewed above, we recommend that, prior to directing further research effort toward new trials of A&F versus usual care, investigators consider the following questions: