A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory

Audit and feedback (A&F) defined as the provision of clinical performance summaries to healthcare providers and organisations [1] is a well-used approach to support clinical behaviour change [2]. The increasing availability of health data in an electronic format (e.g. in Electronic Health Records), significantly increases potential for use of these data to provide electronic A&F (e-A&F).

e-A&F can be defined as the utilisation of interactive computer interfaces to provide clinical performance summaries to healthcare professionals [1, 3‐7]. It aims to support the decision-making process or guide team management [3‐7]. Although A&F is generally used when the patient is not present (e.g. like in bedside consultations, thereby making it distinctly different from computerized clinical decision support tools), e-A&F interventions specifically target clinicians or their managers and can aid improvement of patient care by providing timely or even real-time information for decision-making as part of operational management [8]. Furthermore, the interactive computer interface may allow users to filter, drill down and further explore their performance summaries.

Mechanisms of how A&F leads to behaviour change are variable and largely ignored in both individual and team-based contexts [9, 10]. While individual-based feedback is desirable [11], feedback to providers organised in teams or organisational units (e.g. whole facilities or departments) may offer a more scalable implementation model appropriate for low- and middle-income contexts [12]. In team-based care, multiple healthcare professionals are responsible for the same patients, and complex coordination is required [13]. Given previous A&F research showing team processes to explain more variance in outcome than practice structure[14], e-A&F interventions might additionally better facilitate improvement in team-based settings by addressing the aforementioned features.

A&F is posited to increase accountability and quality of care through implicit behaviour regulation of healthcare professionals [9]—given it involves techniques of goal setting, monitoring and providing feedback [15]—and is postulated to be most effective when its design is guided by theory [9, 16, 17]. However, explicit use of theory in A&F interventions is scarce [18]. As a consequence, little is known on the more specific topic of how e-A&F interventions may enhance the quality of care.

It is noteworthy that barriers to behaviour change can be influenced by A&F [19] and that these barriers differ across clinicians, originating from differences in clinicians’ training, knowledge, work experience, personality and other individual characteristics. These barriers are complex and dynamic (they are influenced by ongoing changes in the healthcare organization which in turn influence clinicians’ behaviours) [20]. Use of theory can help direct predictions on the effect size of audit and feedback used to help clinicians’ behaviour change.

A&F interventions with graphical or written presentations, to our knowledge, provide feedback in the same format for all recipients. In this way, A&F is not sensitive to individual differences in barriers to behaviour change given the media platform. e-A&F could help address this individual-indifferent approach in applying theory to overcome this significant limitation for traditional A&F presentations [21].

However, when explicit theory underlying implementation interventions is absent, it may be possible to retrospectively identify the theoretical domains they were likely to target [22]. This can be achieved through use of broad theoretical frameworks, such as the Theoretical Domains Framework (TDF) [22, 23]. The TDF comprises 12 theoretical domains and 128 constructs from 33 behaviour change theories. It was developed using an expert consensus and validation process to identify an agreed set of theoretical domains that could be used in developing implementation interventions [22, 23].

We expect knowledge, skills, social/professional role and identity, beliefs about capabilities, environmental context and resources, beliefs about consequences, motivation and goals, behavioural regulation and nature of the behaviours and social influences TDF domains to be inherently targeted by e-A&F interventions. This expectation is informed by component theories such as normalisation process theory [24, 25] theory of planned behaviour [26] and control theory [27]. Our detailed justification for the selection of these domains is provided in Additional file 1. However, we are yet to come upon literature detailing how emotion domain was targeted by electronic quality improvement initiatives. Based on the context, not all domains might be relevant in all e-A&F interventions.

Identifying and summarising the theoretical concepts targeted by e-A&F interventions for primary and hospital-based care and exploring how these factors might influence the interventions’ effectiveness could contribute to better e-A&F design. Ultimately, this may lead to e-A&F to become a more reliable approach to improving the quality of clinical practice.

We aimed to conduct a systematic review and meta-analysis of randomised controlled trials that evaluated the effectiveness of e-A&F interventions for clinical practice in primary care and hospital settings. Our objectives were to (1) assess the effect of these intervention on quality of care; (2) identify common aspects of the TDF employed as mechanisms of behaviour change in these intervention, and (3) explore links between identified TDF aspects, their nature or pattern of use across interventions and the magnitude of their effect size.

Table 1 describes the characteristics of included studies and the e-A&F interventions they evaluated. Study settings varied, but all were from developed countries, with three studies conducted in very specialised settings, i.e. ancillary [33] and specialised cardiovascular units [34, 35] respectively. Only three out of the seven studies targeted interdisciplinary clinical teams [34‐36].

Benchmarks provided in the e-A&F reports most commonly offered comparisons of individual performance versus average local and national performance [33, 34, 37‐39], or local site performance versus performance of all participating study sites [36, 40, 41]. Our definition of benchmarks is defined in detail elsewhere [42]. If there were other quality improvement (QI) strategies used, we assessed the extent to which one would reasonably consider e-A&F to be the key intervention to which the effect size would be attributed. Three categories identified were: (1) whether e-A&F was optional (minimal), (2) whether e-A&F was mandatory but included other QI interventions most of which were not implemented within e-A&F (moderate), (3) whether e-A&F was mandatory and included other QI interventions most of which were implemented within e-A&F (core).

In two studies, the intervention allowed clinicians to set their own goals or actions and track them [34, 35], with the rest utilising guidelines from professional bodies and/or evidence from previous studies as the study goals.

With regard to their interactive characteristics, six e-A&F interventions allowed recipients to select which additional indicators to include in their feedback report, in three cases feedback recipients, could drill down to specific patient population details [36, 38, 40, 41]. The implementation of the e-A&F interventions varied in design and form. Four studies created web panels containing patient data to be used for A&F, with one using stand-alone software program [39], and another one implementing an integrated EHR tool [38]. One study combined the electronic performance data with a software program that rendered it digitally and distributed these ‘updates’ regularly [36]. Interventions that had not been integrated in electronic health records had a separate process of data collation, with data being queried from medical records to populate a separate e-A&F tool.

In describing the control arm of the study, only three studies went beyond stating usual care and gave a clear detailed description of what the intervention was being tested against [34, 35, 39]. None of the studies randomised feedback design elements within intervention arms, but one did compare outcomes within the intervention arm based on rate of use of e-A&F [38]. Of the seven studies included, studies with the highest number of participants [35, 41] had a low risk of bias; three had a high risk [34, 36, 38] (see Fig. 2). The most common sources of a high risk of bias related to blinding of participant and personnel selection bias; clarity of reporting regarding the risk of bias was often insufficient (see Additional file 4 for a summary of risk of bias assessments across studies).

Table 2 displays the effect size and associated 95% confidence interval (95% CI) for each included study. Three studies found a positive effect of the e-A&F intervention on the quality of care. Peiris et al. reported a statistically significant difference of 9.4% (OR 1.47, 95% CI 1.41–1.53) between the intervention and control group for the number of patients who received appropriate screening of cardiovascular risk [40, 41]. Thomas et al. reported a statistically significant difference of 13% (OR 1.72, 95% CI 1.20–2.47) and 11.7% (OR 1.75, 95% CI 1.18–2.59) between the intervention and control group for patients who received appropriate haemoglobin and cholesterol testing, respectively [39]. Carlhed et al. reported statistically significant difference of 10.6–14.9% between the intervention and control group in four out of five Swedish national guideline-derived quality indicators of acute myocardial infarction [34]. None of the other studies found an effect of the intervention on all the outcome measures evaluated.

The weighted odds ratios of each primary outcome for each study and all studies combined for e-A&F are shown in Fig. 3, with substantial heterogeneity observed across studies (P _{heterogeneity} < 0.001, I ²  = 99.12%, 95% CI: 98.25–99.68). The weighted odds ratio of compliance with desired practice was 1.93 (95% CI: 1.36–2.73) when considering e-A&F to no A&F. Please note that due to the high variation as illustrated by I ² value, this average effect should not be considered reliable. We could not reduce the heterogeneity by considering subsets of outcome measures. The summary odds ratio of e-A&F comparing the intervention arm with access to e-A&F to control arm without the same access was highly unreliable due to high heterogeneity. Carney et al. was omitted from meta-analysis due to missing information in their report, i.e. they did not include the number of screening mammograms with a recommendation for immediate follow-up (a positive result) for both intervention and control arms of the study at intervals 1 and 2 [33, 37]. In Gude et al. [35], both arms of the study received e-A&F (but for different sets of outcomes, and so were each other’s control). Given the evidence of contamination effect that A&F might have on overall quality of care in general even if a subset of indicators are being tracked (Susan Gachau, et al., Effects of audit and feedback delivered within an emerging clinical network in Kenya on multiple indicators of the process of paediatric hospital care – a longitudinal observational study. BMJ Quality and Safety, submitted), and the admission in this study’s report of risk of contamination, it was omitted from the meta-analysis.

Further exploration of possible sources of heterogeneity as detailed in previous reviews [2] showed the number of theoretical domains targeted in the control arm, feedback characteristics (graphical feedback, A&F head-to-head comparison and real-time feedback frequency) and intervention duration to be the biggest explicators of the level of heterogeneity (Table 3). The components of the meta-regression reported in Table 3 were tested univariately, i.e. each separately from one another. A multivariable meta-regression model adjusting for effects of all components on intervention effect was not possible due to the small number of studies (n = 5) and outcomes (n = 14) included in the meta-analysis.

Only two studies explicitly reported using theory (adult learning theory and control theory) to inform intervention design and reported to have tested theoretical concepts with the trial [33, 35]. The coding for the domains targeted in the intervention and control groups for each of the studies is shown in Table 4 below. The reference table with the TDF domains and explanations that guided coding decisions is provided in Additional file 3.

Table 5 presents the number of times each of the domains were coded for both arms in each included study. For five studies, we identified at least six domains identified in the intervention arm [33‐35, 37, 39‐41]. The study informed by adult learning theory had the most domains identified in the intervention arm, but did not describe its control arm with the same rigour [33].

The most frequently coded domains in the intervention arm were ‘knowledge’, ‘motivation and goals’ and ‘social influences’ (all seven studies). The knowledge domain was also coded for the two studies that included a description of the intervention in the control arm [34, 39]. The most commonly coded domains when intervention and control arm of trials were combined were knowledge (coded ten times) and motivation and goals and social influences (both coded nine times). We did not identify any studies whose interventions targeted ‘social/professional role and identity’ or ‘emotion’.

Of the three studies that found a positive effect of the e-A&F intervention on the quality of care, one had the second highest number of coded domains in intervention arm [40, 41], and the other two were the only studies with domains coded in both intervention and the control arm [34, 39]. The low number of studies identified did not allow any inferences about patterns of theoretical domains identified and their link with effect sizes.

Our meta-analysis of five studies revealed the included electronic audit and feedback (e-A&F) interventions to be highly heterogeneous, even when subsets of outcome measures were considered. The weighted pooled odds ratio of compliance with desired practice was 1.93 (95% CI 1.36–2.73) when considering e-A&F to no A&F. This pooled average effect suffered from distortion as studies had varied sizes, differed in results and tended to be biased. Additionally, the meta-regression results would likely be biased given that they tend to have poor performance where there are few studies [43]. We therefore considered this average effect to be unreliable. Using the theoretical domains framework (TDF) to identify the theoretical concepts underlying the interventions, we found that the TDF domains of knowledge, motivation and goals and ‘social influences’ were most commonly targeted; professional identity and emotion were not targeted by any of the interventions. Due to the small number of studies identified, inferences about patterns of domains and their link with effect sizes could not be made.

To our knowledge, we are the first to perform a meta-analysis of the effectiveness of e-A&F interventions. Ivers et al. [2] identified 140 randomised controlled trials (RCTs) of A&F interventions that objectively measured provider performance in a healthcare setting. However, whereas they included feedback in any format, we focused on A&F interventions that were delivered electronically. Two studies included in our review [36, 39] were also identified by Ivers et al. Their meta-regression of studies featuring dichotomous outcomes included 82 comparisons from 49 studies. For dichotomous outcomes, Ivers et al. found the weighted median risk difference to be 4.3% increase in compliance with desired practice, unlike our results where we could not determine a reliable average effect due to high heterogeneity. This difference might partly be due to their exclusion of studies with a high risk of bias from analysis whereas we included all the studies with sufficient baseline information. Our exploration of sources of heterogeneity yielded similar findings to Ivers et al. on significance of instructions for improvement and feedback frequency for A&F. However, the inability to make firm conclusions from the analysis of heterogeneity due to the indirect nature of the comparisons is common in both studies. Our heterogeneity findings are further compounded by the small number of identified studies. We identified two more intervention characteristics (number of TDF elements on control arm and intervention duration) as being possible sources of heterogeneity in e-A&F interventions. The electronic and interactive component of feedback—which they did not evaluate—captures key aspects of feedback possibly associated with improved effectiveness. Specifically, e-A&F facilitates auto-delivery of feedback more frequently than other formats, including offering real-time updates; offers the ability to easily track measurable practice goals and adherence to specific action plans in real-time and is customisable. Additionally, e-A&F overcomes the pragmatic consideration of additional costs associated with providing personalised feedback more frequently, which plays into its added effectiveness.

Colquhoun et al. [15] examined the extent to which explicit theory was used in the 140 RCTs of A&F interventions identified in Ivers et al.’s review. In contrast to our study, they limited their approach to explicit use of theory and only included 14% of trials (n = 20), similar to what we found with only two out of seven studies explicitly stating that theory informed the design of their intervention [33, 35]. In contrast to Colquhoun et al.’s approach of classifying theories by application field, TDF represents common psychological aspects that most theories target. Our approach broadened the scope from explicit theory use while focusing on e-A&F. This was motivated by evidence showing e-A&F to influence contextual effect modifiers and intervention design [9, 21] but at the same time providing limited insight of how they can best be aligned to provide feedback supporting clinical practice, given their increasing popularity.

There are other examples of efforts to use TDF within systematic reviews such as Little et al. [22], which examined theoretical factors that post-fracture interventions aimed at patients at risk of osteoporosis, but did not include physician-directed A&F component. Similar to our study, they applied TDF retrospectively to explore implicit use of theoretical domains. They identified five commonly targeted domains out of the possible fourteen, with all studies targeting at least four out of the five domains identified. In line with Little et al., we found that all our studies targeted knowledge and social influences domains. While they found an inverse relationship for both number of times and number of different domains coded and the effect size, our analysis found the number of different domains coded in the control arm might be considerably associated with effect size, with the number of unique domains coded in intervention arm having no effect. However, this difference might be due to the risk of bias of studies included in review—although they did not report on risk of bias assessment. Also, the heterogeneous nature of studies in our review is possibly substantively higher than in Little’s review and might also account for the differences.

Knowledge, motivation and goals and social influences were the most frequently coded domains. In most studies included in our analysis, national guidelines determined the desired state of practice, rather than localised action-planning and goal setting. This is consistent with other studies where investigators concluded that clinical teams lacked key knowledge about practice needed to improve behaviour [35]. At the same time, goal setting recommendations propose clinicians not to be highly motivated to achieve evidence-based population-level quality targets, but instead tend to prioritise competing organisational and clinical goals [44].

Our findings related to common theoretical domains may be indicative of the belief that inclusion of local and national peer performance comparators offered a sense of importance and urgency of outcomes for teams to work towards. This reflects how feedback linked to team roles is part of a broader transformation of any clinical team, and an acknowledgement of the behavioural unit the team represents [13]. Hysong et al. argued that there is a need to understand how changes in the individual’s performance impact team outcomes, and if and how feedback practices are aligned to support teams [13]. Additionally, the studies possibly assumed that using peer ranking as social comparisons of practice behaviour would (1) instil a conscious desire among team members to maintain a certain degree of similarity in performance; and (2) help highlight a distinctive pattern of culture and practice behaviour shared by team members. One study illustrated how performance closer to benchmarks motivated change in practice [33]. This is consistent with evaluations of team practice behaviour which show improved perceptions of effectiveness and appreciable changes in practice performance where clinical teams have been regarded as a behavioural unit rather than individuals [13, 34]. Gauging the level of interdependence, enabling efficient care coordination and encouraging parity among all individual clinicians in the quality improvement endeavour require insight into change mechanisms involved in setting a shared quality agenda [45]. However, our results do not highlight how peer ranking as a social pressure encouraged goal setting as part of regular behaviour for team members.

With regards to differences in use of theory across studies, implicit targeting of ‘memory’, attention and ‘decision process’ and ‘behaviour regulation’ domains represents particularly intriguing findings. Within the identified e-A&F studies targeting memory, attention and decision process, those with explicit use of theory in intervention design found no significant differences between the study arms compared to studies without explicit theory use which reported significant differences. Behavioural regulation domain, which is a fundamental pillar of how and why feedback purportedly works, could not be confidently identified from two studies [36, 38]. Where the processes of goal selection, prioritisation and monitoring, coupled with action planning were not included in the feedback process, it is difficult to ascertain the active components that had a significant effect on outcomes [9]. This might signal a tendency to overestimate the impact of theoretical domains on outcome effect where active components of A&F are not well defined or targeted [2, 9]. It can also be indicative of how lack of adoption of a menu of theoretical domains in intervention design limits the ability to validate each domain within the context of e-A&F [46]. However, due to the small number of studies identified, it was difficult to theorise the relationship between the differences in theoretical domains targeted across studies and their impact on the effect size.

Professional identity and emotion were rarely coded, although we presumed emotion domain would not feature in e-A&F interventions given the lack of evidence within digital health on how it has previously been targeted. This was indicative of how clinical team practice behaviours might have been assumed not to be influenced by these factors. Yet, these domains are posited to influence clinical practice [47‐49]. Addressing this gap in future studies might further increase the understanding of effect of e-A&F interventions on practice.

Feedback reports delivered electronically have the potential to deliver adaptive feedback to individual team members [13, 21]. The possibility for individual clinicians to track personal goals while still aiming to conform to group performance targets implicitly imposes expectations for future designs of e-A&F that: (1) these interventions offer the ability to capture the intentions of team members at an individual level, and (2), they might be more informative if they cater for the evaluation of individual-team goal setting interaction. As such, future studies on e-A&F should aim to conduct head-to-head comparisons between individual versus team spanning: (i) goal attainment–where the feedback recipient has individual targets apart from the team’s, (ii) differences in frequency of updating target goals and nature of goals pursued and (iii) differences in memory, attention and decision process as delivered by e-A&F and in light of contextual effect modifiers.

The rationale for the interventions in our study and in some cases, how interventions were delivered was sometimes inadequately described. Descriptions of the control group specifically were often absent. This persistent problem in lack of descriptive clarity in A&F studies [50] makes it difficult to disentangle the active ingredients of the interventions from the delivery method [9]. This curtails the ability to identify the true underlying nature of observed (lack of) behaviour changes, and it constrains the studies’ replication in wider settings [2, 51, 52]. Future studies should therefore employ explicit use of theory in designing and evaluating A&F interventions as a clear effort to improve upon understanding of A&F mechanisms of action [9].

Additionally, testing of various theoretical concepts in a multi-component e-A&F interventions is now feasible through approaches such as AB testing[51]. Future e-A&F studies ought to consider stepwise research designs, which embed tuple-wise testing of theoretical domains within audit cycles. This would allow determination of separable direct additive effects of each domain on practice behaviour. Also, varying frequency, content and delivery of feedback would help inform future intervention designs [9].

The search strategy used to identify studies included a newly developed filter for identifying electronic interventions. As there is no consensus in definitions and terms used to describe e-A&F, we cannot be certain that we did not miss studies based on the search terms we used. However, the rigour of the approach used for developing the electronic filter, coupled with an A&F filter which has been used in a Cochrane review strengthened our search strategy [2]. We manually screened all included A&F trials in Ivers et al’s review to ensure that the search had picked up all e-A&F studies.

Due to small numbers, we included five studies in the meta-analysis regardless of their risk of bias. As the one study with a low risk of bias was also the one with the highest weight in the analysis, we deemed a sensitivity analysis to be non-informative. However, we also examined whether differences in the level of the unit of analysis (groups of professionals/individual professionals versus patients) was a source of heterogeneity, since analyses conducted at different levels can result in different effect estimates. Overall, in hindsight, there is an argument for not doing a meta-analysis at all given the high levels of heterogeneity and the small number of studies identified. We cannot make a conclusion that electronic feedback is better than any other type of feedback, e.g. written or verbal.