Implementing evidence-based recommended practices for the management of patients with mild traumatic brain injuries in Australian emergency care departments: study protocol for a cluster randomised controlled trial

The aim of this trial, which is part of the Neurotrauma Evidence Translation (NET) programme [48], is to test the effectiveness of a targeted, theory- and evidence-informed implementation intervention to increase the uptake of three key clinical EBCPG recommendations regarding the management of adult patients (18 years of age or older) who present to Australian EDs with mild head injuries, compared with passive dissemination of the EBCPG.

More specifically, the primary objective is to establish whether the intervention is effective in increasing the percentage of patients for which a prospective measure of post-traumatic amnesia (PTA) using a validated tool is performed in the ED until a perfect score is achieved or the patient is transferred or admitted.

Secondary objectives include establishing whether the intervention is effective in: increasing the percentage of patients for which two other assessment methods of PTA were performed (assessment without using tool, and whether the administration of the validated tool was completed at least once); increasing the percentage of patients for which CT scanning is appropriately performed; increasing the percentage of patients who receive patient information upon discharge home from the ED; changing factors that are thought to mediate the effect of the intervention (such as increasing the staff members’ knowledge, or decreasing negative beliefs about consequences); and improving selected patient clinical outcomes and quality of life.

In addition, our study aims to: quantify the trade-off between the hypothesised improvement in patient management and patient health outcomes, and the additional costs (savings) arising from delivery of the implementation intervention and from any subsequent changes in clinical practice and healthcare utilisation by conducting a cost-effectiveness analysis; and evaluate intervention fidelity, assess acceptability of the intervention, and assess perceptions around success of the implementation processes and influencing factors, by conducting a process evaluation.

EDs will be allocated to the intervention and control groups using the method of minimisation proposed by Pocock and Simon [59]. Minimisation has been shown through statistical simulation studies to outperform stratified allocation methods for maintaining baseline balance in prognostic factors [60]. In addition, the method allows for balancing across a greater number of prognostic factors. The minimisation factors will include: the existence of a protocol for mTBI (consistent with our recommendation for appropriate PTA assessment); size (annual presentation rate 2012), included as a proxy for other variables that are harder to measure such as staffing; rurality; and type of participation (NET or NET-Plus). Minimisation is essentially a deterministic method [60], and therefore use of this method can increase the risk of selection bias [61]. We plan to reduce the risk of selection bias through inclusion of a random component and allocation of batches of EDs by a statistician independent of the trial. The statistician will be provided with a file containing only ED codes and the minimisation factors (thus no identifying information). For each batch of EDs, the statistician will randomise the order of EDs before using the minimisation algorithm to allocate the EDs to intervention and control groups.

Due to the nature of the intervention, it will not be possible to blind ED staff members to group allocation. To limit the possibility of selection and detection bias, chart auditors will be independent, where possible, of the hospital. In instances where the hospital does not approve an independent chart auditor, a solution will be discussed which aims to reduce the risk of selectively identifying patients (for example, using a staff member from another unit who will receive training from the research team). External chart auditors and the trial statistician will be blinded to group allocation.

Tables 3 shows our clinical practice outcomes and data collection methods.

Chart auditors will be trained to identify eligible patients in the medical record system, and to log data into the secure web-based system in participating hospitals. Auditors will assess the same pilot records, and training will continue until sufficient consistency is achieved. Auditors will receive a data collection manual with instructions (such as definitions and data dictionary). The web-based database will be designed to minimise errors (for example, inability to leave particular fields open, or warning when answer is outside expected range). Double data entry will be used for all paper-based questionnaires and returned questionnaires will be checked for consistent errors or missing data.

The primary outcome in the NET trial is appropriate PTA screening. In the calculation of sample size for this outcome, adjustment needs to be made for the clustered nature of the design. The variance inflation factor, or design effect, used to achieve this is a function of the average cluster size, variation in cluster size, and the intra-cluster correlation (ICC) [79].

The assumed ICC was calculated from two sources reporting ICCs from various datasets [80, 81]. We included only ICCs for process measures in secondary care settings since empirical research has demonstrated that ICCs tend to be larger for 1) process measures compared with patient outcomes (median 0.063 versus 0.030, respectively), and 2) secondary care compared with primary care (median 0.061 versus 0.045) [63]. Thirty three ICC estimates for process measures in secondary care were available, and the median ICC was 0.18. The variation in cluster size was incorporated into the sample size calculation through an estimate of the coefficient of variation, defined as the ratio of the standard deviation of the cluster sizes to the mean cluster size [79]. The estimated coefficient of variation of 0.47 was calculated from annual attendance at EDs across 170 hospitals in Australia in 2009.

There were limited studies providing estimates of rates of compliance for our primary outcome [82], and no studies undertaken in adult populations. We therefore undertook a retrospective audit of two Australian metropolitan EDs. The estimated rates of appropriate PTA screening for patients presenting in these hospitals (with an initial GCS of 14 or 15) were 0% (95% CI 0% to 14%; n = 24) in one hospital, and 31% (95% CI 24% to 39%; n = 164) for a second (which had an mTBI protocol in place) (Bosch M, McKenzie J, unpublished observations). We expected that hospitals without a protocol for mTBI would be more interested in participating in the trial, and therefore calculated our sample size based on a control group rate of 10% (that is, closer to the estimated rate in the hospital with no protocol in place). We wish to detect an absolute increase in the rate of appropriate PTA screening of at least 20%. Our rationale for selecting a difference of 20% is based on considering the size of effect required to justify a resource intensive intervention such as NET, and estimates of the effects of local opinion leaders on professional practice observed across a range of studies [83].

To detect an absolute increase of 20% in the rate of appropriate PTA screening (equivalent to an odds ratio of 3.9, log odds 1.3) (assuming a control group rate of 10%, an ICC of 0.18, coefficient of variation = 0.47, an average of 30 patient participants per ED, and a 5% significance level) with 80% power, we will require 15 EDs per intervention group. A total of 30 EDs will provide 900 patient participants for whom ED staff management will be assessed. For 900 patient participants, the width of the 95% confidence interval for the observed difference in appropriate PTA screening rates between groups will be approximately ±14% (equivalent to a width of ±0.51 on the log odds scale). Allowing for 10% attrition in EDs, we plan to initially recruit 17 EDs per intervention group.

Power is dependent on the selected parameters, but in cluster trials is primarily determined by the chosen ICC and control group rate. Assuming all other parameters are as defined above, a lower control group rate (for example, 5%) will result in greater power (87%), while a higher control group rate (for example, 30%) would result in less power (60%). A smaller ICC (for example, 0.10) would result in greater power (94%), while a larger ICC (for example, 0.30) would result in less power (59%).

Sample size calculations were undertaken using the module clustersampsi[84] implemented in the statistical package Stata (StataCorp LP, USA) [85].

The effectiveness of the intervention on management outcomes, predictors of management outcomes, and patient outcomes will be estimated with marginal modelling using generalised estimating equations. These models will appropriately account for correlation of responses of individuals within EDs. We plan to fit an exchangeable correlation structure where responses from the same ED are assumed to be equally correlated [86]. We will use robust variance estimation which yields valid standard errors even if the within-cluster correlation has been incorrectly specified [87, 88]. For binary outcomes, a logit link will be used.

Models will include adjustment for minimisation factors (size (annual presentation rate in 2012), existence of an mTBI protocol, NET or NET-Plus, and rurality) and pre-specified potential confounding variables (Figure 3). The potential confounding variables have been selected through discussion amongst the investigators and from published research. All pre-specified confounders will be included in the models even when no baseline imbalance exists. In addition, for continuous outcomes which are collected at both baseline and follow-up (for example, clinicians’ self-reported cognitions, social and environmental influences in relation to the key behaviours, cognitions about their organisation and team climate) we will include the baseline measure of the outcome in the model. Adjustment for the baseline measure of a continuous outcome yields unbiased estimates of intervention effect in circumstances where there is baseline imbalance, and has the benefit of providing the most powerful analysis [89]. Our primary effectiveness analysis will be the model (as described above) that estimates the intervention effect on the primary outcome, appropriate PTA screening.

Estimates of intervention effect from these models with binary outcomes will yield odds ratios. To aid interpretability, we plan to also provide estimates of risk differences [90]. Estimates of risk differences will be computed from marginal probabilities estimated from the fitted logistic models [91].

Missing data for the clinical practice outcomes are likely to be minimal since eligible patients will be identified and data extracted through chart audit. Missing data could arise if EDs choose to withdraw post-randomisation but prior to data being extracted from clinical records, resulting in empty clusters. Accounting for empty clusters requires strong assumptions to be made about patient characteristics and outcomes based on ED staff or ED characteristics. We will attempt to examine the potential impact of empty clusters on the intervention effects for the clinical practice outcomes using weights to allow for patterns of ‘missingness’ [92]. For continuous outcomes, generalised estimating equations yield unbiased estimates of intervention effect when 1) data are missing completely at random, or 2) when the correlation structure is correctly specified and known covariates that are associated with the missing data mechanism are included in the model (missing at random) [93, 94]. For the continuous proxy measures of clinical practice and hypothesised mediators, we plan to identify potential predictors of missing data through modelling (for example, [95]), and include these predictors in the primary analysis model. We will investigate methods to impute missing outcome data collected at baseline (for example, baseline behavioural constructs) [96].

All estimates of intervention effect will be reported with 95% confidence intervals. No adjustment will be made for multiple testing. All tests will be two-sided and carried out at the 5% level of significance. A full statistical analysis plan will be developed and written prior to undertaking any data analysis.

Potential confounders at patient level are collected via chart audit. Potential confounders at the level of staff members, as well as organisational history of change [97], organisational learning [98], leadership [99], staff culture [99] and team tenure [100] will be collected via self-report surveys. ED structural characteristics will be measured via a telephone checklist completed by the ED Director (or delegate).

Our theory-based evaluation of factors along the proposed causal pathway (see Table 4 and Figure 2) will be complemented by other components that form part of our process evaluation [101, 102]; these are: 1) an assessment of intervention fidelity (whether the intervention was successfully and consistently delivered as planned (delivery), and whether it reached the target group (receipt)) [103, 104]; 2) perceptions of success of delivery and receipt [105], and factors that contributed to this [103]; and 3) perceptions of acceptability and feasibility of the roll-out of the intervention [106].

Table 6 summarises the main elements included in the evaluation, methods planned, and indicative measures. A more detailed description of the fidelity assessment will be provided in a future publication.

We plan to conduct semi-structured interviews with ED Directors, local opinion leaders and staff members in eight intervention hospitals. We will sample for maximum variation (for example, size and location of EDs). The interview guide and analyses will be guided by a theory-based framework that was used in phase two of this project (the interview phase). Analyses and interviewing will be done concurrently to inform emerging themes. If data saturation has not been reached after conducting interviews in eight hospitals, we will add sites and continue sampling until data saturation [107].

In addition, a brief checklist-guided structured interview will be conducted by telephone with ED Directors or delegates to assess sustainability of the changes in intervention hospitals (for example, plans to integrate a validated tool into charts).

Pre-recruitment ethical approval of this trial protocol has been granted by Alfred Health Human Research Ethics Committee (approval Number 398/12). When participating hospitals have been identified, local ethics and research governance procedures will be completed. This will include a “letter of understanding”, which contains information around what the trial entails, what the expectations are from hospitals and researchers, and the process to authorise the independent chart auditor to undertake the data extraction from medical records. The document will be signed following recruitment, and prior to randomisation. It will still be possible for EDs to withdraw from the study at any time should they wish to do so.

Consent procedures will be in line with recent guidance regarding ethical issues in CRTs [54, 108], and include the following.

Confidentiality of data will be ensured via the following processes. All information collected will be entered via a web-interface into a secure server database. Each user will have a unique username and will be assigned a role that will provide them with access to only the information they need to see. The database administrator will have access to the full data, and will sign a privacy statement acknowledging responsibilities and restrictions on handling identifiable information according to ethics guidelines.

A number of previous studies have estimated the costs and benefits associated with adherence to various diagnostic management strategies for mTBI [110‐112]. Stein and colleagues [112] conducted a modelled cost-utility analysis comparing selective CT scanning based on the CCTHR, CT for all patients, skull radiography for all patients, prolonged ED observation, 24-hour hospital admission, and no treatment. Smits and colleagues [111] conducted a modelled cost-utility analysis based on data from the Computed Tomography in Patients with Minor Head Injury (CHIP) trial [26] (comparing selective CT as per the New Orleans Criteria, CCTHR, and CT in Head Injury Patients (CHIP) criteria) and CT for no patients. Holmes and colleagues [110] conducted a modelled cost-utility analysis comparing selective CT scanning (as per National Institute for Health and Care Excellence (NICE) guidelines, the CCTHR and various other guidelines), CT for all patients, and discharge for all patients without testing. Findings from each of these studies suggest that – at the mean – selective CT scanning is cost-effective in comparison to other strategies. However, in interpreting these findings, Holmes and colleagues [110] noted that “… the cost of CT scanning is very small compared to the estimated cost of caring for patients with brain injury worsened by delayed treatment” (page 1,423) and concluded that “… all hospitals receiving patients with minor head injury should have unrestricted access to CT scanning for use in conjunction with evidence based guidelines” (page 1,423). Comparisons between alternative criteria for selective CT suggested that they offer “broadly similar costs and quality-adjusted life years” [110] (page 1,428), though adherence to the medium risk CCTHR had the highest probability of being the most cost-effective strategy in the study conducted by Holmes and colleagues [110]. Results reported by Stein and colleagues [112] and Smits and colleagues [111] are broadly consistent with these findings.

The costs and benefits associated with adherence to various diagnostic management strategies are clearly relevant to the question: what is the cost-effectiveness of an implementation intervention to increase adherence to key recommendations regarding diagnostic management? However, addressing this question entails a number of complications that were not addressed in the studies by Stein and colleagues [112], Smits and colleagues [111] and Holmes and colleagues [110]. First, the potential for cost-effective implementation is predicated on the existence of an evidence-practice gap (see Table 1). Previously published studies by Stein and colleagues [112], Smits and colleagues [111] and Holmes and colleagues [110] set aside the difficulties associated with closing the gap between evidence-based recommendations and clinical practice. Statements of cost-effectiveness taken from these previous studies are therefore conditional upon the assumption of perfect adherence to each of the evaluated diagnostic management strategies [27]. Second, implementation typically entails an additional investment that may or may not be offset by any health gains and/or reductions in health service utilisation derived from increased adherence to evidence-based recommendations. The economic evaluation described here will be the first to consider the trade-off between the hypothesised increase in adherence due to implementation of key clinical EBCPG recommendations regarding the ED management of adult patients with mTBI, and the additional costs (savings) arising from the implementation intervention.

Additional methods for the economic evaluation including methods for the identification, measurement and valuation of outcomes and resource use are described in Additional file 2. Results from the economic evaluation alongside NET will be expressed as additional costs (savings) per patient appropriately screened for PTA, per patient who received patient information upon discharge home, and per patient safely discharged. Results from the economic evaluation alongside NET-Plus will be expressed as additional costs (savings) per point difference on anxiety questions of Hospital Anxiety and Depression Scale at 3 to 5 months post-discharge, additional costs (savings) per point difference on the Rivermead Post-concussive symptoms checklist, and additional costs (savings) per point difference in SF6D utility index scores.

The results from the trial will be published regardless of the outcome. Reporting of this trial will adhere to the relevant, and most up-to-date, CONSORT (Consolidated Standards of Reporting Trials) statement [90] and its relevant extensions [114‐116].