A pragmatic randomised controlled trial of the Welsh National Exercise Referral Scheme: protocol for trial and integrated economic and process evaluation

In Wales, in the early half of the last decade, most local health board (LHB) areas operated ERS, each following different protocols [6]. The development of high quality schemes was identified as a key action area in a number of Welsh Assembly Government documents [19, 20] and in 2006, existing good practice was assessed and standardised in Wales-wide protocols [21]. The National Exercise Referral Scheme (NERS) subsequently replaced local schemes and was rolled out in three phases from 2007. Supported by the Welsh Assembly Government working in partnership with Local Authorities, Public Health Wales (formerly the National Public Health Service) and LHBs, the NERS provided funds for a dedicated Exercise Co-ordinator (EC) and a number Exercise Professionals (EP) in each local health board area.

NERS consists of a series of motivational interviewing (MI) [22] consultations with an EP based in a community sports centre and access to a tailored, subsidised 16 week activity programme. To be eligible for NERS, participants must be sedentary (defined as not moderately active for 3 or more times per week or deconditioned through age or inactivity), and have at least one medical condition, covering CHD risk factors, mental health, musculoskeletal, respiratory/pulmonary and neurological conditions (see Table 1 for scheme inclusion criteria). The primary aim of NERS is for participants to achieve 30 minutes of moderate physical activity on at least 5 days per week. Common features of the scheme are detailed below.

Consultations occur at entry, 4 weeks (by phone) and 16 weeks. Following this, participants are contacted by telephone at 8 months to monitor progress and at 12 months they are invited to attend a review session. Routine programme monitoring systems are maintained by EPs and capture the dates of and records from initial, 4 and 16 week and 8 and 12 month consultations.

The Welsh Assembly Government commissioned an independent evaluation of the scheme as it was implemented in 13 of the 22 LHBs in Wales during phase one. This evaluation was to focus on the effectiveness of the scheme among two priority patient groups, those referred for mental health reasons (anxiety and depression) or for CHD risk factors This paper reports upon the design of the national evaluation, which utilised a randomised controlled trial design with nested process and economic evaluations, and reflects on issues in the development and implementation of the evaluation design for current challenges in policy evaluation [25, 26]. Ethical approval for the study was obtained from the Thames Valley Multi-centre Research Ethics Committee (Ref: 06/MRE12/85). Approval from medical directors within each LHB was also obtained.

The evaluation study comprised three key components. In order to evaluate the overall effectiveness of the intervention, a randomised controlled design was employed. Mixed methodology RCTs have the potential to provide unbiased estimates of the effectiveness of interventions as well as identifying contextual influences on intervention delivery [27]. A nested process evaluation therefore examined how the initiative was implemented, gained a more in-depth understanding of the views of providers and users, and facilitated interpretation of outcome effects. A nested economic evaluation was also undertaken in the form of a cost utility analysis which measured costs and benefits from a public sector multi-agency perspective, with results expressed as an incremental cost per quality adjusted life year (QALY) gained. In this way, the evaluation not only addresses the question, 'Does it work?', but also considers 'What works?', 'For whom?' 'Under what circumstances?' and 'At what cost?'[28]. A summary of the study design is presented in Figure 1.

Participants were recruited using opportunistic referral by a range of health professionals. NERS coordinators in each area undertook briefings with primary care teams to explain the trial and to provide referral forms and trial information materials. Health professionals who chose to participate in trial recruitment identified suitable sedentary patients who met scheme inclusion criteria (see Table 1) as part of their normal clinical practice. Patients were provided with basic trial information and a referral form was forwarded to the evaluation team, who assessed trial eligibility criteria (see Table 1). For patients referred for mental health and/or CHD reasons, NERS was only available to those who were willing to participate in the trial. Patients referred for reasons other than mental health and/or CHD were able to access the scheme without participating in the trial and their referral from were forwarded to ECs for processing.

Referred patients who were eligible for the trial were sent full informed consent materials and a brief baseline questionnaire for them to return by post. Recruitment If patients did not want to participate in the trial, they could gain access to the scheme by joining a 12 month waiting list.

Those consenting to participate in the trial were randomly assigned to an intervention (exercise referral scheme) or control trial arm using a random number generator, with gender and LHB as stratification variables. Randomisation of forwarded referral forms occurred every 2 weeks, with treatment allocation blind and remote from participants and practitioners. The control group received usual health care and a leaflet highlighting the benefits of exercise and including a website address listing local leisure facilities. Control participants were offered priority access to NERS at 12 months.

The research team were responsible for distributing, collecting and processing postal questionnaires at baseline, 6 months and 12 months. Respondents completed an initial informed consent form and subsequent mailings highlighted that they were free to withdraw from the study at any point. Questionnaires were anonymised, featured a unique ID and were accompanied by a freepost return envelope. Questionnaire non-responders were sent a repeat mailing two weeks after the first. For the primary outcome at 12 months, a specialist health research call centre team based at Cardiff University were employed to conduct telephone interviews using a standardised protocol [50] and blind to condition. Researchers again highlighted that respondents were free to withdraw from the study at any point. All telephone interviewers were trained and monitored to maximise standardisation of interviews. A strict protocol was adhered to in all cases, with each participant dialled on at least 15 occasions at different times and days of the week before being categorised as a non-respondent. Responders to 6 and 12 month questionnaires and 12 month telephone interviews were offered the opportunity to enter separate free prize draws for prizes of £200, £100 and £50 each time.

The trial sample size was determined to detect a difference in the primary outcome, total minutes of weekly exercise at 12 months. The planned sample size of 1052 participants in each group, has 90% power to detect an effect size of 0.15 with no loss to follow-up, and, more realistically, 84% and 87% power to detect an effect size of 0.15 if 25% and 20% respectively of randomised participants are lost to follow up.

The primary analysis of the primary outcome is a regression model with the stratification variables (gender/LHB area), age group (16-44, 45-59, 60+) and baseline activity level (GPPAQ) included as covariates. In the event that the primary outcome has a highly skewed or bimodal distribution, it will be recoded as a five level ordinal variable, and ordinal regression used in the primary and secondary analyses. These analyses will be conducted on an intention-to-treat basis, in which each participant is coded according to the treatment condition to which they were randomised. A secondary analysis of the primary outcome repeats the primary analysis, but excluding baseline activity level as a covariate. For the primary outcome only, the primary and secondary analyses will be conducted (i) for all participants completing the 7-DAY PAR and (ii) with the addition of multiply imputed values of the primary outcome for those who did not complete the 7-DAY PAR but who did complete either the Baecke or GPPAQ instruments at 12-month follow-up.

The primary analysis for the primary outcome, and for imputed values of 7-DAY PAR, is repeated to conduct sub-group analyses for each of the following variables: gender, age group (16-44, 45-59, 60+), referral reason (mental health only, CHD only, or combination of CHD and mental health), Welsh Index of Multiple Deprivation tertile, fidelity of programme implementation in LHB area (high/low). In each case, the statistical significance of sub-group effects is assessed by including an interaction term in separate models for each respective sub-group variable, with each model also including the main effect for the respective sub-group variable. An analysis to identify whether outcomes vary in terms of exposure to the programme replaces the binary intervention group variable with the three level programme attendance variable.

The analyses described in the preceding two paragraphs are also conducted for the secondary outcome. However, the primary analysis for the secondary outcome (HADS anxiety and depression scales) is conducted only among participants referred for mental health only or a combination of mental health and CHD reasons. The analyses of the secondary outcome among all participants is secondary to this analysis.

Other analyses to be conducted, but not associated with formal hypothesis testing, will include an analysis of baseline demographic characteristics and their relationship with outcomes and the relationship between 6-month intermediate variables and the primary and secondary outcomes. This analysis will require complex multivariable models and mediation analyses estimated within a multilevel modelling framework to allow for independence in measures within individuals (over time) and within exercise specialist.

A nested process evaluation examined programme theory, programme implementation and how NERS was received, facilitating an in-depth understanding of the views of providers and users, as well as interpretation of outcome effects [51]. Guided by the framework proposed by Steckler and Linnan [52], the process evaluation explored diffusion of the national policy across local contexts, fidelity and dose of implementation, patient experiences, programme reach and recruitment. The process evaluation adopted a pluralistic multi method approach involving triangulation [53] of multiple perspectives and methods (Table 2). All analysis will be conducted prior to any knowledge of outcome effects to guard against interpretation bias

In order to elicit programme theory and provide a framework for implementation checks, a programme logic model was developed through examination of scheme protocols and telephone and email contact with policy representatives. Quantitative assessments of programme implementation in terms of fidelity (i.e. consistency with programme theory) and dose delivered, draws on a combination of self reports from professionals and coordinator telephone interviews, as well as routine monitoring data and structured observation of tape recorded consultations. Exercise professionals were supplied with recording equipment and asked to provide recordings for at least one initial consultation, rated by two independent coders for fidelity to motivational interviewing principles using the Behaviour Change Counselling Index (BECCI); [54]. Programme uptake and reach are quantitatively assessed using routine monitoring data.

Experiences of diffusing the national policy into local contexts were explored through qualitative semi-structured telephone interviews with policy representatives and all local area coordinators. All exercise professionals were invited to participate in one to one telephone interviews exploring issues including experiences of implementing the scheme and perceived patient responses to the programme. In order to incorporate patients' perspectives on the programme, six case study centres were sampled purposively to represent a range of geographical areas and levels of deprivation and two members of the research team visited each centre. Patients attending the centre on the day of the visit were informed in advance of the researchers' attendance, and were invited to participate in group or one to one semi-structured interviews immediately after classes finished. In addition, GPs, practice nurses and practice managers referring patients to the scheme were interviewed, exploring influences on trial and scheme recruitment. Nine group interviews were conducted across areas, practice size and level of referral.

The nested economic evaluation [55‐57] took a public sector, multi agency perspective, spanning the Welsh Assembly Government (WAG), local government and the NHS. It costed the national exercise referral programme using WAG and local authority exercise programme budgets. In addition, telephone interviews with all principal leisure centre managers ascertained any additional expenditure incurred by local authorities in establishing and running NERS. Similarly telephone interviews ascertained the costs to WAG of establishing and coordinating the Scheme. The questionnaire data recorded trial participant primary and secondary care health service use, costed using national unit costs [58, 59]. With this, we will conduct a primary cost utility analysis to calculate the cost per QALY of NERS. We will undertake sensitivity analysis to explore what happens when we add in the cost of a specific GP consultation to facilitate referral onto NERS, and the effect of participants paying £1 or £2 per exercise session. Sensitivity analysis will be particularly important in the application of economic evaluation methods to a large pragmatic public health trial [60]. We will undertake subgroup analysis to investigate how age, gender, adherence and main reason for referral e.g. CHD risk or mental health problems, affects our estimates of the cost per QALY of NERS. We will use EQ-5D as our measure of utility to generate a cost per QALY and Cost Effectiveness Acceptability Curve (CEAC) for comparison with the NICE ceiling of £30,000 [61].

Trial recruitment occurred in 12 of the 13 areas, with one area failing to start a scheme during the trial period. Between July 2007 and October 2008, 4,779 health professional referrals were received by the evaluation team. Of these, 1,493 were not eligible for the trial and their referral forms were forwarded to the ECs for processing directly onto the scheme. The remaining 3,286 were sent full informed consent information and a baseline questionnaire. In total 2,160 patients were recruited to the trial, with 890 not responding and 236 not consenting to participate in the trial.

Respondents ranged between 16 and 88 years old, with a mean age of 52 (SD 14.7) and the vast majority classed themselves as white (96%). Whilst, measures of area deprivation ranged from 2.3 to 81.0 with a mean of 22.6 (SD 14.6). Table 3 shows that the majority were referred for CHD risk factors on their own (72%) or in combination with mental health conditions (24%), with only 4% referred solely for mental health reasons. Those recruited were most likely to be female (66%), to be married or with a partner (61%) and there was a fairly equal split between those employed (31%) and retired (32%). Finally, whilst the majority referred onto the scheme classified themselves as either inactive or moderately inactive (74%), 24% defined themselves as either active or moderately active.

The difficulties in facilitating rigorous evaluations of policy interventions is well recognised, particularly for ERS [25, 26]. This evaluation included a substantial pre trial development phase. Early engagement between evaluators, the national policy co-coordinator (EM) and local implementers at the intervention development and implementation stage meant that a rigorous design could be negotiated that met both scientific and practical considerations. Regular meetings with policy makers and practitioners also ensured that the research design was understood and maintained. The involvement of government social researchers (JH, CR) who promote RCTs of policy evaluations with ministers and support policy colleagues when they are implemented was key to the success of this process. Such an approach is enshrined in the work of the Public Health Improvement Research Network (PHIRN) in Wales which develops and supports multidisciplinary research groups of academics, policy makers and practitioners to identify and develop opportunities for rigorous policy evaluations.

Collaboration with national policy leads and local practitioners was also key in developing and implementing programme monitoring systems. The provision of a national, rather than a number of local schemes, offered the opportunity to introduce a standardised national monitoring system. This included the provision of a common database and protocols for data collection and assessment. High quality programme monitoring data can provide important information, such as referral and completion rates and useful data on programme fidelity. The evaluation will use this information to assess the impact of adherence and fidelity on effectiveness Robust programme monitoring systems also provide the opportunity for long term self evaluation once trials such as this are completed.

In Wales, the NERS was introduced within a RCT design in the context of the termination of pre-existing local schemes. This was particularly challenging, as a significant barrier to facilitating rigorous designs has been professional reluctance to refer into trials when there is a perception of change or withdrawal of service, regardless of the evidence base for the effectiveness of that service [25]. Establishing effective recruitment strategies was further complicated by a reliance on primary care clinicians as the main source of referral into the scheme. It has been argued that such clinicians are well placed to promote increased physical activity [62, 63]. However, previous studies have found that physical activity promotion was not a priority during routine consultations [42, 64], that there is a lack of consensus in how clinicians perceive their role in changing patients' behaviour and that patients are frequently referred in an unsystematic way [42, 65].

Despite this, the trial achieved referrals from 12 of the 13 implementation areas (with one area failing to start the scheme) and a high referral rate from clinicians. This may be in part a function of the size of the scheme being evaluated, but was also facilitated by a number of strategies employed in the conduct of this study. This included briefing sessions for professional stakeholders by the evaluation and national policy team at an early stage to address concerns and promote the evaluation. At the local level, the provision of Exercise Coordinators who were able to promote the trial with clinicians also facilitated professional participation. Particularly important was the fact that trial recruitment was undertaken by the evaluation team after clinicians had provided basic information to patients. This placed a low research burden on health professionals who promoted but did not recruit to the trial.

Examining characteristics of those recruited to the trial highlights the greatest proportion were referred for CHD risk, that there were a larger number of females and a significant minority who classified themselves as active or moderately active. Although the scheme was targeted at sedentary individuals, this latter finding reflects the pragmatic nature of the trial which placed minimal control on the implementation of the scheme other than randomisation. Results are therefore likely to be highly reproducible and have high external validity. This is particularly important given previous criticisms of RCTs of ERS that what is gained in internal validity is often at the expense of external validity [66]. Influences on NERS referral processes and the acceptability of trial recruitment strategies will be assessed within the nested process evaluation which will explore influences on clinician referral such as lack of time during consultations; knowledge about ERS, views on evaluation; and medico- legal responsibility [42, 67‐69].

Good quality evaluative research not only quantifies outcomes, but also helps us to understand how an intervention produces the outcomes that it does [70]. This is challenging for the evaluation of ERS which are typically heterogeneous and driven by differing latent programme theories, which can only realistically be compared with other similarly conceived schemes [71]. However, no previous evaluations have clearly articulated ERS programme theories and attempted to establish congruence between the conceived and the delivered intervention. In this study, comprehensive nested process and economic evaluations will attempt to address these shortcomings. An intervention logic model will be developed through discussions with implementers which will identify key proposed programme inputs and the anticipated intermediate outcomes associated with these inputs. This will provide the basis for a comprehensive assessment of intervention implementation and will inform attempts to map causal pathways.

Programme implementation will be assessed initially by case studies of community sports centres, purposively sampled to reflect a range of area level socioeconomic deprivation and geographic regions. These will examine intervention delivery across contexts [27] and explore participant experiences of NERS [72‐74]. Data will also inform the development of structured interview schedules for national and local implementers to assess aspects of implementation such as diffusion, reach and fidelity. Aspects of fidelity will also be assessed via routine monitoring data records and systematic checks of treatment integrity [75]. This follows recommends that the delivery of motivational interviewing should be assessed through coding of tape recorded consultations, rather than relying upon self reports of the delivering party [76]. Finally, complex interventions also need to be understood as leading to outcomes through the activation of mechanisms of change, with the activation of these mechanisms varying across contexts and between subgroups [77]. This study has therefore includes a number of variables assessed at 6 months that will be used to model causal processes amongst a number of sub groups. Taken together with the nested economic evaluation these elements provide the opportunity not only to judge the effectiveness of the NERS, but to understand why it may be effective, for whom, under what circumstances and at what cost [28].