CONSORT to community: translation of an RCT to a large-scale community intervention and learnings from evaluation of the upscaled program

Implementation science is an emerging area of research that studies the translation of evidenced-based interventions into routine practice and is particularly applicable to healthy lifestyle public health initiatives. The ideal translation continuum begins with an evidence-based randomised controlled trial (RCT) to establish efficacy, followed by community trials to determine clinical effectiveness (i.e. does the intervention work in the real world setting) and finally, adoption and routine delivery by health care delivery systems [1]. The translation of an intervention into the community setting is an iterative process requiring adaptation of the intervention itself, adoption of the program by the community, and implementation often under quite different conditions to the original RCT. Similarly, the evaluation of the intervention evolves with the move from RCT to a program delivered by health providers rather than researchers. Evaluation of an RCT aims to demonstrate efficacy, with little emphasis on process evaluation which probes how the program is received, understood and utilised by participants. Evaluation priorities of community programs include program reach, uptake, fidelity and sustainability to ascertain the feasibility of incorporating the program into routine health service delivery which account for broader contextual social, political and economic factors associated with successful program implementation [2].

While there is a growing body of literature on changes to the implementation of healthy eating programs across the translation continuum, there is little dissemination of the corresponding changes in evaluation, or reflections on evaluation challenges once upscaled. The aim of this paper is to describe the translation of the evaluation framework from a randomised controlled trial (RCT) to a large-scale community intervention using the PEACH™ Program as a case study. This paper also aims to provide the authors’ experiential reflections on their lived tension between RCT and implementation research paradigms, and provide examples along with key learnings which are beneficial for future evaluation of upscaled programs to assist the implementation of programs along the translation pathway.

The evaluation framework of PEACH™ QLD was designed to meet the needs of the funder and as much as possible reflected its evolution from the RCT and community trial settings, comprising a range of outcome, impact and process evaluation indicators. It also included a range of broader environmental- and systems-based measures reflecting the Team’s experience in evaluating OPAL, South Australia’s settings-based community-wide childhood obesity prevention initiative [12]. Despite this broader evaluation design, there were some tensions between the evaluation framework and implementation/program delivery model which were not anticipated. This required a certain degree of reactivity to be introduced to the evaluation which in some cases presented significant practical/operational obstacles. For example, while community lifestyle programs need to be dynamic and have the flexibility to adapt to the needs of its participants during implementation, RCT intervention protocols are more strictly adhered to and do not normally change. In responding to participant needs and to improve participant engagement, there were iterative changes to program delivery including organisational settings, session scheduling, recruitment, and the order of session delivery. Some of these changes required modifications to the evaluation tools and consequent changes to the coding of variables collected and the syntax used for analyses. Furthermore, the governance under which the evaluation team operated required these changes to be submitted to multiple ethics committees for variations. Such changes were largely unanticipated, although in retrospect were necessary and appropriate actions to meet the needs of participants, service providers and the funding body. Achieving a balance between an evaluation framework that was flexible and adaptable enough to respond to change yet robust enough to maintain integrity was challenging. Additional learnings from key evaluation challenges encountered and actions undertaken during PEACH™ QLD are described in Table 3.

While external evaluation has several strengths for research rigour and impartial expert assessment, it presented some challenges in the present study, particularly as the implementation and evaluation teams were two organisationally and geographically separate teams based in Brisbane, Queensland and Adelaide, South Australia, respectively. The main drivers for the Brisbane-based Project Implementation Team were ensuring a smooth running program and satisfying the tender milestone requirements, principally child enrolment numbers which were tied to financial payments. In contrast, the main driver for the Evaluation Team was the amount and quality of data collected with little direct connection to program facilitators.

Overall, attendance was a concern and the catalyst for modifications to program delivery, in particular a change from fortnightly, to weekly delivery of sessions 1–9. It was a challenge for the Evaluation Team to provide process evaluation data, in particular participant concerns, program satisfaction and acceptability in real-time to support this change as these data are only captured at the end of the 6-month program in the follow-up questionnaire. In response to this, additional qualitative research efforts to explore factors associated with enrolment [16] and engagement to better understand program access, need and attendance were driven by the Project Implementation Team. While not considered formal evaluation data, other information captured by the Project Implementation Team during administrative phone calls at enquiry and drop out times provided valuable insights into engagement with the program. Future programs should recognise and establish mechanisms for systematically capturing these data which informed program changes and sustainability of the program. Both the Project Implementation Team and the Evaluation teams worked hard to maintain a cohesive approach by having regular meetings and communicating well. Nevertheless, our experience from PEACH™ QLD has led us to question whether evaluation is best performed by two groups: process evaluation by the Project Implementation Team and outcome evaluation by an external evaluator. Implementation staff require data in a timely manner to facilitate program service delivery, however the present study consisted only of pre-post evaluation and the evaluation framework did not permit data collection or reporting in real time. During planning, teams need to work together to ensure clinical, service and system factors are adequately captured by the evaluation framework. During delivery, implementation staff must review process data sources and collection tools to ensure sufficient data are captured in order to address delivery challenges.

Implementation research is a relatively new and rapidly emerging field and theoretical approaches and frameworks underpinning implementation science have become more common over the past decade. There are now a number of conceptual frameworks for implementation including REAIM [11], Proctor [17, 18] and Consolidated Framework for Implementation Research [19]. While there is overlap between these frameworks, some elements or domains may be of greater importance in one implementation project compared to another. In practice, these models can be tailored to suit a specific implementation research program. In the PEACH™ QLD project, we largely carried forward the evaluation framework from the RCT and retrofitted this to the REAIM framework which we felt to be the best fit in 2012. While the Proctor and CFIR models were both published in 2009, there was at that time little exemplary literature regarding their use. On reflection, choosing the reporting framework a priori may have prompted us to capture additional data e.g. formal data on those who enquired about the program but did not enrol or attend, in order to describe ‘Reach’ in terms of those who do not participate as well as those who do. Ultimately, the scope of evaluation and project deliverables were negotiated with the funder, and the budget for evaluation was based on these deliverables. The funder of the program hence approved the evaluation framework based on the measurement and reporting of program outcomes to demonstrate effectiveness without concern for selection of a conceptual framework in which measures of effectiveness were embedded.

In order to present a case for a program to be implemented long-term or become adopted as part of routine practice, it is critical that there is solid evidence, which requires collection of comprehensive and valid data. Collection of these data requires adequate funding for health service surveillance or specific program-targeted funding, as appropriate. Evaluation of upscaled programs in the real world is challenging as unlike gold standard efficacy studies in clinical or research environments, behavioural interventions and evaluation can evolve over time to meet participant need and fit within changing public health systems and settings. There may be some loss of control when upscaling potentially affecting participant engagement, intervention fidelity and hence evaluation rigour. Ultimately, there can be many trade-offs in translation research and researchers who find themselves working in this space need to be prepared to settle at times for something less than ideal.

Understanding practicalities in data collection and management may help others to design their data collection tools and processes to optimise data quality and ease and efficiency of data collection. Our learnings from the evaluation of PEACH™ as a service delivery program in Queensland provide new knowledge which can assist to inform the evaluation of future evidence-based, family-focussed interventions for children or other upscaled programs.