Systematic evaluation of implementation fidelity of complex interventions in health and social care

Carroll et al. [9] proposed a framework for evaluation of implementation fidelity, which currently is the most complete conceptual framework for implementation fidelity. The framework includes components of implementation fidelity and factors that may influence the degree of fidelity, referred to as moderating factors. The measurement of implementation fidelity is a measurement of adherence, with its subcategories -- content, frequency, duration, and coverage (dose). Thus, adherence relates to the content and dose of the intervention, i.e., whether the active ingredients of the intervention have been received by the participants as often and for as long as was planned. Intervention complexity, facilitation strategies, quality of delivery, and participant responsiveness were included in the framework as moderating factors. Intervention complexity has been found to influence the implementation fidelity, i.e., complex interventions were more difficult to implement with high fidelity than simple interventions [14]. Complexity refers to both description of the intervention and the real nature of the intervention. Interventions described in detail are more likely expected to be implemented with high fidelity than ones with vague descriptions. Facilitation strategies, such as provision of manuals, guidelines, training, and feedback, may be used both to optimize and to standardize implementation fidelity. However, more facilitation strategies do not necessarily mean better implementation. Instead facilitation might be highly dependent on the complexity of the intervention. Quality of delivery concerns the appropriateness of the delivery process for achieving what was intended. Dusenbury et al. [12] defined quality of delivery as 'the extent to which a provider approaches a theoretical ideal in terms of delivering program content.' Participant responsiveness refers both to individuals receiving the intervention and individuals responsible for delivering it. Higher levels of implementation fidelity are assumed to be achieved if those responsible for delivering an intervention are enthusiastic about it. Similarly, the uptake of the intervention depends on the responsiveness of those receiving it. The authors of the framework suggest that the moderators have complex relationships to each other and to the implementation fidelity. For example, facilitation strategies may improve quality of delivery, which in turn may influence participants' commitment to the intervention. However, the potential moderating effect of intervention complexity makes that impact more complicated. The authors suggest that there may be interaction effects between moderators, i.e., when the effect of one factor is dependent on the level of another. In summary, the framework suggested that different moderating factors might affect, positively or negatively, the implementation process and its fidelity. These factors interact with each other and the effect of one factor on fidelity might be influenced by another moderating factor. The framework suggested that all these factors should be evaluated systematically when conducting a process evaluation. In this project, two additional moderating factors were included in the framework, namely context and recruitment. The modified framework is presented in Figure 1. Importance of context for program implementation has been highlighted by several other authors [3, 15‐17]. Lipsey [3] emphasized the importance of taking into account surrounding social systems, such as structures and cultures of organizations and groups, inter-organizational linkages, and historical as well as concurrent events, when assessing program implementation. Pettigrew and Whipp's [15] model of strategic change management defined context together with content and process as main determinants of change. Recruitment refers to procedures that were used to attract potential program participants. Baranowski and Stables [18] argued that recruitment was a key process evaluation component. Some of the aspects to be evaluated were suggested to be reasons for nonparticipation among potential participants, subgroups that were less likely to participate, and consistency of recruitment procedures among potential participants. Steckler et al. [13] argued that an evaluation of recruitment can contribute to correct generalization of findings, i.e., not generalization results for subgroups that have chosen not to participate. In this study, the modified framework will be used as a conceptual model to structure the data collection and analyses to identify mechanism and factors that might influence the implementation of complex interventions.

The multiple case study method will be used to investigate the implementation processes of three intervention studies. Thus, a case is defined as an intervention study. Each case will be investigated in depth and longitudinally, using both quantitative and qualitative methods. The case study method has been proposed to be a suitable method for conducting longitudinal research of change processes [17, 19].

All three interventions are complex in nature, including several active ingredients. In addition, the interventions are conducted in complex health or social care environments where several professional groups or care provider organizations cooperate. The three interventions are briefly described below.

The intervention involves development, implementation, and evaluation of an integrated care chain for frail elderly people. The care chain will cover the older person's contacts with community care providers (home help services, home nursing, and rehabilitation), primary care, the hospital emergency department and hospital ward. A case manager and a multi-professional team will help the elderly people and their relatives to coordinate care contacts. The theoretical framework for the study is that integrated care with a case manager creates networks of resources and services over time and between different healthcare providers, particularly between health and social care. This is expected to improve health-related quality of life, increase satisfaction with care and rehabilitation, decrease older persons' emergency care consumption, and influence cost efficiency at the community level. The study design is a randomized controlled study with a total of 200 participants divided into intervention and control group. The study is conducted in the city of Mölndal in western Sweden. The possible effects of the intervention on participants' capability to perform activities, health-related quality of life, satisfaction with care, and emergency care consumption will be evaluated at three, six, twelve and twenty-four months after the baseline measurement. The responsible researchers are Synneve Dahlin-Ivanoff, PhD (Medicine), Professor and Katarina Wilhelmson, PhD (Medicine) at the Sahlgrenska Academy at Göteborg University.

The intervention involves development, implementation, and evaluation of a support program in palliative care for nursing staff and workplace leaders working in community nursing facilities for older people. The support program is based on a study circle model that combines participants' reflections and discussions based on their expertise and work experiences. The model includes having participants learn to question their work practices and develop new ways of working and solving problems at work. The program consists of regular meetings for each professional group and cross-professional workshops. Reading materials regarding palliative care and other related relevant subjects will be provided prior to a study circle. An external facilitator will lead the circles. Participants are expected to get direct support in terms of improved knowledge as well as indirect support in terms of a collective platform for reflections, discussion, stimulation, and concrete improvement work. The theoretical framework for the study is that support in terms of expertise development and coaching positively affects staff attitudes towards care recipients, their work satisfaction, and general wellbeing. The intervention is conducted in a quasi-experimental design in six nursing homes in the city of Malmö in southern Sweden. The possible effects of the intervention on staff satisfaction, work climate, and general well-being are evaluated at six and eighteen months after the baseline measurement. The responsible researcher is Anna-Karin Edberg, PhD (Medicine), Professor, Lund University.

The study involves implementation and evaluation of an evidence-based method, supported employment (SE), for vocational rehabilitation for people with mental illnesses. The purpose of SE, according to the Individual Placement and Support model, is to help people with severe mental illness find and keep employment in a competitive market. SE has been widely recognized as the most effective approach to increasing work opportunities for people with severe mental illness [20‐22], but has not been tested in a Swedish welfare context. This is the first randomized controlled SE study in Sweden. The theoretical framework for the study is that work enables people to integrate socially and provides them with opportunities to explore and master their environment and thereby become integrated in society. The study is conducted in the city of Malmö in southern Sweden. The possible effects of the intervention on clients' vocational outcomes, such as employment rate and monthly income, as well as on their non-vocational outcomes, such as quality of life and sense of empowerment, will be evaluated at six and eighteen months after the baseline measurement. The responsible researcher is Ulrika Beijerholm, PhD (Medicine), Lund University.

The author of the paper is the principle investigator of this implementation project. The project is a part of larger research programs at the Vårdal Institute, where the three above-mentioned intervention projects are also being conducted. The responsible researchers mentioned above are responsible for designing, conducting, and evaluating the impact of the intervention studies. Thus, the investigation of implementation process and its fidelity is conducted by other researchers than those involved in the development of the interventions. However, some data collection is conducted in collaboration so that participants and other stakeholders need not experience excessive burdens with interviews, questionnaires, and observations.

An overall process evaluation plan was developed for this project. This is presented in Table 1. In addition, more detailed evaluation plans for each intervention study were created (see additional files 1, 2 and 3). The modified framework for implementation fidelity was used to define the areas to be measured. These are presented at the first column of Table 1. Steckler et al.'s [13] stepwise approach to designing a process evaluation was used as a tool for planning the practical steps in the evaluation process. In accordance with the approach, first a description of the actual program and its theoretical basis, purpose, core inputs, and expected outcomes was made. This description was summarized in a logic model. Separate logic models were created for each of the three intervention studies. As an example, the logic model for the Continuum of care for frail elderly persons, from the emergency ward to living at home intervention project is presented in Table 1.

In the second step, a detailed description of the components of the programs was created. At this stage, each component of the intervention and its intended delivery was described as these were stated in a program plan. Also, the content and delivery of the program for the control group was described. Amount of intervention services and frequency of delivering these services were described in detail. Table 2 presents the delivery process of the Continuum of care for frail elderly persons, from the emergency ward to living at home intervention.

In the third stage, general process questions were developed (second column in Table 3). One to three questions were developed for each fidelity component and potential moderating factor. For instance, subcategories of fidelity are measured through questions: 'Was each of the intervention components implemented as planned?,' 'Were the intervention components implemented as often and for as long as planned?' and 'What proportion of the target group participated in the intervention?.' To measure participant responsiveness, three questions were developed: 'How did the participants become engaged in the intervention services?,' 'How satisfied were the participants with the intervention services?' and 'How did the participants perceive the outcomes and relevance of the intervention?.' After developing the general process questions, more specific questions for each of the intervention projects were developed. These are described in additional files 1, 2 and 3.

Data will be collected for each of the three intervention studies during the entire intervention period. A multi-method approach will be used. Data collection methods include key informant interviews, non-participant observations, questionnaire studies, analysis of participants' logbooks, and other document analysis. The data collection methods for answering each of the process questions are described in the third column in Table 3. For instance, observations of work practices, project leaders' logbooks and interviews with project leaders, participants, and other relevant stakeholders will be used to answer the questions concerning implementation fidelity. In addition, in the third intervention a previously developed and validated instrument, Supported Employment Fidelity Scale [23], is used to evaluate implementation fidelity. To evaluate adequacy of strategies to facilitate the implementation, process interviews with the relevant stakeholder will be conducted and questionnaire items will be added in follow-up questionnaires to intervention participants. To measure the complexity of the intervention programs, a group of external researchers will be used. Contextual factors will be measured with interviews, logbooks, and questionnaire items. For instance, in the Palliative Care In Community Older People Nursing Homes -- Support For Nursing Staff project, a previously validated questionnaire, the Dimensions of the Learning Organization Questionnaire [24, 25], will be translated to Swedish and used to measure participants' perceptions of learning culture in their organizations. More detailed descriptions of data collection methods for each of the intervention studies are presented in additional files 1, 2 and 3.

Some data collection such as interviews and observations will be conducted in collaboration with other researchers focusing on process aspects at the Vårdal Institute. This is done to minimize the load for respondents and to best use the resources of the researchers.

Content analysis of the qualitative data, i.e., logbooks, interviews, observations, and document material will be conducted. As has been suggested [26], a coding scheme will be created and tested prior to the analyses. Results from the questionnaire surveys will be analyzed with both descriptive and analytical methods. Results of the surveys enable analyses regarding variations within the intervention group and its possible relationship with outcome variables. For instance, users' participation in the program can be studied in relation to their results in the follow-up outcome measurements.

Data collection in this project was included in the ethical applications of the intervention projects. Ethical approvals have been granted for the first intervention study: (Gothenburg University dossier number 413-08) and the third intervention study (Lund University dossier number 202/2008). For the second study, an ethical application was sent to the research ethics committee at Lund University. The committee reported that they didn't identify any ethical hinders for conducting the study (dossier number 2009-527), but made a decision that in accordance with legislation no formal ethical approval was needed for the study.