Community-based implementation and effectiveness in a randomized trial of a risk reduction intervention for HIV-serodiscordant couples: study protocol

To determine the cost-effectiveness of implementation of Eban II, as measured by implementation costs and potential cost savings associated with a couples-based intervention.

The study incorporates several innovative features. First, planning, governance, and specific implementation activities will be conducted in full partnership between the research team and community stakeholders, to facilitate sustainability following completion of the grant-funded project. The implementation process will involve the State of California Implementation Network (SCIN), a reciprocal, multidirectional information and technology exchange between the research team and the collaborating CBOs, an approach designed to foster effective agency adoption [26]. Second, implementation of Eban II in multiple geographically, culturally, and functionally disparate agencies requires grounding in organizational change theory because participating agencies are being asked to embrace a new evidence-based, couples-focused intervention approach. Like individual behavior change, organizational change is notoriously difficult [27]. Historically, interventions introduced into clinical settings with little attention to the context have generally met with resistance and limited success. Greater attention is now being paid to strategies for supporting organizational change, particularly by focusing on identifying barriers and facilitators to implementation and using this information to refine implementation strategies [28, 29]. For example, organizational ‘readiness for change’ [30], as well as staff expectations, perceptions of their workload, and attitudes toward evidence-based practices may encourage or inhibit adoption of these practices [31], and thus are critical to investigate and address [32]. In this study, it will also be critical to examine the broader context in which the participating agencies are situated, as the pressures that these agencies face may well impact their unique and collective capacity to deliver and sustain evidence-based practices. Third, this implementation study is organized conceptually around the Texas Christian University (TCU) Program Change Model (PCM) [33], which was developed to guide the process of transferring research into practice [34]. The PCM involves four action phases: training, adoption, implementation, and practice improvement. Fourth, the project incorporates a nine-month sustainability period in which time research team involvement in site-level activities will be limited to ‘arms-length’ evaluation, without any intervention or implementation-related activities. As recently noted in a systematic review, very little research has examined the extent, nature, or impact of adaptations to interventions that have been implemented [35]. This project will serve to fill a gap in our knowledge about sustainability as it unfolds prospectively. Finally, this project is unique in its incorporation of a cost-effectiveness analysis during the course of implementation. This analysis stands to contribute substantially to the ways in which costs affect and are impacted by community-based implementation of evidence-based practices.

Using an effectiveness/implementation hybrid type II research design [25], this study will investigate factors associated with successful implementation of Eban in a sample of routine service delivery settings, and real-world clinical effectiveness of the intervention as it is delivered to 180 couples in these settings.

To achieve our two implementation evaluation aims (aims one and two), a mixed methods evaluation will be conducted throughout each phase of the TCU Program Change Model. To achieve our intervention effectiveness aim (aim three), we will randomly assign couples in a 2:1 design to Eban II (intervention) or to a waitlist comparison condition in each of the participating CBOs. This design allows us to compare the relative effectiveness of the intervention with couples that receive the intervention immediately vs. those who are waitlisted.

We operationalize ‘successful implementation’ [36] as a combination of number of couples served (minimum of 18 couples per agency), three completed cycles of the intervention, delivery of the intervention with high fidelity, and high level of satisfaction with the intervention. As noted above, we will also be studying sustainability and cost effectiveness of the intervention for couples. Staff and organizational characteristics will be assessed at all participating CBOs in the first year, and engagement and training will be priorities. Following successful training, implementation of Eban II will commence in two randomly selected CBOs in the first year. The remaining eight CBOs will be phased in randomly with refresher trainings in years two through five. At first glance, it might seem optimal to start all participating agencies at the same ‘baseline,’ but the structure of the funding mechanism makes such a design prohibitive due to the costs associated with implementation in each agency. By limiting training to two to three CBOs per year, we will be able to devote sufficient effort to test whether our implementation approach does in fact contribute to high fidelity implementation. Also, this randomized ‘roll-out’ implementation, or dynamic wait-listed design, has very good statistical properties, including higher power than traditional wait-listed designs [37] and less vulnerability to external, uncontrolled factors [38]. After each of the collaborating agencies completes three eight-week intervention cycles, sustainability will be studied for nine months, during which time only technical assistance will be provided, as discussed further below.

HIV infection rates in California rank third in the U.S.; Los Angeles and Alameda Counties have the largest concentrations of African Americans in California and, accordingly, the highest proportion of African American HIV/AIDS cases [39]. A recent report indicates that the City of Oakland (Alameda County) has the highest and most rapidly growing incidence rate of diagnosed and undiagnosed HIV in the country and is struggling with what has been referred to in the news as an ‘unstoppable epidemic’ [40]. The State of California and local stakeholders have responded to this challenge with active prevention programs such as Get Screened Oakland [41, 42].

We recruited ten HIV/AIDS CBOs in Los Angeles and Alameda Counties to serve as study sites. All of these agencies are well-established CBOs that serve large numbers of HIV-infected African Americans. In fact, these CBOs expressed particular interest in offering services to couples because they currently do not provide such services. These agencies were identified as having met seven key elements identified in the literature as important in determining agencies’ readiness to implement a new intervention [43]. These include: a respected local community advocate; strong administrative support; formal organizational commitments and stability; commitment of necessary resources to incorporate the program into existing services; program credibility within the community; adequate facilitators/staff; and potential for the program to be self-sustaining or willing to seek additional funding. In addition, all of the agencies have adequate space for conducting private assessment and group sessions.

Approximately 200 staff members from these agencies (~20 per agency) will complete the staff- and organizational-level measures described below. A subset of these individuals (approximately 50 ‘key stakeholders,’ including agency administrators, site coordinators, and facilitators) who are directly involved in implementation will complete semi-structured interviews.

In addition, in order to support potential scale up and spread of the intervention, each of the agencies will be asked later in the project to identify two to three additional agencies that may be interested in learning about Eban II. In the last six months of the study, the Management Team will meet at a retreat with the key stakeholders and representatives from the agencies that they have identified. Preliminary findings will be discussed, and participating agencies will describe their experiences and engage in dialogue about implementation of Eban II.

The study employs multiple community outreach and marketing strategies to recruit and screen 215 to 230 couples, expecting to enroll 180 African American heterosexual couples. The dropout rate in the Eban randomized controlled trial (RCT) was low at 13% in the intervention condition [44]. We conservatively project a 17% to 22% overall attrition rate since this study will be conducted in CBOs rather than in university-based study sites. Eligibility criteria are as follows:

Our implementation approach for Eban II is phased to correspond conceptually to the PCM and involves several strategies and tools that support implementation in the participating CBOs (see Table 1).

These phases involve: developmental evaluation, delivery of the Eban II intervention, couples-level data collection (aim three), implementation-focused evaluation, progress-focused evaluation, and assessment of costs.

Agencies will be activated sequentially as cohorts are completed. The flow of agency participation will be as follows: year one, two agencies; year two, two agencies; year three, three agencies; year four, three agencies. All agencies will be required to complete three Eban cycles (eight weeks/cycle). In year five, three-month follow-up evaluation data collection will continue and sustainability will be studied at the agencies that complete the active phase in year four.

Initial phone or in-person contact with project staff will be made by the HIV-positive partners who will be screened to determine their eligibility. If eligible, they will inform their partner who will contact the staff and be screened to confirm their eligibility. Consented couples will be scheduled for the baseline assessment. Sociodemographic information on couples who do not meet criteria or refuse to participate and reasons for non-eligibility or refusal will be collected to permit comparisons between eligible participants and non-participants. The assessment battery will be repeated at the posttest and three-month follow-up. If couples break up or a partner is incarcerated or dies, the remaining individual will be followed throughout the three-month follow-up. These individuals will receive a ‘break up’ curriculum individually and will be assessed in order to examine their adherence to safer sex behaviors with new partners. Couples will be randomly assigned in a 2:1 ratio to either Eban II active treatment or to waitlist control (WL), with 2/3 (n = 118) of the couples assigned to Eban II and 1/3 (n = 62) assigned to WL in a cross-over design for group comparisons. At the end of the three-month follow-up, couples assigned to WL will be offered the intervention and their three-month data will serve as their baseline for the treatment analyses. During their time on the waitlist, couples will be called weekly by the site coordinators to ensure that they are receiving standard care.

This eight-week, standardized, manualized intervention is facilitated by a male–female team. The intervention (see Table 2) involves: strategies that address the broad array of factors influencing risk behavior among HIV-affected couples; dyadic and group processes that take advantage of relationship and group dynamics; and educational and culturally appropriate sessions [45].

Unlike a traditional RCT, individuals in this study will not be incentivized for attending sessions. However, our experience indicates that incentives enhance attendance and active participation; community agencies frequently provide such incentives to enhance participation in routine programs. Therefore, we will provide minimal participation incentives similar to those often provided routinely by the agencies, such as bus tokens and gift cards at pre-, post-, and three-month follow-up. In addition, usual care, services, and referrals will be available to the WL couples while they wait to enter the risk reduction program. We will learn more about the impact of incentives during sustainability, when these incentives are no longer provided.

The self-report assessment developed for Eban and pre-programmed into Audio Computer-Assisted Self-Interview (ACASI) [46] takes about 90 minutes to complete. ACASI provides both audio and video presentation of the questions and response options on a laptop computer. ACASI has been shown to significantly decrease social desirability bias [21].

Several types of assessments will be used to track implementation processes, specifically with regard to fidelity, dose, and intensity of intervention. Throughout the intervention, we will monitor: session attendance, session completion, and participant satisfaction with sessions. Facilitators will also complete short surveys noting adherence to the curriculum, level of participation, specific obstacles that may have arisen, and what components appeared to be most/least appropriate for the session’s participants. Facilitators will also complete overall ratings of each couple’s engagement, competency, and knowledge. To assess fidelity, sessions will be digitally recorded. For the first intervention cohort at each site, all recordings will be reviewed and feedback will be provided to facilitators. Subsequently, a random 15% sample of all session recordings will be reviewed and scored for fidelity to core elements, with a criterion of 80% or more of the total elements considered acceptable [47].

At the organizational level, this component will involve assessing the monthly SCIN conference calls. Minutes of all calls will be maintained (including attendance rosters), as will minutes from all project-related meetings. Regarding progress at the couples-level, this component will involve weekly calls between the regional project managers and the site coordinators. A periodic onsite review of procedures at each site will be conducted, and a random selection of 5% of pre- and post-test and follow-up interviews for each site coordinator and data collector will be reviewed to ensure that the protocol is administered in a standardized fashion.

The cost of the delivered services will be calculated using a customized cost-analysis spreadsheet which uses state of the art recommendations for cost analysis as articulated by the US Panel on Cost-Effectiveness in Health and Medicine [48]. The cost analysis data will be collected by the data collectors and sent on a quarterly basis to the management team, and reviewed with the cost consultant in quarterly conference calls to analyze the cost information, identify trends, and solve any problems with data collection.

Phase four involves interpretive evaluation, assessment of sustainability, and a project retreat. Because this project will occur in waves, practice improvement will be examined after each agency completes the active intervention phase (delivery of the groups and completion of three-month follow-up). Sustainability will be examined for nine months following the completion of the required three groups and follow-ups.

Key stakeholders will have completed interviews regarding their satisfaction with Eban II during active implementation, and their expectations regarding sustainability and cost effectiveness. Following sustainability, these stakeholders will be re-interviewed.

We define the sustainability phase as beginning after the active implementation phase is completed at each site (i.e., baseline through the 3-month follow-up). At this point, the reliance on grant funds ends and sites will be encouraged to integrate Eban II into their usual services. At each site, the nine-month sustainability period will be preceded by a pre-sustainability workshop to identify and address any issues that the agencies perceive to be barriers [49]. During sustainability, Eban II will be delivered in the CBOs with trained staff serving as co-facilitators, supported by technical assistance (including quality assurance) from the management team. Technical assistance will include retraining in the intervention, sharing resources, offering suggestions on lessons learned, and review of session tapes to assess fidelity. Agencies that discontinue participation during this phase will be contacted to encourage participation and/or to identify the reasons for discontinuation. Agencies that independently deliver two eight-week cycles of the intervention with two to three couples in each cycle and maintain fidelity to the intervention core elements will be considered to have achieved sustainability. Pre-post couples-level measures will also be collected in order to examine outcomes. Sustainability will be assessed by documenting the number of couples that each agency recruited and enrolled in Eban II; the number of couples who completed the intervention; and how much technical assistance they needed and received.

Additional funds will be sought to support a project retreat for the management and implementation teams to come together with additional representatives from the SCIN to discuss preliminary findings and the future of the intervention. Individuals from interested agencies that were not part of the original 10 CBOs will be invited to attend a designated portion of this retreat, and a structured discussion guide will be used to explore likelihood of adoption, perceived need for the intervention, preliminary perceptions of the program, and its trialability.

An electronic Staff Survey will capture basic demographics of staff (~20 per agency) including education level and professional experience. The 50-item Evidence-Based Practices Attitudes Scale (EBPAS) [50] will be used to assess staff attitudes toward evidence-based practices. The widely used Maslach Burnout Inventory (MBI; Maslach) [51] will be used to assess staff experience of workload. Subscales from the TCU Survey of Organizational Functioning (TCU SOF) [52] will be used to assess motivational factors, program resources, staff attributes, and organizational readiness. All subscales of the SOF have demonstrated good internal consistency and validity [53].

In addition to these structured measures, semi-structured interviews will be conducted with a subset of key stakeholders (approximate n = 50; five per agency). Process-focused interviews will include questions about feasibility, satisfaction, perceived fidelity, barriers, desired revisions, perceived cost-effectiveness, and expectations for sustainability. Sustainability-focused interviews will include similar questions, but will refer to the sustainability period and their satisfaction with and future expectations of the utility and cost-effectiveness of Eban II as a long-term option for agencies’ usual menu of services.

We will use a cost-analysis spreadsheet approach to measure the cost of HIV prevention, case management, and adherence services for CBOs providing front-line services. The spreadsheet incorporates key cost analysis principles, calculates the cost per client of HIV/AIDS services delivered, and also conducts threshold analyses to set performance standards for how many HIV infections would have to be averted for Eban II to be considered cost-saving or cost-effective to society.

Primary behavioral and physiologic outcomes are as follows:

We will also track concurrency, seroconversion, and incident STIs. In order to control for response bias, several steps will be taken, including: assessing sexual behaviors over a relatively brief period (i.e., past three months at baseline and follow-ups and past seven weeks at post-intervention); providing participants with a calendar for benchmarking salient dates; involving only trained data collectors in the data collection; using the ACASI; and administering a five-item measure of social desirability bias [54].

Potential covariates include demographic characteristics; mental health status as measured by the 45-item Brief Symptom Inventory (BSI); history of sexual abuse as measured by the Wyatt Sex History Questionnaire (WSHQ) [55]; and history of physical abuse as measured by the 19-item Revised Conflict Tactics Scale (CTS) [56], which assesses both history and recent (i.e. past 90 days) experiences of abuse. Hypothesized mediators [57] include condom use self-efficacy, which has been linked to effective condom use [58, 59], as measured by the Condom Use Self-Efficacy Scale [60]; condom communication self-efficacy as measured by a nine-item scale measuring confidence in negotiating the use of a male condom [60]; and couples’ sexual communication skills as measured by a seven-item scale on the level of reported comfort on safer sex communication items [61]. Moderators include several variables identified in Eban I as likely to moderate intervention effects [57]: the length of the couple’s relationship, because those in longer relationships are less likely to practice safer sex [45]; relationship satisfaction will be assessed with the Dyadic Adjustment Scale [62], which asks how partners relate to each other about family finances, time spent together, relational factors, and intimacy; substance use/abuse (frequency and amount of alcohol use each day in the past three months and age of first use) as measured by the CAGE [63] to assess dependence on alcohol, and section B of the NIDA Risk Behavior Assessment (RBA) [64] to assess the frequency, modality, and level of use of licit and illicit drugs in the past month.

Sample size calculation and power analyses were conducted at both the agency- and couple-levels to ensure that we have sufficient statistical power for all analyses. At the agency level, the power analysis was based on the comparisons of key measures from providers (e.g., provider attitudes toward evidence-based practices). From the 10 selected agencies, we will collect data from 100 non-clerical staff (10 providers per agency) in order to provide sufficient power to examine the within- and between-agency variation. With a type I of 0.05, type II error of 0.2 (or power of 80%), intra-agency correlation at 0.15 level, 100 raters from the 10 agencies will enable us to detect an effect size as small as 0.9 in standard deviation unit for measures from staff [65, 66]. Non-parametric and parsimonious models will be used to accommodate the limitation of a relatively small number of agencies.

At the couples level, calculations were carried out based on the comparison of key outcome measures of incidents of unprotected sex and an increase in the proportion of condom use (i.e., the number of times condoms were used during intercourse divided by the number of times sexual intercourse is reported) between intervention and waitlist control. With a repeated measures design, type I error 0.05, type II error 0.20, intra-couple correlation 0.4, the mean number of available repeated measures 2.5 (from baseline, post and three-month measurements), and a 2:1 ratio of intervention and waitlist control, a total of 174 couples would allow us to detect an effect size as small as 0.30 in standard deviation unit for number of unprotected sexual intercourse acts [65, 67]. Thus, our planned sample of 180 couples provides sufficient power.

All interviews will be digitally recorded and transcribed by trained staff. ATLAS.ti will be used for qualitative analysis. Using the constant comparison analytic approach [69], a preliminary codebook will be developed both inductively and deductively from a sub-sample of interviews within and across agencies at baseline. Qualitative findings at baseline will be augmented by preliminary analyses of staff-level data from the structured measures described above (e.g., burnout, attitudes toward evidence-based practices, etc.), and a baseline profile will be developed for each agency. These profiles will be used to tailor implementation at each site. This approach of using baseline data as diagnostic and informative for tailored implementation has been employed by the lead author (AH) in a prior implementation study [29]. The codebook will be elaborated upon and adjusted as each round of interviews is reviewed until thematic saturation is achieved within and across cycles of interviews. Interviews will be compared within each agency, across agencies, across different types of respondents, and over time. Additional sources of qualitative data (i.e. meeting minutes, archival information) will also be included in the data set. We will analyze the data specifically for barriers to and facilitators of implementation, including but not limited to the ways in which the project’s strategies and tools affect adoption, fidelity, and sustainability. In addition to identifying themes and patterns qualitatively, we will examine statistical associations between important process and outcome variables such as satisfaction with the intervention, fidelity, and retention, and improvement in behavioral outcomes. Agency profiles will be revisited and further developed at the end of the active implementation phase, and again after sustainability.

We will use repeated measures regression models and multi-level hierarchical modeling [70, 71] to evaluate the effect of characteristics of agencies, staff, and couples on the behavioral and psychosocial outcomes. For repeated measures regression models, we will directly incorporate measures into models and deal with intra-class correlations from agency and staff through covariance matrix. With the multi-level nested structure from individuals, couples, agencies, and staff, the hierarchical models will be fitted in three possible steps: (a) to set the foundation for a class of hierarchical models, first we will construct individual-couple-level models with repeated measurements (baseline, post and three-month) within a couple that involves time (and intercept); (b) we will then construct across the couples-level models that involve explanatory variables such as couple demographics; health history; history of alcohol and recreational drug use; HIV/AIDS risk reduction knowledge; perceived couple sexual norms; intervention components, and a binary variable indicating immediate active treatment or wait list control; (c) we will model the regression coefficients obtained from step b at the agency level as functions of explanatory variables of agency characteristics (e.g., organization size and budget, number of staff, etc.). Finally, we will then combine models from steps a, b, and c to form the complete model, which will be estimated through special procedures and software, such as WinBUGS and MlwiN [72, 73].

To determine the cost-effectiveness of implementation of Eban II, standard cost-effective analysis will be conducted using health economic analysis approaches. Two-staged models (instrumental variable regression) [74, 75] will be applied and implementation costs and potential costs saving through the intervention will be estimated.