Systems analysis and improvement approach to optimize outpatient mental health treatment cascades in Mozambique (SAIA-MH): study protocol for a cluster randomized trial

The SAIA-MH implementation strategy builds off continuous quality improvement [28] (CQI) [29] influenced by the Donabedian model [30], a framework for assessing quality of care based on interconnected components of structure, process, and outcomes. Structure describes attributes of the setting in which a provider delivers care (organizational structure, along with material/human resources). Process describes what is done to the patient as that individual receives services. Both can be measured and manipulated. Health outcomes are a function of the structure and process and changes made to them. The SAIA strategy is a multicomponent package of distinct implementation strategies [31] targeting a variety of individual, organizational, and contextual factors that foster or impede implementation success [32‐34] (Table 1). In addition, SAIA uses systems-engineering [35] principles to support task-shared workers to diagnose and prioritize problems; evaluate decision options using optimization; and recommend, implement, and evaluate actions considering the treatment cascade as a whole. By design, the SAIA model guides workers to generate ideas to address local problems in the treatment cascade, resulting in a high likelihood that proposed changes are appropriate for the local context and acceptable to the implementing team. Thus, the SAIA model focuses on increasing the chances of system changes being sustained over the long term.

The SAIA-MH implementation strategy blends external facilitation [36], ongoing clinical consultation, and service provider implementation team meetings with system-engineering tools in an overall CQI framework [28, 37] (Table 1; Fig. 2). External experts in the SAIA implementation strategy partner with respected, knowledgeable local district/provincial management in the health system. These local leaders work with the external SAIA team to contextualize and adapt the SAIA strategy to the specific context. Such adaptation has been performed as part of the original HIV-focused SAIA trial [29], as well as the SAIA-MH pilot study [25]. This included (1) specifying a MH “care cascade” to be optimized that was both feasible to populate with routine health systems data as well as relevant in the Mozambican public-sector context; (2) developing enhanced MH clinical registries and patient tracking tools to allow population of the MH care cascade; (3) finalizing the Mozambican mental health cascade analysis tool (MHCAT) used to provide a rapid systems-level view of drop-offs along the MH care cascade with an optimization function to help prioritize which step in the care cascade is weakest. Following contextualization, clinic directors, managers, and all staff directly involved in MH care delivery form a facility-level MH systems improvement team and attend a 1-week training co-led by the local district/provincial management and external SAIA-MH experts. Afterwards, the external SAIA-MH team and local management provide facilitation with facility-level system improvement teams, guiding them through the monthly iterative 5-step SAIA-MH implementation strategy process [29]. This process gives facility teams a systems-wide view of the treatment cascade performance within their health facility, guides them in collaboratively understanding their current system, helps them to diagnose problems, prioritize areas for improvement, and then recommend, implement, and evaluate improvements [35].

The 5 steps of SAIA-MH are the following:

The present project proposes to conduct a 3-year parallel cluster RCT across 8 intervention and 8 attentional control facilities to evaluate the real-world effectiveness of the SAIA-MH implementation strategy on mental health functional improvement (primary) and patient retention in care/medication adherence (secondary). Specific aim 1b will include testing causal pathway models to analyze mechanisms of action of the SAIA-MH strategy using longitudinal structural equation modeling as well as specific aim 2 estimating the incremental cost and cost-effectiveness of scaling-up SAIA-MH in Mozambique using micro-costing, time-and-motion observation, and a Markov model parametrized with cost and outcome data from the SAIA-MH cluster RCT.

We will implement a 3-year cluster RCT, randomizing 16 clinics currently providing outpatient mental health care to treatment (8 clinics) and attentional placebo control (8 clinics). We will include a 6-month baseline period prior to randomization, followed by a 2-year intervention or attentional placebo control period, and an additional 1-year sustainment phase (Table 2). During the 6-month baseline phase, all clinics will receive enhanced patient registries and patient tracking tools as used in the pilot study [25]. All facilities will receive monthly in-facility mentorship on the new data tools, allowing for examination of baseline cascade performance. During these monthly visits, data from the patient registries and tracking tools will be entered into the CommCare database by study staff. Following the baseline period, clinics will be randomized to the SAIA-MH implementation strategy or attentional placebo control.

Those receiving SAIA-MH will attend a 1-week in-person training for facility improvement teams. Facility-level teams will be determined by facility staff but will need to include a minimum of one psychiatric technician, one psychologist, and the clinic manager/director who oversees MH delivery. Following the 1-week in-person training, SAIA-MH standard operating procedures will be implemented, including: (1) structured external facilitation and clinical consultation following tablet-based guides used in pilot study (1× per week first month; 2× per month for next 2 months; 1× per month for remainder); (2) facilitation in the 5-step SAIA-MH improvement process, including: (3) cascade analysis and prioritization using the MHCAT; (4) process mapping for bottleneck identification; (5) defining system changes/developing implementation plans; and (6) evaluating and revising system changes. As in the original HIV-focused SAIA trial [29], facilities will be able to request reimbursement of up to $50/month for essential supplies needed to test systems changes. During these regular facilitation visits, clinic staff who are participating in the intervention will complete questionnaires designed to measure moderators, preconditions, and hypothesized mechanisms of SAIA-MH. In-facility facilitation and clinical consultation early in SAIA-MH implementation is intensive and has proven critical to helping teams begin to think from a “systems-engineering perspective” [35]. SAIA-MH protocols suggest cycles occur monthly, corresponding with monthly MHCAT systems analysis data, thus, we anticipate that facilities will complete 24 systems improvement cycles during the 2-year intensive implementation phase and 12 during the sustainment phase.

We will employ an attentional placebo control design whereby control facilities will mimic activities of the intervention group in time and contacts, but without the “active ingredient” of the SAIA-MH implementation strategy [40]. This design protects against key threats to internal validity, such as the Hawthorne effect [40], testing effects [41], and generally accounts for non-specific effects of SAIA-MH activities in a similar way that a placebo is used to account for expectancy effects in a drug trial [40]. Thus, facilities randomized to attentional placebo control will attend a 1-week in-person training for facility learning collaboratives determined by facility staff, including the same minimum staff above for SAIA-MH. This training will focus on a review of the enhanced patient registries and patient tracking tools, as well as discuss ethics, mental health stigma, and burnout for mental health professionals. Following the 1-week in person training, attentional placebo control facilities will receive monthly in-clinic facilitation following the same schedule as SAIA-MH paired with monthly data collection on moderator, precondition, and hypothesized mechanisms. External facilitation will focus on review of the use of the enhanced patient registries and patient tracking tools. As the SAIA-MH implementation strategy occurs at the systems level in public sector clinics in Mozambique, there will be no restrictions placed on concomitant care. Any observed external changes in the health system suspected of influencing study outcomes will be noted in final published materials. Important protocol modifications will be communicated to both ethics committees at the University of Washington and in Mozambique and a communication plan with stakeholders developed under their simultaneous guidance.

During the 1-year sustainment phase, management of data collection to produce the MHCAT will transfer fully to the provincial statistics office and SAIA-MH facilitation will be done only by local management to examine effectiveness and sustainability under fully routine conditions. Provincial authorities currently collect patient-level MH data monthly to report diagnostic and outcome data within the Ministry of Health system. Primary effectiveness analyses of the SAIA-MH implementation strategy will occur at the end of the 2-year intervention period. This project will partner with Ministry of Health staff and support the automation of existing monthly reports, with the simultaneous creation of the MHCAT cascade analysis for data feedback to the facility level.

Manica and Sofala Provinces (population: approximately 4 million; Fig. 1) were selected because of the deep relationship between investigators and the local Ministry of Health as well as the absence of existing structural interventions for MH systems improvement. The Mozambican Ministry of Health has been a leader in sub-Saharan Africa in scaling-up task-shared outpatient mental healthcare led by mid-level specialist providers called psychiatric technicians. These providers can treat all categories of mental, neurological, and substance-use disorders using both pharmacological and psychosocial evidence-based interventions. In 2014, Mozambique achieved its goal of having at least one psychiatric technician providing outpatient mental healthcare in each of the 128 districts of Mozambique [18]. However, existing evidence suggests that the quality of mental healthcare in Mozambique remains low [42] due to limited financial and human resources, few opportunities for supportive supervision, training, and re-training [18], stock-outs of essential mental health medications [43], and mental health stigma [44] among other challenges [25]. In central Mozambique, over 98% of formal health services are offered through the public sector [45], improving the potential for population-level impacts for supply-side systems interventions.

Primary SAIA-MH study outcomes will be individual-level patient functional improvement (primary), medication adherence (secondary), and retention in care (secondary) (Table 3). Patient functional improvement was selected as the primary trial outcome because it had the largest relative increase in the pilot study (aOR = 3.7, 8.9% increase, improvement from 4.2 to 13.1%) and it is the ultimate goal of MH treatment as the last step in the care cascade; therefore, improvements in upstream cascade steps and process/quality indicators should be expected to also improve functional improvement. The WHODAS 2.0 for primary outcome measurement was developed by the World Health Organization (WHO) specifically to allow cross-cultural disability measurement across settings and patient populations [46]. It has good reliability, item-response characteristics, and a robust factor structure that has been replicated across sub-Saharan Africa [46‐49]. In the original WHO validation study across 19 countries (9 LMICs) the test-retest reliability had an intra-class coefficient of 0.98, the total Cronbach’s alpha (n = 1565 across 19 countries) was 0.98 and also 0.98 for MH patients [46, 49]. The WHODAS 2.0 has been used extensively in LMICs [46‐50], shows good performance among diverse MH diagnoses [51‐53], has been applied successfully in Mozambique in previous studies [50], including our pilot study, and has been extensively applied in LMICs (including Sub-Saharan Africa) to evaluate functional improvement in pragmatic MH trials [54‐59]. Primary study outcomes will be sourced from a census of individual-level patient records from outpatient routine care, which will also be used to populate the MHCAT tool.

Eligible facilities are public-sector Ministry of Health clinics naïve to SAIA-MH, with ≥ 100 average annual outpatient MH consultations from 2019 to 2020 and the presence of a minimum of one psychiatric technician and one psychologist. For feasibility regarding project implementation in an area with challenges during rainy season, inclusion criteria for facilities included being within a 3-h one-way drive from Chimoio City, Manica Province, or Beira City, Sofala Province. Last, quaternary or provincial hospitals were excluded due to the complexity of mental health workflows across specialty services. Given these criteria, there were 9 eligible facilities in Manica Province and 8 eligible facilities in Sofala Province (Table 5). Eligible facilities will be allocated 1:1 to intervention or control using constrained randomization to maximally balance province, level of health facility, MH human resources, number of annual outpatient MH consultations, and baseline cascade performance. Baseline cascade performance will be facility-level achievement of patient functional improvement during the 6-month baseline period (Table 3). There will be no blinding of facility assignment at the level of providers, managers, or investigators as it is not feasible given SAIA-MH implementation procedures. The only blinding will occur at the level of the outcome assessor (statistical analyst). Randomization will be conducted in Stata 16.0 using the ccrand command for covariate-constrained randomization in cluster-randomized trials [60]. The allocation sequence will be generated by the senior author (BHW) and enrollment—including informed consent—led by technical implementation leads in Mozambique (VFJC and AM).

Patient- and facility-level process and quality information (Table 3) will be abstracted into a separate CommCare [61] database by study team staff. CommCare is an android-based platform to support mobile data collection specifically in low-resource settings globally. De-identified aggregate patient and visit-level data from CommCare will be directly linked to Tableau [62] data visualization software to visualize facility-level MHCAT results in real-time. MHCAT visualizations will be shared with each facility during the scheduled supervision and implementation team meetings. Separate CommCare forms will be used to track action planning, facilitation activities, mechanisms, and costs of SAIA-MH implementation. These supervision forms will be filled out by the study team during each supervision visit. Thus, primary study outcomes, implementation outcomes, and other process measures will be sourced from: (1) CommCare patient-level data collection and supervision forms/questionnaires; and (2) qualitative CFIR interviews. Monthly data will include longitudinal documentation of changes in mechanisms of effects for all 16 facilities (Fig. 4), fidelity to the SAIA-MH model (intervention only), and barriers/facilitators to implementation. During the sustainment phase, collection, and management of the CommCare and Tableau data will transition fully to the provincial statistics offices of Sofala and Manica provinces. Research and clinical staff will be trained to identify potential adverse events and instructed to report them immediately to the senior author (BHW), implementation leads (VFJC and AM) and the country director for implementation activities in Mozambique (IR). Adverse events will be reported under the schedule expected by the University of Washington ethics committee.

Individual-level primary, secondary, and process/quality outcome data will be analyzed using generalized linear mixed models using random effects at the facility and individual level. Family and link functions will be determined based on outcome distributions, although a priori specification will utilize binomial family and logit links to evaluate the odds of progressing through the MH care cascade. Absolute and percentage changes in WHODAS 2.0 scores between patients in intervention and control facilities will also be computed. Primary analyses will compare changes from the 6-month baseline to the 2-year intensive implementation period for intervention versus control facilities. Analyses will be repeated comparing effects over the sustainability phase to both baseline and implementation phases. Patient (age, sex, HIV/TB diagnosis, suicidal ideation) and health-facility adjustment factors (patient load, staff turnover or absence, distance from district/provincial headquarters) will be considered for inclusion. Sex as a biological variable will be assessed using pre-specified interaction sub-analyses to examine differential SAIA-MH effects for females vs. males. Dose-response analyses by levels of fidelity to protocol will also be examined. We will also conduct a controlled, segmented time-series analysis [63, 64] that incorporates monthly facility-level estimates from the entire 42-month study period (segmented into 6-month pre-intervention; 24-month intervention; and 12-month sustainment periods). This secondary analysis will allow assessment of SAIA-MH on number of new diagnoses, treatment initiation, total patient load, as well all other MH cascade outcomes, addressing both serial/intra-cluster correlation, and temporal patterns in SAIA-MH effectiveness. These facility-level time-series analyses will also potentially allow exploratory examination of the effects of specific systems modifications. Modifications found to have large effects will be documented as part of identifying best practices for MH care cascade improvement. All statistical analyses will be conducted blinded to study arm (intervention vs. control).

In the pilot, SAIA-MH increased patient functional improvement with an adjusted odds ratio of 3.7 and an absolute increase of 8.9% (4.2% control; 13.1% intervention) [25]. The intra-cluster correlation coefficient using outpatient mental health data in the same setting is estimated at 0.26 [https://​pophealthmetrics​.​biomedcentral.​com/​articles/​10.​1186/​s12963-015-0043-3]. Selected clinics in Table 4 conducted an average of 774 outpatient MH visits per 6 months in 2019–2020. In the pilot, 40% of patients returned for follow-up within 60 days in control periods; thus, we estimate target clinics will see approximately 300 follow-up patients per 6 months. Using these assumptions, we will have ≥ 90% power to detect an increase as small as 2.2% in functional improvement (primary), 3.6% for adherence (secondary), and 4.9% for retention (secondary); (Table 5).

At the end of the 2-year intensive implementation phase, we will conduct focus group discussions (FGDs) among each facility-level improvement team (n = 16) to collect in-depth qualitative data on CFIR constructs of interest (Table 6). Questions will be sourced from CFIR interview guides (http://​cfirguide.​org/​), supplemented by targeted elicitation of satisfaction with tools/protocols, intent-to-continue-use, and deviations from SAIA protocols. FGDs will occur after primary quantitative analyses, with facilities classified as high, medium, or low performing in terms of fidelity to SAIA-MH as well as cascade improvements. By classifying clinics, we intend to uncover salient features and determinants of successful implementation. Following analyses of mechanisms of effect, we will also classify a sub-set of individual staff (n = 10–20) and facilities (n = 4–6) into groups who showed large improvements in causal mechanistic pathways and those with little improvement across targeted mechanisms. These staff will be targeted for additional in-depth-interviews (IDIs, individuals) or FGDs (facilities) organized around the CFIR and paired with targeted elicitation of qualitative explanatory data on barriers/facilitators to specific mechanisms of action. These data will help: (1) inform the need for tailoring of SAIA-MH for specific facility and individual-level contexts that may hinder or enhance mechanism activation; (2) understand other potential moderators, mediators, preconditions, or mechanisms not captured in current causal pathway models; and (3) further unpack the mechanistic reasons SAIA-MH may show effects in some settings, and for some individuals, and not others. To refine models and understand implementation mechanisms, processes, and determinants under routine conditions, both rounds of FGDs and IDIs will be repeated at the end of the sustainment phase. IDIs and FGDs will be conducted in Portuguese by an experienced facilitator accompanied by a note-taker, audio-recorded, and transcribed verbatim into Portuguese by trained Mozambican staff. Using existing CFIR codebooks as a guide, two researchers will use a stepwise, iterative process to review transcripts and identify key deductive themes (using select CFIR constructs, implementation outcome, and targeted mechanism themes) while allowing for flexibility of other inductive themes to emerge. Following iterative coding of transcripts, the team will convene with the PI to identify coding discrepancies with the coding process repeated until consensus is achieved.

Using theoretical bases informing SAIA implementation strategy development, a review of literature for similar distinct implementation strategies (audit and feedback [65‐69], quality improvement [70‐75], facilitation [76, 77]), and mixed-methods quantitative [27] and CFIR qualitative data [27] from the original HIV-focused SAIA trial and the SAIA-MH pilot study, we have developed an initial causal pathway model to test in the present study (Fig. 4). Each moderator, precondition, and hypothesized mechanism will be measured each month during baseline, intensive implementation, and sustainment periods across all 8 intervention and 8 attentional placebo control facilities during facility-level facilitation/supervision visits. For clarity, we hypothesize that attentional placebo controls will show no significant activation of SAIA-MH mechanisms but will account for key threats to validity such as testing, maturation, external shocks, and Hawthorne effects. Mechanisms will be assessed monthly at the individual provider level and then aggregated within facilities across time, while MH cascade performance will be sourced from individual patient CommCare records and then aggregated within facilities across time. Primary analyses of mechanisms will occur after the intensive phase (Table 2). Secondary analyses will occur at the end of the sustainment phase.

Overall, this study was powered to specific aim 1a examining mean treatment effects of SAIA-MH on MH treatment cascade performance. Given the paucity of literature on methods, theories, and frameworks underlying mechanisms of multicomponent implementation strategy effects, this aim will focus on an exhaustive examination in search of practically and clinically meaningful mechanism, moderator, and precondition effect estimates for further examination in subsequent research. The goal of this aim is to contribute to theory building to inform future studies specifically powered to test hypothesized causal pathway models. No confirmatory hypothesis testing driven by p value inference will be performed, as recommended when conducting exploratory studies by Leon (2011) [78]. That said, we will have a sufficient sample size to have stable effect estimates with reasonable precision based on our design including 6 baseline measurements and 24 implementation measurements among a minimum of 3 providers across all 16 facilities. Assuming an intra-cluster correlation coefficient of 0.25 [20], if a binary mechanism leads to a function improvement odds ratio of 1.5, we will have > 0.85 power to detect the indirect effect with variance of the mediator assumed to be 0.5. Continuous mechanisms will have appreciably higher power.

Our analytical plan will progress in 3 stages and across 3 separate patient-level outcomes: functional improvement (primary), medication adherence (secondary), and retention (secondary). Repeating the analytic process across 3 dependent variables provides built-in replication, allowing the identification of consistent effects (direct, indirect, and moderated). First, we will conduct univariate examination of hypothesized mechanism effects using multi-level longitudinal SEM accounting for multiple providers nested within facilities across time. The purpose of this will be to examine single main effect estimates for hypothesized mechanisms. For example, we will estimate a univariate multi-level longitudinal SEM of SAIA-MH strategy effect predicting changes in teamwork, then predicting MH treatment cascade outcomes (Fig. 4). We will conduct diagnostics, including assessing linearity of mechanism effects, model residual distributions, and variance accounted for in mechanisms and outcomes. We will then test potential moderators in univariate models, in search of those that strongly affect strength or direction of univariate effects. Moderators will be examined for their effect on SAIA-MH strategy activation of mechanisms (left side of Fig. 4) as well as mechanism effects on outcomes (right side of Fig. 4); preconditions will be tested alongside moderators. Second, we will include all mechanisms simultaneously in a multivariable longitudinal multi-level SEM. Including all mechanisms, even those showing no effects in univariate models accounts for potential suppressor effects [79]. Moderators that showed meaningful effects in univariate models will then be included in a final multivariable model. Final model diagnostics will be verified, and variance accounted for in mechanisms, as well as outcomes, will be reported. This multivariable model will allow examination of (1) mechanisms most affected by SAIA-MH (left side Fig. 4); (2) mechanisms that had an effect on outcomes (right side Fig. 4); and (3) the product of #1 and #2 path coefficients indicating the indirect effect of SAIA-MH on treatment outcomes operating through a given mechanism. Separating out paths #1 and #2 can inform improvement of the SAIA-MH strategy; for example, if a mechanism had a large effect on outcomes (path #2), although was only weakly activated by SAIA-MH (path #1), the SAIA-MH strategy could be modified to enhance activation of this potentially high-leverage mechanism. Findings from path analyses will be incorporated into targeted elicitation of explanatory qualitative information during the second round of CFIR interviews. Individuals/facilities will be classified into those showing large mechanism changes compared to those with little improvement.

We will conduct activity-based costing and time and motion observation at both intervention and control clinics to estimate the incremental cost of implementing SAIA-MH. Throughout study implementation, we will estimate all costs that would be incurred by the Ministry of Health to administer the SAIA-MH intervention. These costs will be collected and tabulated using activity-based cost menus during start-up, intensive implementation, and sustainability periods. Costs include patient visits, medication, personnel, supervision, supplies, buildings and overhead, strategy delivery, facility support for systems changes, training, and equipment. Time-motion observations following protocols previously used by our group in Mozambique [80] will estimate personnel time needed for SAIA-MH tasks.

We will develop a Markov cohort model using R software reflecting the natural history of MH disease progression and disability, and treatment, which has been used for previous cost-effectiveness analyses of MH interventions in LMICs [81‐87]. The model will simulate a cohort with different MH conditions stratified by retention in care, medication adherence, and functional improvement. We will parameterize the model with cost and outcome data from the SAIA-MH cluster RCT and facility-level contextual data (patient loads and trends; human resources; facility types) and demographic data (gender; age distributions). In line with previous studies and economic guidelines [81, 88], costs and effectiveness will be discounted at 3% annually (varied from 0-5% in sensitivity analyses). Disability weights for DALYs will be obtained from the Global Burden of Disease 2017 [89]. Scale-up scenarios will model changes in patient retention, adherence, and functional improvement based on study results and varied in sensitivity analyses. Prior to conducting cost-effectiveness analyses, we will assess model validity/reliability by evaluating our model’s ability to accurately project trends in care cascade outcomes (retention, medication adherence, functional improvement).

We will simulate the health and economic impacts of scaling-up the SAIA-MH implementation strategy in Mozambique at district, provincial, and national levels. Health outcomes include effects on the outpatient mental health cascade (proportion retained, medication adherent, and with functional improvement). The model will estimate the effects of scale-up scenarios on: (1) retention and adherence in MH care by diagnosis; (2) functional improvement by diagnosis; and (3) Disability Adjusted Life Years (DALYs) [88] averted, cost per DALY averted, and budgetary impact from the payer (Ministry of Health) perspective. Different scenarios will examine rate and breadth of expansion (number/characteristics of future facilities/districts/provinces). We will estimate incremental costs, as well as treatment costs incurred and averted due to the implementation strategy.

We will project SAIA-MH effects on functional improvement, adherence, and retention compared to standard of care. We will calculate the cost per mental health patient achieving functional improvement and the incremental cost-effectiveness ratio (ICER) per new mental health diagnosis and DALY averted for each scale-up scenario. Analyses will be from the payer perspective, using economic productivity data to estimate mental health disability averted and incorporating costs of healthcare to evaluate economic impact.