Adaptation and validation of social accountability measures in the context of contraceptive services in Ghana and Tanzania

The theory of change drew on existing empirical and theoretical work on social accountability more broadly and specifically related to sexual and reproductive health (see [7, 8, 36]). This informed the dimensions in the measure. The theory of change, Fig. 1, details the inputs of the social accountability process (across the top of the diagram), how these correspond to the cumulative intermediate outcomes at three levels: (1) service users, (2) social networks and (3) service providers, which in turn, effect intended reforms in the quality of care that contribute to contraceptive choice, including increased uptake and use of modern contraceptive methods. As detailed in elsewhere, social accountability engages community members and health services actors in dialogues to identify shared challenges and develop action plans that can lead to improvements in service quality in the health system and in at the individual level, the service user or potential user knowledge and engagement with the health system, both in terms of their own health seeking behaviour and their participation in dialogues with authorities [36].

Following the development of the dimensions, we identified existing validated measures for each domain. We drew heavily on CARE’s Women VOICES tool, a validated measure the aimed to capture similar intermediate outcomes concerning maternal health in Malawi [34]. We added three domains to those used in the VOICES tool to represent the CaPSAI theory of change fully. First was the ‘Knowledge and awareness of rights’ domain that aimed to capture the service user’s perception of rights were also included [42]. For service users’ efficacy with health care providers, we used the National Health Service (NHS) measure of patient activation [28]. To capture changes in service user’s awareness of how to bring about changes and improve their local services, we created items based on theoretical work on political capabilities [22, 41]. We adapted five VOICES validated scales with acceptable reliability to the contraceptive services and local context; for example, family planning services have a charge in Ghana but not in Tanzania. A total of 14 domains were included with 75 items (see Tables 2 and 3). A five-point Likert scale was used for all the item with the exception of two sets of items had different ranges in their original format. A 6 point scale was used for Self-efficacy with health care providers (set A) ‘Women’s participation in household decision- making’ and a dichotomous scale was used for ‘Self-efficacy with health care providers (set A)’ as originally used [28, 34].

We ascertained content validity of the overall items through consultation with experts in social accountability and family planning, and the Principal Investigators who reviewed the questionnaire. Also, the World Health Organization (WHO) Forms Committee, which was composed of technical experts in contraception, social scientists, biostatisticians, and data managers, reviewed the instruments.

English surveys were available in both countries and translated in Akan in Ghana and into Kiswahili in Tanzania. Back translation was within a normal range, and pretesting the questionnaire was satisfactory for use in the study populations.

The same eligibility criteria for participants were used in both sites (see [36]). A sample of over 750 women aged 15 to 49 years accessing contraceptive services was interviewed prior to the start of the intervention in each country. Sampling was calculated using a priori sample size calculation with the ratio of 10 responses per item ratio and guidance of more than 500, which equals a very good sample for validation [4, 34]. Our sample calculation was based on 75 items of the full survey of items, including post test items. The same items were administered as part of a client exit interview survey in Tanzania and Ghana. The item related to cost of service varied between the countries as there is a nominal fee charged in Ghana. In Tanzania, the item was ‘The government ensures that family planning services are free of cost’ and in Ghana it was ‘The government ensures that family planning methods are free of cost.’

A total of 118 questions were asked of respondents upon leaving a facility, and only 58 scale items and 9 domains were included in the following analysis. Five domains and 17 items were excluded from this analysis because they were not scales or were items that applied after the intervention had been implemented.

The other survey items included questions about demographics, reproductive and family planning history, relationship status, income, occupation, and religion.

Client exit interviews were conducted on the day of recruitment at the facility in a private location. In Ghana, a total of 15 data collectors (5 females and 10 males) were trained in the survey over 3-day training in April 2018. In Tanzania, a total of 14 data collectors (8 females and 6 males) were trained over in the survey over 5 days in March 2018. Data collection was conducted using a tablet-based questionnaire to capture real-time data using OpenClinica and was later uploaded onto a secure server. Data collection in Tanzania started on 26 March and was completed on 25 May 2018, and all respondents choose to be interviewed in Kiswahili. In Ghana, data collection started on 9 April 2018 and was completed on 4 June 2018, and 46.4% choose to be interviewed in English while 53.6% choose to be interviewed in Akan.

Having being assessed for eligibility, respondents completed the informed consent process. There were no incentives given to women and girls to participate in the study. However, study participants who agreed to participate were reimbursed for their travel cost, where it was permitted by country-specific ethical requirements. In Ghana, the research team supported the travel cost to the facility with five Ghana cedis (~ 1 US dollar) given after the interview. In Tanzania, no reimbursements were given.

We assessed the item and scale reliability followed by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) for each country. We assessed the reliability of items and scales to test the internal consistency. The EFA aimed to identify the relationships among items and then group the items as part of a factor. CFA was conducted to confirm the theory behind the grouping of items.

We started with assessing reliability using the Cronbach’s alpha to determine item -to- item correlation (or homogeneity) of all 58 observed items and determined the overall alpha for each scale in each country. A Cronbach’s alpha of 0.60 was considered acceptable reliability and 0.70 or higher to be good reliability [16]. Items were removed, according to standard procedures, if the overall alpha improved substantially if an item was removed [23]. Scales with Cronbach’s alpha of ≥.60 were retained [34].

We conducted exploratory factor analysis (EFA) to determine how all 58 observed items clustered together and explore the underlying factor structure in each country. We computed the communality for each item, defined as the proportion of variance in the item attributable to common factors; and used a Kaiser-Meyer-Olkin (KMO) of Sampling Adequacy to assess the suitability of items for the factor analysis. Overall, and factors with KMO value > 0.5, for factor analysis were considered suitable for factor analysis [19, 21]. To determine the factors, we used eigenvalues in accordance with the Kaiser Criterion [20]. We examined the eigenvalues and the scree plot of eigenvalues, and factors with Eigenvalues greater than 1.0 were retained [20].

We used a rotated factor analysis using the maximum likelihood estimation (MLE) with oblique oblimin rotation to determine the factor loadings and variance. Factor loadings assess how items are weighted for each factor and the correlation between the variables and the factor. We used the proportion of variance in the item explained by the factors jointly to assess the reliability of the item in the context of all the factors. Items with factor loadings with values less than 0.40 were excluded. A minimum of three items per factor is recommended, and factors with two items or less were excluded [26].

Confirmatory Factor Analysis (CFA) was done to confirm whether the constructs identified in EFA had a good fit statistically. We applied three recommended models to test for goodness-of-fit [32]. The Standardized Root Mean Square Residual (SRMR) is a measure of the mean absolute correlation residual with a threshold of ≤0.08; the Root Mean Square Error of Approximation (RMSEA) measures the estimated discrepancy between the population and model-implied population covariance matrices per degree of freedom, and a score of ≤0.06 is acceptable, and Comparative Fit Index (CFI) measures the relative improvement in the fit of a researcher’s model over that of a baseline model, and a CFI ≥ 0.95 considered an acceptable fit [6]. The CFA structural model was is presented for each country.

In total, 750 in Ghana, and 752 women in Tanzania completed the survey. This sample is based on 10 respondents per item to produce a reliable estimate [6]. Table 4 shows the demographic characteristics of respondents.

The mean age of the women that participated in the survey was 28.4 in Ghana and 27.8 in Tanzania. In Tanzania, 84.8% of women were currently married compared to 63.5% in Ghana. A higher percentage of women in Tanzania were never-married (30.0%) than Ghana (10.3%). In both countries, injectable and implants were the predominant current contraceptive method.

To assess the scales in Tanzania and Ghana, we conducted three sets of analyses to (1) assess the reliability of the subscales, (2) determine how many factors to retain and reduce the items, and (3) verify our proposed groupings separately per country.

We started with a reliability analysis using the Cronbach’s alpha to determine item -to -item correlation of all 58 observed items and determine the overall alpha for each subscales in Tanzania (see Table 5). The standardized alpha was greater than 0.60 (the acceptable reliability in social science research) for 6 of the 11 subscales and was retained. When the alpha was lesser than 0.70, items were removed if their removal improved the overall alpha scale. A total of 6 items were removed; this included questions 38, 65, 70, 82, 83, and 84. Subscales with Cronbach’s alpha of ≥.60 were retained.

We undertook a KMO measure of sampling adequacy, and all items had a KMO of 0.5. The overall KMO score was 0.85, suggesting that there is a sufficient correlation between the variables to conduct exploratory factor analysis.

The EFA used principal factors with oblique oblimin rotation for all 58 items that yielded 11 factors. All factors had eigenvalues greater than 1.0; see Table 6 (which explained 100% of the variance).

In total, five items loaded on factor 1, five items loaded on factor 2, four items loaded on to factor 3, four items loaded on to factor 4, two items loaded on to factor 5, three items loaded onto factor 6, three items loaded onto factor 7, three items loaded onto factor 8, three items loaded onto factor 9, three items loaded onto factor 10, and two items loaded onto factor 11 Table 7. We discarded items with factor loadings of less than 0.4, and 12 items were removed, see Table 7. Factors with less than two items were excluded as this is not a sufficient number for factor analysis. On this basis, two factors were removed, and 10 factors were retained. A total of 40 items were retained.

To name the factors, we assessed what items were retained from the reliability analysis and the exploratory factor analysis and the original domain groupings. Factor 1 continues to reflect the domain items related to a client’s knowledge of their health rights and is named ‘Knowledge of health rights.’ The items included in factor 2 reflected knowledge of household decision-making in relation to finances and sexual and reproductive health and retained the name ‘Women’s participation in household decision-making’. The items included in factor 4 relate to clients’ perceived mistreatment by health care providers and is named ‘Mistreatment by health workers’. Items in factor 5 relate to the clients’ ability to attend a community meeting and are named ‘Ability to attend a community meeting.’ Factor 6 included items related to the ability to actively participate in a community meeting and is named ‘Ability to participate in a community meeting.’ Items in factor 7 relate to the client’s ‘Perception of quality’ and retain the name, ‘Perception of quality services’. The items in factor 8 relate to the understanding of how to bring about change in contraceptive clinics and are named ‘Awareness of accountability mechanisms.’ Factor 9 includes items that relate to the sense of social capital and cohesion and retains the name ‘Collective efficacy’. Factor 10 included items related to a sense of support from others during a crisis and retains the name ‘Community support in the time of crisis.’ For factor 11, the items related to how clients thought change could be achieved and retains the name ‘Mutual responsibility for and support of services’ (Fig. 2).

We started with a reliability analysis using the Cronbach’s alpha to determine item -to- item correlation of all 58 observed items and determined the overall alpha for each subscales in Ghana (see Table 8). The standardized alpha was greater than 0.60 for 8 of the 11 subscales and was retained. When the alpha was lesser than 0.60, items were removed if their removal improved the overall alpha scale. A total of 8 items were removed; this included questions 38, 39, 41, 65, 70, 82, 83, and 84. Scales with Cronbach’s alpha of ≥.60 were retained.

In the KMO measure of sampling adequacy, all items had a KMO of 0.5, and the overall KMO score was 0.80, suggesting that there is sufficient correlation between the variables to conduct exploratory factor analysis.

The EFA used principal factors with oblique oblimin rotation yielded 13 factors with eigenvalues greater than or equal to 1.0 (see Table 9).

The revised item data was used in the confirmatory factor analysis to analyze a good fit. The SRMR score of 0.05, the CFI score of 0.94, and the RMSEA score of 0.03 are good. All three score suggests acceptable goodness-of-fit.

In total, seven items loaded on factor 1, five items loaded on factor 2, three items loaded on to factor 3, four items loaded on to factor 4, four items loaded on to factor 5, five items loaded onto factor 6, three items loaded onto factor 7, three items loaded onto factor 8, three items loaded onto factor 9, four items loaded onto factor 10, four items loaded onto factor 11, two items on factor 12 and one item on factor 13 Table 10. Factors with less than two items were excluded as this is not sufficient for factor analysis, this reduced the number of factors from 13 to 10. We discarded items with factor loadings less than 0.4, and two items were removed, see Table 11. In total, 10 items were removed, and 48 items were retained.