Pre/post evaluation of a pilot prevention with positives training program for healthcare providers in North West Province, Republic of South Africa

Nurses and counselors were trained at four PHC clinics serving high volumes of patients receiving pre-ART and ART services in Bojanala Platinum District of North West Province. Clinic selection was conducted in collaboration with the District Management Team of the Department of Health. HIV prevalence in the district was 35.0% among pregnant women aged 15–49 attending antenatal clinics, compared to 29.5% nationally as noted above [4]. HIV fatalism, restrictive gender norms, HIV-related stigma, and general discomfort communicating about sex have been identified at the community and clinic levels in the district, and hamper progress in prevention and treatment [15].

The three training objectives were: 1) describe the importance of making recommendations to HIV positive patients about HIV prevention; 2) explain the importance of each step that makes up a comprehensive strategy for addressing prevention with HIV positive individuals; and 3) demonstrate use of the provider job aid for delivering key prevention recommendations and strategies to HIV positive patients.

The training focused on the Five HIV Prevention Steps for People Living with HIV/AIDS: A Tool for Health Care Providers. This job aid is presented in Additional file 1. Providers were taught to assess for risk and deliver prevention messages according to five themes in the job aid: 1) prevention recommendations on sexual behavior, safer sex, and alcohol and substance use, 2) adherence to ART and other medications, 3) signs and symptoms of STIs, 4) pregnancy status and intentions, and 5) condom demonstration.

I-TECH worked with clinic managers to develop a training implementation plan. The curriculum was the same in all four facilities and the same trainers conducted all sessions. Fidelity of trainings to methods and content were similar across facilities and sessions. The four-hour training session was delivered several times over one to four months, with a range of two to four training dates per clinic. All nurses at two of the clinics were trained (19 of 19 at clinic B, and 10 of 10 at clinic D), 90% at clinic A (9 of 10), and 66% at clinic C (14 of 21). 18 counsellors and other staff across the four clinics were also trained. As part of the training session, providers developed personal action plans for conducting the assessments and delivering the messages. Following the trainings, copies of the action plans were given to the providers and to a training coordinator. The coordinator contacted each trained provider by phone, every two weeks for eight total weeks, to review progress on and document revisions or updates to the action plans.

We used patient exit surveys to collect data from a convenience sample of HIV positive patients pre- and post-training at four clinics. This survey was designed to avoid revealing the patients’ HIV status to the data collection team or to others. Patients responded to all questions anonymously via Audio Computer Assisted Self Interview (ACASI), including to the question about HIV status. ACASI has been shown to be a feasible and reliable data collection method in South Africa [16‐19]. To avoid disclosure through participation, all patients were invited to complete the questionnaire, regardless of HIV status. All patients were asked about HIV prevention assessments conducted by the provider, and messages from the provider, though HIV negative patients skipped questions that were only appropriate for patients with HIV. Data from HIV negative patients served as a control for other pre/post secular trends and changes in provider behavior at these clinics. Patients were eligible to participate if they received care at the clinic on the date of survey, were aged 18 and above, and were fluent in English or Setswana.

Survey questions about HIV prevention assessments and messages varied across respondents depending on their self-reported risk behaviors and HIV status. For example, only patients who reported being sexually active in the previous three months were asked whether their provider discussed the HIV status of their sexual partner(s). The exit interview questions for HIV positive patients are presented in Additional file 2.

Eligible participants gave informed consent and received instruction on using an electronic tablet. The ACASI questionnaire was designed with the Open Data Kit and data were uploaded onto FormHub© each evening [20, 21]. The pre-test survey was conducted from December 2013 to February 2014, and the post-test was conducted from August to December 2014.

Sample-size calculations assumed that training would be associated with a 20% absolute increase in appropriate messaging among HIV positive patients, giving a sample size of 140 per time period for a two-sample test of proportions with a power of 0.90 and alpha of 0.05. This was increased to 175 to allow for 25% incomplete or missing data. Patient interviews continued until 175 HIV positive patients were interviewed during each time period.

The evaluation protocol was reviewed and approved by the North West Province Department of Health Research Committee and the Human Sciences Research Council of South Africa (REC 8/20/02/13). The Human Subjects Division at the University of Washington determined that the evaluation did not meet the regulatory definition of research under 45 CFR 46.102(d).

Informed consent was obtained from all interview participants. No identifying information was collected. Participants sat where no one could observe their answers, used headphones, and answered questions by tapping a tablet screen. Participants did not receive compensation.

Seven outcome variables were generated to measure provider messaging. Three were derived from the prevention theme of the PWP training package: sexual activity, safer sex, and drugs and alcohol. One outcome variable was generated for each of four remaining themes of the intervention: adherence to ART and other medications, signs and symptoms of STIs, pregnancy status and intentions, and condom demonstration. These were summed to calculate a summary provider messaging score.

Poisson regression was conducted with the patient as the unit of analysis. The outcome variables were the number of appropriate messages received, adjusting for the number of potential messages that should be delivered as the exposure. The number of potential messages varied by patients’ HIV status, behaviors, and responses to assessment questions. Skipped questions, as noted in Additional file 2, were not included in the denominator. Models adjusted for time period, HIV status, the interaction of the two, and seven additional potential confounders chosen a priori: 1) clinic, 2) profession of healthcare provider (nurse, counselor, doctor, other) 3) patient age (18–29, 30–39, 40–49, 50+), 4) gender, 5) education (no schooling/some primary/completed primary, some secondary or completed secondary, college/university/technical, or other, 6) race, and 7) language. All analyses used robust standard errors to adjust for over-dispersion, and were performed in Stata 13 [22].

The primary analyses were performed with complete cases. Those with missing data on provider assessment or messaging were excluded based on the assumption of data missing at random. Three sensitivity analyses were conducted to test alternative assumptions regarding missing data: 1) provider did not deliver the message, 2) provider did deliver the message, and 3) missing message could be excluded from the denominator.

Three additional sensitivity analyses were conducted. The first focused on nurses and counselors, excluding patients who saw doctors (n = 96) and other providers (n = 12), because nurses and counsellors were the intended audience for the training. The second focused on the three clinics with relatively high training coverage, excluding Clinic C (n = 332) at which only 14 of 21 nurses were trained. The third adjusted for the delay between training and post-training data collection among HIV positive patients. Observations were analyzed by quartiles from the shortest to the longest delay.

Among HIV positive patients, mean percentage of appropriate messages reported was lower post-training for all seven outcomes. The summary score decreased from 56% pre-training to 50% post-training. Among HIV negative patients, the mean percentages increased post-training for sexual activity and pregnancy status and intentions, but decreased for drugs and alcohol, and signs and symptoms of STIs. The summary score for HIV negative patients was 32% at both time periods (Fig. 2).

Table 2 reports the regression results. Among HIV positive patients, the proportion of appropriate messages delivered was lower post-training than pre-training. For the total score, the adjusted rate ratio was 0.92 (95% CI = 0.87–0.97). Rate ratios were less than 1 for six of the seven outcomes, and statistically significant for three outcomes: 1) adherence to ART or other medications, 2) pregnancy status and intentions, and 3) condom distribution. The exception was drugs and alcohol, for which the proportion increased post-training, but the increase was not statistically significant; adjusted rate ratio 1.13 (95% CI = 0.94–1.36).

Among HIV negative patients, the proportion of appropriate messages delivered was generally the same pre- and post-training. For the total score, the adjusted rate ratio was 1.05 (95% CI = 0.98–1.12). For sexual activity, the proportion of appropriate messages increased significantly post-training; adjusted rate ratio 1.52 (95% CI = 1.20–1.93).

We failed to reject the null hypothesis that the effect of the training program was equivalent among HIV positive and HIV negative patients. In the difference-in-difference analysis, the effect was smaller and negative among people who were HIV positive. For the total score, the adjusted ratio of rate ratios was 0.89 (95% CI = 0.81–0.95). The exception was drugs and alcohol, for which the adjusted ratio of rate ratios was 1.33 (95% CI = 1.01–1.75).

Results of the sensitivity analyses were substantially the same with regards to alternative assumptions for missing values and when excluding patients who saw providers other than nurses or counselors. However, results for the sensitivity analysis among the three clinics with relatively high training coverage (excluding Clinic C) differed substantially from the primary analysis. Among HIV positive patients, the total score post-training was the same as pre-training; adjusted rate ratio 1.00 (95% CI = 0.92, 1.07). The proportion of drug and alcohol messages increased significantly post-training; adjusted rate ratio 1.33 (95% CI = 1.03, 1.71), and the proportion of pregnancy status and intention messages decreased significantly; adjusted rate ratio 0.77 (95% CI = 0.60, 1.00). Results of the sensitivity analysis of the delay between training and post-test data collection also differed from the primary analysis. Results for all outcomes were more favorable among HIV positive patients interviewed immediately post-training. The summary score post-training was similar to pre-training in the first quartile of time delay; adjusted rate ratio 0.95 (95% CI = 0.87, 1.04). As the delay increased, training was associated with a decrease in the proportion of appropriate messages for the summary score and six of the seven outcomes.