Capacity building for evidence-based decision making in local health departments: scaling up an effective training approach

Intervention activities were delivered in four U.S. states: Michigan, North Carolina, Ohio, and Washington. Prevention Research Centers (PRCs) in these states formed partnerships with either Public Health Practice Based Research Networks (PBRNs) or Public Health Training Centers (PHTCs) to conduct capacity-building activities for their state’s local health departments. For a PRC to be eligible for this study, the following criteria had to be met: a PBRN and/or PHTC existed in the same state; the PRC-PBRN or PRC-PHTC pair had a track record of productive collaboration; there were at least 30 LHDs in the state; the PRC had a strong mission and track record in training public health practitioners; and it had not already conducted extensive trainings in EBPH with LHD practitioners. The following PRC-PBRN/PHTC pairs were chosen:

The EBPH curriculum consists of nine modules (see next section for a list of modules and learning objectives) and adheres to adult learning principles (i.e., learning through problem solving and active involvement, integrating the experiences of faculty and participants into course discussions) [14],[23]. Seven of the nine modules (excluding Modules 1 and 6) include interactive exercises in which participants work in small groups (e.g., using local data to develop a concise problem statement, searching PubMed for literature on a specific topic, developing an action plan based on a logic model).

One course was conducted in each of the four states during the months of April-June 2013, and 130 participants completed the course. North Carolina and Ohio conducted 3.5-day in-person trainings. To reduce travel costs and the burden of time away from the office for attendees, the other states opted to deliver three of the nine modules via interactive webinars (Michigan: Modules 3, 5, 7; Washington: Modules 1, 5, 7) with the remaining modules delivered in two days of in-person sessions. The PRC-PBRN/PHTC partners, with help from their state health departments, recruited participants through website postings, announcements and flyers at conferences, and emails to various public health electronic mailing lists. Each state had a waiting list for their training course.

Module 1: Introduction:

Module 2: Community assessment:

Module 3: Quantifying the issue:

Module 4: Developing a concise statement of the issue:

Module 5: Searching and summarizing scientific literature:

Module 6: Developing and prioritizing options:

Module 7: Economic evaluation:

Module 8: Developing an action plan and building a logic model:

Module 9: Evaluating the program or policy:

Control group selection began with a merged database of two national surveys previously conducted by the research team. In October-December 2012, a random sample of 1,067 U.S. LHDs was drawn from the database of 2,565 LHDs maintained by the National Association of County and City Health Officials, resulting in available pre-test data from 517 LHD directors or their designees (54% response rate) [24]. Respondents of this survey identified program managers within their same LHD, resulting in the collection of 332 additional responses from December 2012 to February 2013 (67% response rate) [25]. The focus of these surveys was to identify evidence-based training, practices, and related decision-making activities.

A subsample of the merged directors’ and program managers’ surveys (n = 849) was selected to be retested to serve as the control group. Because baseline surveys found that governance structure and population of jurisdiction were significantly related to administrative evidence-based practices [24],[25], we used these variables, along with job position, to guide our sample selection. Because all LHDs in the four intervention states are locally governed, the sample was first restricted to respondents whose LHD followed a localized (decentralized) governance structure. Next, we eliminated anyone who attended or had a colleague who attended the EBPH training. Finally, we stratified the remaining group by job position and population of jurisdiction and selected participants to best parallel the intervention group’s stratification at a 3:1 ratio. Despite the improved balance across control and intervention groups, they still significantly differed (p < 0.05) on these matching variables, as there were not enough controls to match in the higher population categories and the lower job positions. These differences were therefore controlled for in the analysis. Of those invited to the control group (n = 330), 40% came from the directors’ survey and 60% from the program managers’ survey.

Baseline surveys were identical for control and intervention groups; development of this instrument is described previously [24],[25]. From this baseline instrument, the post-test questionnaire retested a set of questions related to perceived importance and availability of EBDM competencies. This set of questions was originally informed by a previous study that rated competencies for evidence-based cancer control [26] and has been used in other assessments of state and local public health practitioners [22], Jacob RR, Allen P, Baker EA, Dodson EA, Duggan K, Fields R, Sequeira S, Brownson RC: Training needs and supports for evidence-based decision making among the public health workforce in the United States, submitted]. The 10 EBDM competencies, along with their descriptions as provided on the survey tool, are listed in Table 1.

The entire baseline survey instrument underwent cognitive response testing (n = 12) and test-retest processes (n = 38) for refinement and to document validity and reliability. Cronbach’s alpha values were 0.94 and 0.89 for the importance and availability of EBDM questions, respectively, with 8 of 10 EBDM importance questions having substantial reliability and 7 of 10 availability questions rated with substantial or nearly perfect reliability [27].

Additionally, the intervention group’s post-test questionnaire asked participants to assess how frequently they used EBDM skills and to rate benefits and barriers to using course content. These questions have been used in previous evaluations of the EBPH course [16],[19].

All data were collected using Qualtrics survey software [28]. A unique link was emailed to each participant, and non-respondents received email and phone call reminders to bolster response rates. For the control group, baseline data were collected from October 2012 to February 2013 and repeated in October-December 2013. Baseline data were collected from course attendees prior to their trainings and were repeated six months after each training (October - December 2013). Respondents were offered a $20 Amazon gift card incentive for completing the pre-test and a $10 Amazon gift card for completing the post-test. The median pre-test administration time was 14 minutes, and the median post-test was five minutes. Human participant approval was obtained from the Washington University Institutional Review Board.

An average of 33 participants completed each EBPH course (n_MI = 27, n_NC = 32, n_OH = 33, n_WA = 38). Among those invited to complete a post-test (n_control = 330, n_intervention = 130), data were collected from 236 controls (response rate 72%) and 112 intervention subjects (response rate 86%). Excluding participants who no longer worked at the same organization or who had an undeliverable email address (n_control = 22, n_intervention = 6), response rates were 77% and 90%, respectively. Efforts were made to update any undeliverable email addresses by contacting the LHDs and by conducting Internet searches for the individual, but survey invitations were not forwarded if the individual was working for a new organization. Although unique survey links should have ensured that the same person completed the pre- and post-test, we compared demographic data from pre- to post-test to determine if survey links were shared without our knowledge. This resulted in the exclusion of 11 cases from the control group. Another 11 controls who did not answer the majority of the EBDM competency questions were also excluded. Among the intervention group, 14 represented state health departments or other non-LHD organizations; they are excluded from all analyses. An additional 16 intervention subjects did not complete a pre-test or did not answer the majority of EBDM competency questions. A total of 214 control and 82 intervention subjects were used in the quasi-experimental analysis (Tables 1 and 2) while the previously mentioned 16 intervention subjects were retained for the analysis represented in Table 3 (n = 98).

Respondents rated perceived importance followed by availability of each EBDM competency. Availability was defined as `how available you feel each skill is to you when you need it (either in your own skillset or among others’ in your agency).’ Importance and availability were measured on a continuous 11-point scale in which only the endpoints were defined (0 = unimportant/not available, 10 = very important/available). A `gap’ score was computed by subtracting each availability score from its corresponding importance score. A net difference was calculated for importance, availability, and gap scores by subtracting the pre-test score from the post-test score for each respondent. Difference scores were normally distributed and were used as the outcome variable in simple linear regression models. The estimated regression coefficient of a group assignment variable (coded as intervention = 1, control = 0) represented the average change in the outcome variable associated with the intervention. Standard multiple linear regression models adjusted for job position, population of jurisdiction, highest degree, gender, age, state, and years of public health experience. Frequency of EBPH skill use was measured as weekly, monthly, quarterly, and seldom/never. Benefits and barriers were measured on a 5-point Likert scale, and combined `agree’ and `strongly agree’ categories are reported. Chi-square tests assessed differences between categorical groups.

Based on this evaluation, EBPH training courses can effectively improve the availability of several skills essential to EBDM among LHD practitioners. With the development of successful partnerships and the availability of experienced trainers, such a course can be tailored and replicated in nearly any environment. Based on the experiences of the trainers and on participants’ onsite evaluations of the course, we share below some lessons learned from the adaptation and implementation of the EBPH course in this train-the-trainer model:

Similar to previous evaluations of the course [16],[20], participants requested more specific examples of how to apply an evidence-based process to practical work, more tailored materials (to their specific program areas), and more problem sharing amongst course participants. They appreciated hands-on activities and exposure to new resources and take-home tools. If possible, it is recommended to have previous course attendees share experiences in using the new knowledge and making changes within their agency.

Participants consistently requested more guidance on economic evaluation. This competency also had the lowest mean pre- and post-test availability scores among both control and intervention groups. Participants may benefit from a more simplified approach to presenting this content, with a greater focus on accessing, rather than conducting, economic evaluations.

Curriculum related to the competencies with low availability gains and small decreases in gap scores (e.g., `community assessment,’ `qualitative evaluation’) should be reviewed for opportunities to incorporate new tools, exercises, or teaching points. In some cases, low availability gains may reflect existing training efforts in that area (e.g., a state health department has invested in community assessment trainings) and the EBPH curriculum should be coordinated with those existing efforts.

Trainings were strengthened by the participation of trainers with a diversity of experience and expertise and by coordination among presenters in advance of the training to ensure consistent messaging and localization of data and concepts.

Having teams of two or three individuals from an agency attend the course together creates a `critical mass’ of trained staff in an agency [30],[31] and enhances the likelihood of influencing the agency’s decision-making processes.

A focus on training leaders with targeted or more advanced EBDM sessions is also important. Leadership buy-in is critical when building skills, fostering expectations for EBDM, and conducting participatory decision-making [32]-[34].

With promising results from the implementation of this intervention, a next step is to identify practices for further scaling up EBDM capacity-building efforts among the nation’s 2,565 LHDs. Health departments, particularly those applying for PHAB accreditation, need to enhance their workforce’s capacity to implement EBDM. The effectiveness of webinar formats should be investigated, as they can be an efficient way of addressing the increasing demands placed on public health professionals as they face declining government funding, staff reductions, and travel restrictions.

Our study was not designed to test webinar effectiveness. Only two EBDM competencies could be related to EBPH modules delivered via webinar (Module 3: Quantifying the Issue for Michigan and Module 7: Economic Evaluation for Michigan and Washington). An assessment of the related competency differences in importance, availability and gaps for participants from these states versus others yielded no significant findings (p < 0.05). While we cannot draw conclusions due to small sample sizes, these findings may imply that webinars were as effective as in-person training. It is currently unknown if web-based public health training is as effective as in-person training, and further research is indicated.

Effective webinar development can incorporate adult learning principles, focusing on scenario-based (rather than lecture-based) learning and thereby increasing participants’ engagement and ability to apply lessons to their work [35]. Maintaining the local tailoring of course material for webinar development may sustain some of the advantages (e.g., locally relevant examples and credible, familiar trainers) experienced in this trial.