Association between multi-component initiatives and physical activity-related behaviors: interim findings from the Healthy Schools Healthy Communities initiative

The health benefits of regular engagement in moderate-to-vigorous physical activity (PA) are well-established [1, 2]. Meeting the recommended 60 min of PA per day has musculoskeletal and cardiovascular health benefits and supports the maintenance of healthy body weight among children [3, 4]. More specifically, regular PA decreases the likelihood of becoming overweight or obese [1], with some estimates suggesting by as much as 70% [5]. In contrast, Tremblay et al. [5] found sedentary behavior significantly increases the likelihood of becoming overweight or obese by as much as 61%. Furthermore, evidence has emerged that sedentary behaviors are ubiquitous and represent a health risk independent of PA [6].

Traditional public health efforts have primarily targeted the individual, emphasizing education and behavioral skills training, and have not resulted in sustained behavior change [7, 8]. Consequently, there is an increasing focus on multi-component initiatives (MCIs), or efforts comprised of numerous strategies that work on multiple levels (i.e., individual, interpersonal, organizational, and community) at the same time, and in a best case scenario are synchronized across levels [9]. Such initiatives tend to target multiple settings such as schools, parks, and streets simultaneously—and do so in a coordinated manner to create greater intensity, effect, and sustainable change [9‐19].

To ensure success, MCIs must include evidence-based strategies, as well as extensive community engagement and attention to community needs, wants, and strengths [20]. This kind of organization and coordination of strategies across different settings is challenging, yet there are a number of successful MCIs [10‐13, 15‐17, 21]. For example, Romon et al., [13] found that over a long period of time, MCIs with various strategies (e.g., new sporting facilities built, educators employed to promote PA in schools, and family strategies organized) had a synergistic effect on overweight prevalence. Shape Up Somerville — a comprehensive intervention that included a variety of school and community-wide initiatives — was found to decrease BMI z-score in children at high risk for obesity [22].

Assessment of MCIs prove to be very challenging [18, 23], and rigorous evaluations (e.g., randomized control trial, quasi-experiment) may not be viable especially when assessing the long-term effectiveness of MCIs. Moreover, while funders and other stakeholders want to see population-level health improvements (e.g., reduced obesity), these changes may not be detectable in the short-term [24, 25]. Efforts to assess strategies and their likely positive impact in the long-term are warranted, and have been increasingly explored [15, 26‐30]. Cheadle et al. [31], developed and assessed various attributes (e.g., reach, efficacy and strength) of single strategies, and found those with higher reach and strength were correlated with improved health behaviors [31].

Numerous other researchers collaborated to explore the relationship between child obesity and characteristics of 130 MCIs across the U.S. using three predictors of population health—the purpose (the way the strategy is intended to impact behavior), duration (the length of time of the strategy), and reach (the number of people exposed to a strategy) [15, 27]. Each strategy was assessed for purpose, duration, and reach, and an overall MCI “intensity” score was calculated, based on the science which suggests that lower scores have less likelihood for sustained population-level behavior change compared to higher scoring strategies. Strategies that improve access (e.g., installation of walking/running trail), reduces barriers (e.g., increased time in physical education classes), or changes broader conditions (e.g., lighting repairs and maintenance of neighborhood playground) are higher in intensity compared to those aiming to educate or enhance skills (e.g., information on how to be physically active) [32‐34]. Evidence also suggests that when more people are exposed, and for longer periods of time, the greater the likelihood that the strategy will lead to desired outcomes [27, 28, 31].

These types of studies have made an impactful contribution to the field by establishing a systematic way for measuring MCIs. That said, further exploration of these methodologies as a tool for assessing MCIs, determining the intensity necessary to achieve health improvements, and making strategic improvements is important.

The comprehensive nature of MCIs, with multi-purpose PA strategies, show promise in addressing childhood obesity. Yet, they are extremely challenging to evaluate. Given the timing of the initiative (e.g., first 4 years) and the challenges of detecting changes in long-term outcomes such as obesity, the purpose of this study was to use intensity scoring to assess whether higher intensity MCIs implemented as part of a statewide initiative were associated with improved PA and reduced screen time among youth (dependent variables). Similar to previous research,^{25, 26, 29} we found a statistically significant relationship between higher MCI intensity scores and increased PA and decreased screen time behavior. Specifically, children living in a community with a higher average MCI intensity score had increased PA and decreased screen time. These findings are consistent with Pate et al. [38] who discovered a higher intensity scoring index for PA strategies was positively associated with non-Hispanic white children’s PA.

In a review of three MCI case studies, Mikkelsen et al. [39], found that using the full range of strategies are key to a successful implementation, and efforts to increase knowledge, enhance services, modify access, and change broader conditions should all be included. The HSHC MCIs included a variety of educational events/programs, as well as policy, practice, and environmental changes, but there were over three times more programs and events as compared to policy, practice, and environmental changes (1703 vs. 471, respectively). Although programs and events are important, over half of the HSHC strategies scored low in purpose. Based on our findings, a more even distribution of strategies, fewer lower scoring strategies and more medium-to-high scoring strategies would have led to higher overall MCI intensity scores and likely better improvements in physical activity and screen time outcomes. Furthermore, while the goal of HSHC was to reduce childhood obesity in targeted communities, and the action plans were informed by evidence-based guides, this study did not assess whether strategies within the MCI reinforced each other. Future evaluations of MCIs should not only look at the intensity of the strategies but also whether they are synchronized and reinforcing.

Practitioners implementing MCIs are all faced with the challenge of how to demonstrate impact on population-level health behaviors and outcomes while also being open to participatory approaches and co-creation with local stakeholders. Mikkelsen et al. [39], found in general that program evaluator’s focus on overall effects. Yet, unlike a controlled setting where all possible confounders can be eliminated, MCIs are unpredictable in timing and scope, and populations are exposed (or not) to a potentially causal factor or factors (e.g., a new trail, a policy to ensure children have recess). Moreover, rigorous evaluation designs are often not feasible for local practitioners to conduct. As such, research suggests the importance of gathering and reporting evaluative data through methods that are suitable and acceptable in terms of scientific standards and support a project with timely feedback [40] while also ensuring that they are low-cost and easy to administer [39]. We found that scoring evidence-informed attributes of all strategies within a MCI and calculating an intensity score addresses these factors. Assessing the intensity of strategies can demonstrate progress towards reaching long-term goals, which can take years to be realized, as well as provide timely feedback which can guide program improvements or redirect resources (e.g., to increase the intensity of a strategy or strategies).

Within HSHC, over half of the strategies aimed to increase knowledge or enhance skills, 49.2% happened only once, 51.7% reached less than 5% of the total population. Findings such as these can help stakeholders and funders improve strategies within a MCI (e.g., implement at a greater frequency, work with partners to increase the reach of a program), or provide reasoning as to why a strategy (low-intensity) may be consuming a lot of resources with little return (e.g., likelihood of long-term impact). Adjustments may free up resources to implement fewer, more intense efforts that are more likely to lead to improved outcomes. Additional evaluations should consider the use of intensity scoring for both assessing and redirecting action plans, as well as understanding the impact of a MCI on youth physical activity and sedentary behavior.

This study is not without limitations. First, data were analyzed at the school district level and limited in terms of the overall small sample of districts and number of districts per year. Different types of interventions have varying levels of impact, and thus using a MCI district-level intensity score of all strategies cannot differentiate between intervention types or combinations. Second, student behaviors were self-reported and reading comprehension levels vary between 5th and 8th graders. We took steps in ensuring the appropriateness of the survey: it was piloted with 5th and 6th graders and the survey questions were adapted from valid and reliable sources. Each question averaged a 6th grade reading level and it was deemed appropriate. While we could have used more age-appropriate versions for each grade, we felt it was more important to reduce the potential for error in terms of the coordinators and/or teachers administering the wrong survey to the wrong students (e.g., 8th grade version given to 5th graders). Additionally, using the same survey version made it more methodologically sound to allow for comparisons across grades over time. Third, many previous studies show differences in the level of physical activity and sedentary behavior by age and sex. We controlled for grade, which we felt was reasonable given that there were no significant outliers by age per grade but did not control for sex given the number of controls in our model. Future studies should perhaps consider this variable. Fourth, PA levels were based on a question asking about activity that “increased your heart rate and made you breathe hard for some of the time.” While participants might have underreported the time spent in moderate activity, we found no significant differences between self-reported PA and data collected via pedometers on 5th grade students (not reported here). Future research should consider including multiple sources of data.

Another limitation was that the data used to calculate intensity was self-reported by various school and community stakeholders. It may have been incomplete and/or subjective, and there were likely variances in reporting across individuals. The evaluation team did, however, take necessary steps to ensure consistent and quality data collection by providing guidance and training in various formats (e.g., webinars, protocols, one-on-one technical assistance) and reviewing the data regularly. Upon review, the evaluation team followed-up with HSHC coordinators for clarification as needed. It was, however, left to the HSHC coordinator to provide the final data. Sixth, HSHC’s primary goal was to reduce childhood obesity and focused on MCIs that included both physical activity and healthy eating strategies. The purpose of this investigation was to identify associations between intensity and PA/sedentary behavior – two more intermediate outcomes – rather than the longer-term outcome of obesity. Given the focus of PA/sedentary time, we only considered the strategies that targeted physical activity-related behaviors (even if they also included healthy eating). It is possible that the healthy eating-only strategies, which were being implemented at the same time, may have confounded the results. Finally, the evaluation team was not involved in the enrollment of school districts and communities, which further challenged an evaluation design with a comparison/control group. The number of districts expanded to a total of 33 by 2017, but covered most of the same communities as in the first grant year. Children attending the schools later enrolled into HSHC may have been exposed to community-level interventions implemented prior to their school being on-boarded to HSHC, and their behaviors may have been influenced and baseline data subsequently impacted. Moreover, the time in which any child could have been exposed to a strategy may have varied for similar reasons. It is difficult to avoid confounding factors in an evaluation such as this, where the evaluators have no control in enrollment and implementation, and where the funding is limited. Nevertheless, the study limitations listed above are not unique to MCI being implemented and evaluated in “real-time.”

Regardless of the limitations, this study adds to the growing literature on MCIs addressing physical activity and screen time behavior across multiple settings in various communities, and with stakeholders facilitating and co-constructing the strategies. Findings expand upon others’ efforts to identify a realistic and cost effective way to scientifically evaluate these complex MCIs. While additional research is warranted, practitioners implementing MCIs, especially with limited resources (and access to data on population-level behaviors), may consider intensity scoring as a realistic and cost effective way to assess their initiatives. At a minimum, the use of intensity scoring as an evaluation method can provide justification for, or against, the inclusion of an individual strategy into an MCI, as well as ways to increase the likelihood of the MCI impacting population-health outcomes.

MCIs are difficult to evaluate given the variations within which they are implemented. Findings from this study suggest the value of a systematic scoring approach in assessing MCIs aimed to address physical activity and screen time behavior. In addition to providing a scientific way to evaluate complex initiatives, intensity scores can provide justification for, or against, an individual strategy and ways to increase the likelihood of the MCI impacting population-health outcomes.