Healthy apple program to support child care centers to alter nutrition and physical activity practices and improve child weight: a cluster randomized trial

The HAP pilot evaluation was designed as a cluster randomized trial, with child care centers treated as clusters. The clustered design was selected because the CCHP serves child care providers, and does not have direct contact with the parents or guardians of children. The CCHP health workers and child care providers were not blinded to treatment allocation.

Figure 1 provides an overview of the timeline, design and data collection for the HAP pilot evaluation. Child care centers were randomly allocated to receive routine CCHP services plus HAP (CCHP + HAP) or routine CCHP services plus HAP after a one-year delay (CCHP + HAP Delayed). HAP resources were offered to CCHP + HAP centers beginning in Spring 2013 through Summer 2013. To ensure that all child care centers received the best practice resources, child care centers allocated to the CCHP + HAP Delayed group were also offered HAP resources, only after a delay, in 2014–2015. The one-year delay allowed the evaluation to have a control group, as well as CCHP staff to stagger the outreach and technical assistance workload. Data were collected at the child level and child care center level, each year, according to the academic calendar, which begins with Autumn enrollment. The same measures were collected over time. There was no overlap between the Implementation year 1 activities and Follow-up year outcome measures in 2013–2014. The Follow-up year measures were collected in Autumn 2013, months after the Summer 2013 HAP workshops were completed.

The evaluation design allowed for comparison of the randomly allocated intervention and control groups in the Baseline year, in the Follow-up year (2013–2014, i.e. after the CCHP + HAP centers received HAP resources), as well as  in Implementation year 2 (2014–2015, i.e. after the CCHP + HAP Delayed centers received the HAP resources). The working hypothesis for this design was that the randomly allocated groups would differ significantly with respect to outcomes in the Follow-up year, but not in the Baseline year or Implementation year 2. The evaluation design also made it possible to describe within-group, pre-post intervention changes, e.g. changes between 2011 and 2012 and 2014–2015 for each group, as well as the whole cohort of child care centers served by CCHP.

All child care centers that participated in CCHP nutrition screenings in 2011–2012 were eligible for the HAP pilot. The CCHP provides services to child care centers that primarily serve low income children in San Francisco and do not have federal, state or school district funding. All child care centers had the option to decline each voluntary CCHP screening and/or decline participation in HAP pilot activities. A $25 gift card was offered to one representative per child care center for participation in the HAP pilot. The target population inclusion criteria did not change after trial commencement. Child care centers that were eligible in 2011–2012 remained eligible for the duration of the pilot, even if they declined CCHP services in interim years.

The target population included all children, ages 2 to 5y, enrolled at eligible child care centers. Families consented to CCHP services, including use of data for public health program planning and evaluation, and could opt out of screenings at any time.

Child care centers that were closed in Autumn 2012 or declined CCHP services for 2012–2013 before the randomization were ineligible for the HAP pilot. Child care centers with funding from Head Start, the San Francisco Unified School District, or Community College District were ineligible to receive CCHP screenings, and excluded from the HAP pilot. Child care centers that declined one or both BMI screenings in any given year were excluded from evaluation analyses for that year, because of missing data regarding the primary outcome of interest, annual change in BMI between the Autumn and Spring screenings. Children who declined one or both screenings or were absent on the date(s) of screening in any given year were excluded from evaluation analyses for that year. No other exclusion criteria were applied.

In Summer 2012, the SFDPH epidemiologist randomized child care centers in two blocks, one block for each of two CCHP health workers responsible for BMI screenings. Each health worker has a specific caseload. For each health worker, eligible child care centers were listed in alphabetical order. A list of the same length, of random, unique, unsorted numbers was generated using randomizer.​org.​ For each health worker, child care centers had an equal chance of being assigned to CCHP + HAP or CCHP + HAP Delayed. Enrollment in the child care centers ranged from 14 to 160 children. The mean (SE) enrollment in child care centers did not vary significantly by treatment assignment (48 (9) vs 37 (4)), and remained stable over time.

Table 1 summarizes the services and resources offered to the randomly allocated groups in Implementation year 1, 2012–2013. The HAP resources included an invitation packet, which included information about the HAP, a self-assessment for child care providers (see Additional file 1), and information about the gift card incentive for completing the self-assessment. The HAP resources also included a goal setting worksheet (see Additional file 2), hard copy Tip Sheets (see Additional file 3) and online Technical Assistance materials (http://​healthyapple.​arewehealthy.​com/​resources.​aspx). After Implementation year 1, HAP resource distribution was paused until 2014–2015. In 2014–2015, all child care centers were invited to participate in the HAP.

Throughout the evaluation period, routine CCHP services were given to centers allocated to the CCHP + HAP Delayed group. These services included public health nurse consultation, health education, and hearing, vision, dental, and nutrition screenings and referrals. Each academic year, in Autumn and Spring, the same two trained health workers visited all child care centers that accepted the free CCHP BMI screening. The protocol was essentially the same each year. Screening data were collected as described below (see measures section). Child-specific weight status reports were given to the child care centers to send home to the parents or caregivers. The current prevalence of overweight or obese children at the child care center was reported to the child care provider. In Autumn 2013, CCHP protocol also included drinking water promotion for all child care centers. The promotion included distribution of a pamphlet about the benefits of drinking water for child obesity prevention and child-sized water pitchers for all centers (Help-Yourself Pitchers, Lakeshore Learning, San Leandro, CA). In Implementation year 2, 2014–2015, CCHP + HAP Delayed centers were invited to participate in the HAP, and given HAP resources as described below.

CCHP + HAP centers were offered the same services provided to CCHP + HAP Delayed centers. During Implementation year 1, in addition to the routine services, CCHP + HAP child care centers were invited to participate in the HAP pilot. CCHP public health nurses or health workers introduced the HAP resources and process, in-person, to child care center staff. They delivered the HAP invitation packet to the child care center, and spent up to 16 h per child care center, providing one-on-one support to each child care provider regarding the HAP self-assessment, goal setting, action plans to achieve the goal(s), Tip Sheets and online Technical Assistance resources. The CCHP staff had written instructions and form templates to standardize the delivery of one-on-one support (see Additional files 4 and 5).

In Summer 2013, the San Francisco Children’s Council offered two workshops to address needs identified by the HAP participants. A nutrition workshop addressed ideas for seasonal menu planning, child nutrition education resources for parents, and policies for food for holidays or celebrations. A physical activity workshop addressed how to integrate age-appropriate physical activity and academic learning for preschoolers (http://​www.​pkimbrell.​com/). CCHP and HAP staff extended verbal and/or email invitations to the staff contact at each child care center for any of the center staff to attend the workshops. Child care centers were invited to participate, but not required, because all CCHP services are optional and voluntary. Child care center staff were not paid to attend the workshops.

In Autumn 2013, child care providers who completed the initial self-assessment were invited to re-take the self-assessment. Their most recent self-assessment scores were used to determine HAP award eligibility. A ceremony in Autumn 2013 recognized child care providers who met criteria for a HAP award (http://​healthyapple.​arewehealthy.​com/​HealthyAppleAwar​d.​aspx).

In Spring and Summer 2014, while the HAP pilot data and program structure were reviewed, CCHP staff did not offer HAP materials to any child care centers. HAP resources were made available to child care centers in the CCHP + HAP Delayed group, beginning in Autumn 2014.

CCHP health workers visited all child care centers that requested bi-annual BMI screenings in the Autumn and Spring of each academic year. The health workers recorded child age and sex, and measured child weight and height using a standardized protocol and calibrated instruments (SECA digital scale 874 and SECA Portable, standalone stadiometer with headboard, Hamburg, Germany). Measurements were taken after the child removed outer layers of clothing and shoes. The age- and sex-specific BMI percentile and BMI z-score for each child was calculated relative to the CDC 2000 growth reference [14] using Epi Info 7 software (CDC, Atlanta, GA). The change in BMI percentile was calculated. Incident cases of overweight or obesity were identified.

Every Autumn, the CCHP health workers recorded nutrition and physical activity resources that were visible to them during their visit. They interviewed the center staff about nutrition and physical activity resource needs. During the HAP evaluation period, in Autumn 2012, Autumn 2013, and Autumn 2014, the health workers gathered information about 3 practices which were relevant for tracking changes in response to HAP workshops that were offered in 2013: Use of a physical activity curriculum (yes/no); Staff usually join in physically active play with children (yes/no); and Pitchers of drinking water visible in the classroom (yes/no). Data regarding the 3 index practices were combined into a score to track and compare cumulative changes in these practices in all CCHP + HAP and CCHP + HAP Delayed centers. The score was a simple count of the number of index practices with possible values 0 to 3.

The primary outcome for the evaluation was annual Autumn to Spring change in child BMI percentile at the child level and child care center level. The annual change in BMI percentile was calculated as the difference between the Autumn and Spring screening results. The BMI change was also expressed as annual change in BMI z-score, given that z-scores can be compared across ages and can quantify extremes of the BMI distribution [15]. The incidence of overweight or obesity during the school year was measured as the proportion of children who were normal weight at the Autumn screening, who were overweight or obese at the Spring screening.

The secondary outcome was child’s relative odds of exposure to the 3 index practices available from data routinely collected by CCHP for all child care centers. At the child care center level, the index practice score was evaluated as a dichotomous variable (proportion of centers that had 2 or 3 index practices vs 0 or 1). At the child level, the score was expressed as a continuous variable, proportion of children exposed to 2 or 3 index practices.

HAP process measures, reflecting the intensity of exposure to HAP resources, were collected by the San Francisco Children’s Council Healthy Apple Program Coordinator. The process measures included the number of child care centers that completed the HAP self-assessment(s), set goals, received technical assistance materials, attended workshops, improved best practices, and received a HAP award. The proportion of children, who were potentially reached by child care provider participation in all steps of the HAP process from self-assessment to award recognition, was estimated.

The HAP pilot evaluation used de-identified public health screening data, collected by the CCHP over four school-years (see Additional file 6). Statistical analyses were done using Stata/SE 9.2 software (StataCorp, College Station, TX). Missing data were assumed missing at random, because the primary reasons for missing data were shifts in funding for the child care centers, and provider preference to schedule only one screening visit each year. P-values less than 0.05 were considered statistically significant.

The evaluation population was described in terms of the flow of child care centers through the HAP pilot, age of children attending participating child care centers, and prevalence of child overweight or obesity within about 90 days after Autumn enrollment each year. The primary outcome, change in BMI percentile during the academic year, and related outcome measures including the change in BMI z-score and incidence of overweight or obesity during the year, were described by intervention group and year. The intracluster correlation coefficient (ICC), measure of within-child care center variance relative to between-child care center variance, was estimated to describe clustering in the outcome data in the Follow-up year and Implementation year 2.

The CCHP + HAP vs. CCHP + HAP Delayed groups were compared in intention-to-treat analyses, which accounted for clustered data. Year-specific hierarchical linear models used child-level data to test for a main effect of time between the Autumn to Spring screenings during the academic year (Time) on child BMI percentile and z-score, and modification of the Time effect by invitation to the HAP (Time x HAP interaction). The year-specific models included child sex, age and overweight or obese status (yes/no) in the Autumn, and type of center (CCHP + HAP or CCHP + HAP Delayed) as predictors of the intercept, and CCHP + HAP vs CCHP + HAP Delayed status as predictor of the slope, i.e. annual change from Autumn to Spring. Year-specific model random-effect parameters used an independent covariance structure.

Year-specific logistic regression models used child level data to compare CCHP + HAP and CCHP + HAP Delayed child care centers with respect to the relative odds of incident overweight or obesity during the academic year. The logistic regression models adjusted for age, sex, and BMI percentile at the Autumn screening, and used robust standard errors to account for clusters of child care centers.

Multi-year hierarchical linear models combined child care center-level data from all 4 years to compare year-to-year change. These models tested if the child care center annual mean changes in BMI percentile and z-score in 2012–2013, 2013–2014 and 2014–2015 differed from the corresponding changes in 2011–2012 (main effects of Year), and if differences from year-to-year were modified by invitation to the HAP (significant Year x HAP interaction). The year-to-year models included the child care center mean child age, percent male, and prevalence of overweight or obesity in Autumn as predictors of the intercept, and HAP invitation as predictor of the slope. Year-to-year models specified an unstructured covariance structure.

The multi-year model provided information about pre-post intervention change over the four-year evaluation period for the CCHP + HAP and CCHP + HAP Delayed groups. The main effect of Year in the multi-year model represented an estimate of the change in outcome relative to the Baseline Year. If this change was not significantly modified by the Year x HAP interaction, then the estimate was interpreted as the estimated within-group change for both groups. Multi-year models without the Year x HAP interaction term were used to estimate the change in annual mean BMI changes between the Baseline Year and Implementation year 2 (2014–2015) for all centers served by CCHP.

The secondary outcome, proportion of children exposed to 2 or 3 index best practices, was described by intervention group and year. A year-specific logistic regression model, with robust standard errors, compared the CCHP + HAP and CCHP + HAP Delayed groups with respect to the relative odds of exposure to 2 or 3 best practices in 2013–2014.

The number of child care centers that were invited to the HAP, completed a self-assessment, set practice improvement goals, received technical assistance, attended workshops, improved practices, and received a HAP award was described by intervention group and year.

A total of 43 child care centers were eligible to participate in the HAP pilot. Figure 2 describes the flow of eligible child care centers through the HAP pilot, according to the CONSORT 2010 guidelines [16]. The randomization allocated 19 child care centers to the CCHP + HAP group and 24 centers to the CCHP + HAP Delayed group. Child care centers that declined bi-annual BMI screening after randomization, in 2012–2013, 2013–2014, or 2014–2015, respectively, did not differ significantly from those that accepted screening with respect to the Autumn prevalence of overweight or obesity or likelihood of assignment to the CCHP + HAP or CCHP + HAP Delayed group in 2011–2012.

In 2012–2013, CCHP BMI screenings were not delivered as expected, because child care centers became ineligible for CCHP services due to unforeseen changes in Head Start funding, and/or child care centers declined one or both Autumn and Spring CCHP BMI screening(s) for the year. In the CCHP + HAP group, 5 centers did not participate in one or both CCHP BMI screenings in Autumn 2012 and/or Spring 2013. In the CCHP + HAP Delayed group, 7 centers declined one or both CCHP BMI screening(s) in 2012–2013.

Out of 14 CCHP + HAP centers that participated in both BMI screenings in 2012–2013, 11 centers completed the initial HAP self-assessment. Of these 11, 10 also received HAP technical assistance materials, and completed a HAP re-assessment in Fall 2013. Thus, 10 out of 19 CCHP + HAP centers received a full set of CCHP + HAP services plus resources, as allocated.

The number of child care centers and children included in the HAP evaluation analysis varied each year, depending on the availability of BMI change data (primary outcome). CCHP BMI screenings were offered to all eligible centers, every year, regardless of whether or not the centers had declined a BMI screening or HAP resources in prior years. Some child care centers that declined one or both BMI screening services in 2012–2013 accepted both BMI screenings in later years.

Table 2 describes the number of children who participated in both Autumn and Spring screenings each year. CCHP + HAP centers served significantly older children than CCHP + HAP Delayed centers in 2011–2012 and 2012–2013. Race-ethnic data were not available. Information about the average number of hours children spent in child care was not available. All CCHP + HAP centers and 22 out of 24 CCHP + HAP Delayed centers offered full-day programs.

Table 3 describes the prevalence of overweight or obese children in the Autumn of each year. Over the 4-year HAP pilot period, the prevalence of overweight or obesity at Autumn enrollment increased significantly, by an estimated mean (SE) 4.3 (0.01) (P = 0.005) percentage points each year. The CCHP + HAP centers had a significantly smaller prevalence of overweight or obesity at Autumn enrollment, compared to CCHP + HAP Delayed centers, in the Baseline Year (2011–2012). In Autumn 2013 and Autumn 2014, CCHP + HAP and CCHP + HAP Delayed centers did not differ significantly with respect to the prevalence of overweight or obesity of children.

In 2013–2014, the ICC for change in BMI percentile between Autumn 2013 and Spring 2014 was 0.04 (0.00–0.09); The ICC for incident overweight or obesity was 0.03 (0.00–0.08). In 2014–2015, the ICCs for change in BMI percentile and incident overweight or obesity were 0.05 (0.00–0.09) and 0.08 (0.01–0.15), respectively.

Table 4 describes the unadjusted mean (SE) annual changes in BMI percentile and z-score for the CCHP + HAP and CCHP + HAP Delayed centers in each academic year. Table 3 describes the incidence of overweight or obesity by intervention group for each year. In the Baseline year, the CCHP + HAP and CCHP + HAP Delayed centers did not differ significantly with respect to the primary outcome, annual mean (SE) change in BMI percentile, BMI z-score, or incidence of overweight or obesity between the Autumn and Spring screenings.

In the 2013–2014 year-specific models, the annual Autumn to Spring changes in BMI percentile and BMI z-score were significantly more negative in CCHP + HAP vs. CCHP + HAP Delayed centers (see Table 5). The incidence did not differ significantly by HAP participation for children ages 2–5 years or children age 3 years (see Table 3). The incidence of overweight or obesity was significantly lower, however, for children age 4 years in CCHP + HAP vs. CCHP + HAP Delayed centers.

In the 2014–2015 year-specific models, the annual changes in BMI percentile and z-score did not differ significantly by HAP assignment, consistent with delayed invitation of CCHP + HAP Delayed centers to the HAP. The incidence of overweight or obesity was significantly lower in CCHP + HAP Delayed vs CCHP + HAP centers (see Table 3).

In multi-year models that evaluated year-to-year changes at the child care center level, allocation to the CCHP + HAP group significantly modified change in the annual mean changes in BMI percentile and BMI z-score from the Baseline year to Follow-up year (see Table 5). Allocation to the CCHP + HAP group did not significantly modify the corresponding difference between the Baseline year and Implementation year 2, consistent with the delayed invitation of CCHP + HAP Delayed centers to the HAP.

In Implementation year 2, the BMI percentile and z-score decreased significantly in both the CCHP + HAP and CCHP + HAP Delayed groups, between Autumn 2014 and Spring 2015 (significant main effects of time, with no significant time x HAP interaction in the year-specific model) (see Table 5). In 2014–2015, the annual changes in BMI percentile and z-score in the CCHP + HAP Delayed group were more negative than they were in 2012–2013 or 2013–2014 (see Table 4). The annual changes in BMI percentile and z-score in the CCHP + HAP Delayed group in 2014–2015 were of similar magnitude to the changes in BMI percentile and z-score observed in 2013–2014 in the CCHP + HAP centers (> −0.2 and −0.1, respectively). The statistically significant main effect of Year with no significant Year x HAP interaction for Implementation year 2 in the multi-year model indicated a significant decrease in child BMI outcomes for both CCHP + HAP and CCHP + HAP Delayed groups relative to the Baseline year.

Before HAP implementation, the incidence of overweight or obesity among 4-year old children in the CCHP + HAP Delayed group, changed in parallel with that in the CCHP + HAP group. In the Follow-up year, the incidence among 4-year old children in CCHP + HAP centers decreased, but remained elevated in the CCHP + HAP Delayed centers, such that the incidence differed significantly between groups. In Implementation year 2, the difference between groups disappeared as the incidence in CCHP + HAP Delayed centers decreased (see Table 3).

For all centers combined, the estimated mean (SE) Autumn to Spring change in BMI percentile was −5.6 (2.5) points lower (P = 0.03) in 2014–2015 compared to the corresponding change in 2011–2012. The mean (SE) change in BMI z-score was −0.2 (0.09) lower (P = 0.02) in 2014–2015 vs. 2011–2012.

In Autumn 2012, before HAP implementation, the CCHP + HAP and CCHP + HAP Delayed centers did not differ significantly with respect to the proportion of children exposed to a physical activity curriculum, staff joining in active play, and/or pitchers of drinking water in the classroom. In Autumn 2012, 9% of children in the CCHP + HAP centers and 12% of children in CCHP + HAP Delayed centers were exposed to 2 or 3 of these best practices.

In Autumn 2013, the proportion of children exposed to 2 or 3 best practices was 60% in CCHP + HAP centers vs 19% of children screened in CCHP + HAP Delayed centers. Children enrolled in CCHP + HAP centers were over 6 times more likely to experience staff joining in active play, a physical activity curriculum, and/or have a pitcher of drinking water in the classroom than children in CCHP + HAP Delayed centers (OR: 6.5, 95%CI: 1.1–40.6). In child care centers that received the HAP invitation materials, but declined to complete the self-assessment or participate in activities, the proportion of children exposed to 2 or 3 best practices increased to 65%.

In Autumn 2014, 60% of children in CCHP + HAP centers and 62% of CCHP + HAP Delayed centers were exposed to 2 or 3 of the index best practices. In CCHP + HAP centers that actively participated in HAP, 85% of children were exposed to 2 or 3 of the index best practices.

Table 6 describes child care center participation in the HAP pilot. All centers that were randomly assigned to the CCHP + HAP group in 2012–2013 received a copy of the HAP self-assessment with invitation to participate in the HAP pilot. Between Spring 2012 and Autumn 2013, 10 CCHP + HAP centers made a total of 210 practice and/or policy improvements. The number of improvements per child care center ranged from 5 to 36. Almost two thirds of the children in child care centers assigned to CCHP + HAP were exposed to the HAP process from self-assessment to re-assessment.

Multiple factors in the background setting or context for the HAP pilot may confound or explain the HAP pilot outcomes. Between 2011 and 2015, several initiatives were concurrent with the HAP pilot in San Francisco. Pre-School-for-All subsidies were distributed to child care centers to offer physical activity classes. A citywide Rethink Your Drink media campaign was launched. The CCHP distributed child-sized pitchers of drinking water to child care providers to encourage children to drink water. Statewide, between 2010 and 2014, the prevalence of obesity among children ages 2 to 5 years enrolled in WIC decreased by −1.7 (−1.8, −1.6) percentage points from 18.4% to 16.6% [18]. California was one of 34 out of 56 States, nationwide, to observe a significant decrease in the prevalence of obesity among WIC enrolled children ages 2 to 5 years during the HAP pilot evaluation period [19]. Statewide and Federal efforts, such as the USDA’s revision of the WIC food package, CDC’s Early Care and Education Childhood Obesity program, and Let’s Move! Initiative, could have contributed to decreases in child obesity during the HAP evaluation period [18, 19]. To limit confounding effects of concurrent initiatives, correlated factors and/or trends, the HAP pilot evaluation used a randomized controlled design. It is unknown, nevertheless, if or what background factors were key conditions for outcomes of the HAP pilot.

Consistent with Statewide and National trends [18, 19], the annual Autumn to Spring changes in child BMI percentile, BMI z-score, and incidence of overweight or obesity in children age 4 years were significantly more negative in both the intervention (CCHP + HAP) and control (CCHP + HAP Delayed) groups in 2013–2014, relative to baseline. The significantly steeper decrease in annual BMI change in the CCHP + HAP group vs CCHP + HAP Delayed in the Follow-up year, and the disappearance of this difference in Implementation year 2, after the CCHP + HAP Delayed group also received HAP resources, suggest a ‘true’ beneficial effect of the local HAP beyond background trends.

Results from one other randomized controlled trial provide support for a ‘true’ HAP effect. Alkon et al. [12] report that implementation of NAP SACC resources at the child care center level result in a significantly more negative mean change in child BMI z-score over 7 months [12]. For children ages 2–5 years, who are becoming more mobile, decreases in BMI are normal; Increases in BMI during this period are considered a red flag for possible risk of overweight [20].

The HAP pilot evaluation and trial by Alkon et al. [12] shared several features in common, beyond implementation of NAP SACC resources at the child care center level. Both trials were restricted to child care centers that serve low income children, who are at higher risk of overweight or obesity than higher income children [21, 22]. Both trials involved time spent by nurses working one-on-one with providers, paid incentives for participation, and workshops about structured physical activity. The HAP pilot paid $25 to child care providers who completed the initial self-assessment. The Alkon et al. [12] intervention paid participants $500 to purchase equipment or supplies for physical activity. Both analyses followed children for BMI change for about 6 months. It is unknown if any or all of the similarities between trials, the social and behavioral risk factor profile of the population served, background promotion of physical activity and drinking water, nurse contact, in-person technical assistance, paid incentives, and/or length of follow-up were key conditions for the observed effects.

Replication of the HAP or NAP SACC effects may depend on specifying conditions similar to those specified by the present evaluation and trial by Alkon et al. [12]. Recent systematic reviews report that current evidence for effective strategies to implement nutrition and physical activity best practices in child care and improve child weight is weak and inconsistent [23]. Between-intervention differences, such as differences in how resources are distributed (online vs in-person by a trusted health professional with technical assistance), may explain inconsistent effects in the literature [23].

Unlike the HAP pilot, which invited all child care centers eligible for CCHP services in one county, the Alkon et al. [12] trial involved child care centers selected from 3 communities in different states, based on racial diversity, staff language, and kitchen availability. Whereas the HAP pilot included data for all children who were screened in the Autumn and Spring, the Alkon et al. [12] trial restricted the child-level analysis to children without chronic illness, conditions that affect nutritional status, severe food allergies, GI disorders, or mobility impairment. The HAP evaluation involved a greater number of child care centers and children. The substantive content of the interventions differed. The Alkon et al. [12] trial addressed the type of milk served, healthy snacks, written nutrition and physical activity policies, and information sheets for parents. Significant BMI change effects in both studies, despite these differences, suggest that the intervention effectiveness is not limited to select communities, child care centers, children or a specific mix of workshops.

Significant effects of intervention at the child care center level on BMI measures, which were collected at the child level, suggest some homogeneity of experience during the school day for children enrolled in the child care centers. Neither the HAP evaluation nor the trial by Alkon et al. [12] controlled for individual child behavioral risk factors over time.

Out of 43 child care centers randomized, 41 centers offered full-day programs. In Autumn 2012, over 90% of the child care centers served by CCHP provided Child and Adult Care Food Program (CACFP) funded meals (CCHP Program data), which have standardized composition for all children, daily, all year long. The intraclass correlation coefficients (ICC) estimated in the HAP pilot indicate greater variance between- vs within-child care centers than is typical in human studies, where the ICCs are usually between 0.01 and 0.02 [24]. If between-center variation in risk factors drives BMI change and incident overweight or obesity in child care centers, then HAP implementation at the child care center level may be relevant and effective for obesity prevention in child care centers.

The HAP pilot linked child care center staff to resources that promote over 80 nutrition and physical activity best practices. The multifactorial approach recognizes obesity as the product of multiple interacting determinants. It provides opportunity for interactive, synergistic efforts, as well as incremental improvements, consistent with child care provider readiness and resources. Implementation of NAP SACC resources is associated with improvements in child care provider practices in several communities, in Arizona, California, Connecticut, North Carolina, Maine, and South Dakota [7‐12, 25‐27]. In the HAP pilot, each of the child care centers allocated to the CCHP + HAP group improved at least 5 practices and/or policies.

Child care centers varied with respect to completion of the HAP self-assessment(s), number of goals set, receipt of technical assistance materials, workshop attendance, and HAP award recognition. Two results suggest that the observed HAP effects were not contingent on exposure to all HAP resources. Child care centers which formally declined to participate in HAP, whose only HAP exposure was receiving the HAP invitation materials, made improvements consistent with HAP. Furthermore, the weight change advantage of CCHP + HAP centers that was observed in 2013–2014 disappeared in 2014–2015, after the CCHP + HAP Delayed centers simply received HAP invitation materials from CCHP staff. Only 3 out of 24 CCHP + HAP Delayed centers returned a completed HAP self-assessment in 2014–2015. None attended a workshop before the Spring 2015 BMI screenings were completed. The results suggest that systematic linkage of child care providers to HAP resources by CCHP staff, by itself, may have be effective for significantly reducing annual child BMI change. It remains to be determined if/how continued and/or greater intensity exposure to HAP resources might magnify benefits on child BMI and obesity prevention.

The HAP added to NAP SACC resources by coordinating self-assessment and improvement processes across providers, and by offering local award recognition as an incentive for practice improvement. Monitoring and aggregating HAP self-assessment data from providers, citywide, makes it possible for the HAP to tailor technical assistance and workshops to local needs and goals. The annual local award recognition raises awareness about nutrition and physical activity standards among child care providers, child care funders, parents, and pediatricians.

The HAP pilot delivered the HAP resources in combination with bi-annual child BMI screening by public health nurses and health workers. It is unlikely, that observed HAP effects were due to child care provider awareness of their center’s overall BMI change results for the previous year. Although annual information about the child care center’s mean Autumn to Spring change in child BMI percentile and z-score gives each child care provider a way to gauge if their efforts to improve nutrition and physical activity practices beneficially impact the children they serve during the academic year, each child care center’s mean child BMI change during the baseline year (2011–2012) was not systematically discussed with child care providers as part of the HAP implementation in 2012–2013. Before 2016–2017, CCHP only reported child- and timepoint-specific results to the child care providers. It remains to be determined if/how annual feedback about child BMI change can enhance the self-assessment process developed by NAP SACC.

Strengths of the HAP pilot evaluation include generalizability of the evaluation population to all children served by child care centers with Preschool-For-All funding in San Francisco. The pilot covered all children enrolled in participating centers, as opposed to a sampled population. The primary outcome, child BMI change, was measured in a standardized way using calibrated instruments, over 4 years, by the same two health workers. Unlike obesity prevalence at Autumn enrollment, which is informative about the past experience of children before they entered care, child BMI change is a sensitive index of conditions when the children are in care, during the academic year.

Integration of the HAP with CCHP was a low-cost intervention. The HAP operation required 1FTE staff person at the local child care agency to coordinate self- assessment and quality improvement processes across child care centers, collect HAP process data, and organize workshops. HAP operation costs were covered through grant funds of less than $100,000 per year. The integration of HAP with CCHP required CCHP staff to spend approximately 16 h more per child care center than routine services. CCHP staff time for the 4-year HAP pilot, including time for public health nurse outreach and measured BMI change on almost 1000 children, annually, was provided in-kind by the public health department.

Each year, bi-annual BMI screening occurs systematically in child care centers across the United States. Head Start child care centers, for example, are mandated to screen child BMI at Autumn enrollment and again in the Spring. If addition of HAP support to health screening services significantly improves child BMI in San Francisco, similar linkage of child care providers to NAP SACC resources through child care health screening programs might be possible, inexpensive, and effective, nationwide.

The HAP pilot had several limitations. Randomization may not have eliminated bias related to treatment assignment, due to the relatively small number of child care centers and uneven demographic distribution across child care centers. Randomization would not have controlled for time-varying factors that happened to correlate with HAP exposure, and independently predict changes in child BMI. The health workers who collected the outcome data were not blinded to the intervention allocation. The income, race-ethnicity, and social determinants of health of the children were unknown, and not controlled in the design or analysis. Information was not available about child care centers and children that chose not to participate in both annual BMI screenings.

Potential for measurement bias also limits interpretation of the results in several ways. CCHP routine services did not include tracking of all NAP SACC components, in all child care centers, over time. CCHP providers reported difficulty understanding some of the HAP self-assessment questions. The best practice measures used in this evaluation were subject to variation in provider interpretation. Due to lack of unique child identifiers across child care centers, the analysis did not track the BMI changes of the same children across years. Child care providers reported barriers to improvement that were beyond their control, such as neighborhood crime, which were beyond the scope of the HAP to solve. This study tested a combined exposure, the HAP combined with linkage of child care providers to the HAP by public health staff. It did not evaluate the effect of the HAP alone.