Is the online ‘creating healthy eating and active environments survey' (CHEERS) tool reliable for early childhood educators in Alberta, Canada: a randomized crossover trial

The early years are a critical period in child development where the origins of lifestyle health behaviours can be identified. Patterns of eating and activity, as early as one to two years of age, have been demonstrated to track through childhood and into adulthood [1‐6]. Research suggests that Canadian preschool-aged children consume less than one-third of the vegetable and fruit intake recommended combined with low levels of physical activity [7‐10]. It is estimated that close to 60% of Canadian children 0–5 years participate in some type of childcare arrangement with over half of these attending centre-based care [11]. This demonstrates that early childhood education and care (ECEC) environments represent rich opportunities for interventions that promote healthy eating and reduced sedentary behaviours aimed at preventing childhood obesity and the reduction of the long-term non-communicable chronic disease [12‐14].

Supporting child health and well-being is a public health priority and is an essential global health initiative. The nurturing care framework is an organizing principal that structures key components of high quality care in ECEC environments that is composed of five elements: health, nutrition, security and safety, responsive caregiving, and early learning opportunities [15]. In order to provide this high quality care, educators and public health professionals require access to valid and reliable tools. The creating healthy eating and active environments survey (CHEERS) tool is a community-based, educator-administered audit tool designed to offer ECEC centres an evaluative measure to assess their centres healthy eating and physical activity environment. The constructs measured include the quality of food served, the environment where children are eating, the curriculum connections for nutrition education, and the physical activity opportunities provided. This will also be a valuable tool for public health providers and researchers that can be used to measure the environmental context in a child care setting. The CHEERS tool can facilitate and empower ECEC centres to enhance nutrition and activity environments that support child health. While currently aligned with Canadian best practices, the tool would be adaptable to many international childcare contexts.

The tool has been assessed for reliability and validity with early childhood experts and educators. The CHEERS tool was content validated following a structured process involving a panel of ten nutrition, physical activity, early childhood, and public health professionals with a readability of grade 8.1 using Flesch–Kincaid [16] with evidence of good inter-rater reliability, intra-rater reliability, internal consistency, and concurrent validity [17].

The CHEERS tool was developed as a paper instrument that could be completed by an educator and returned by ground mail for evaluation. Paper-based questionnaires are human resource intensive and introduce the possibility of data omission and errors in data transcription to digital format. On the other hand, there are disadvantages of online-based questionnaires such as inaccurate responses due to misinterpretation of questions and low response rates. The internet and its applications are a continuous area of innovation and an exponentially increasing component of every day life. Electronic data collection is becoming more extensively used in healthcare research that bring with it many advantages such as access and service provision for remote communities, facilitated data entry time, reduced processing cost, and faster survey response return to stakeholder [18, 19]. Moreover, a key message from the early childhood community is the need for reciprocation. While data is collected from the ECEC centre, how can the centre benefit from completing the questionnaire? Educator awareness of policies and practices that support or impair healthy environments for children have been identified as a method of informing or reaffirming health promoting professional practice activities [20].  An online-based survey can increase response time and support just-in-time learning for educators.

An electronic version of the CHEERS tool was developed to align with increasing use of electronic public health outreach and resources provided through website in addition to the desire of educators to access the tool in an online context. While reliability and validity of the CHEERS tool have been demonstrated, it is not known if administration using an online-based interface is consistent with paper and pencil versions. The aim of this study was to assess the reliability and utility of an electronic version of the CHEERS tool.

This study was conducted in Alberta, Canada throughout provincial health zones. Ethical approval to conduct the study was obtained from the Mount Royal University Human Research Ethics Board (no. 100016). Participants were fully informed about the purpose and procedures of the study and provided written consent for paper first participation or digital consent for electronic first participation. No minors were involved in this study.

The results of this study demonstrate that the online version of the CHEERS tool presents good reliability, agreement, and utility to its paper version. Previous research demonstrates that the paper-based, educator-administered CHEERS survey is a reliable and valid audit tool for evaluating child care centre eating and activity environments [16, 17]. Electronic health (eHealth) and mobile health (mHealth) innovations offer new avenues for health promotion [37]. Online methods of health assessment and interventions have the advantages of convenience, ease of operation, adaptability, and accessibility. However, evidence of online reliability, validity, and effectiveness are an important component of building comprehensive evidence for the eHealth initiative [38].

The test–retest reliability of the online-based CHEERS overall score was strongly aligned with the paper-based administration (ICC = 0.91).

These results align well with previously reported paper-based reliability data for the overall score (ICC = 0.81) [17]. Psychometric properties of “reliability and agreement are estimates that vary based on interactions between a tool, its user, and the context of the assessment” [39]. Considering that the current study was done with a new cohort of early childhood educators who had not previously participated and similar intra-rater results were found contributes to the overall psychometric evidence demonstrating reliability of the CHEERS audit tool.

The outcomes of this study are in accordance with outcomes in similar paper to online validation studies of child related health tools. For example, the NutriSTEP® questionnaire screens nutrition risk in preschoolers through a survey completed by primary caregiver/parents. An ICC of 0.91 for CHEERS overall score is in line with the ICC of 0.91 found in the NutriSTEP® adaptation from paper-based to online-based administration [40]. There was similar alignment of ICC scores for the MiniPAQLQ, a quality of life questionnaire completed by caregiver’s of children with asthma, ICC of 0.89 [41]. This is further supported by the results of a meta-analysis suggesting that computer and paper-based questionnaire administrations result in equivalent scores when comparing electronic and paper-based administrations [42].

The standard error of measurement (SEM) is a measure of stability, reflecting the precision of individual scores expressed in the same units as the original scores [23, 30]. While ICC reflects reliability among people, SEM quantifies precision within individuals and permits the calculation of the minimal detectable difference that can reliably meaningful difference to measure change that can be used in future research to determine if an intervention has a significant impact on creating change. The current study has provided evidence of stability as demonstrated by the SEM, MDC₉₅, and MDC₉₅% from paper-based to online-based administration.

Tool reliability and validity are important aspects of health measurement scales. However, a measure must also have utility and ease of use for respondents thus an important component of psychometric tool assessment is the usability, functionality, and availability of the tool [43, 44]. The literature clearly indicates a preference for digital over paper questionnaire completion [40, 41, 45]. eHealth technologies can provide improved efficiency and accessibility for health promotion services for stakeholders. This study adds to the reliability and utility of the CHEERS tool through the addition of a user-friendly electronic version.

Potential limitations exist with this research. First, all participants in this study were women which brings into question the representativeness of the study sample. However, the fact that the ECE workforce is comprised primarily of women, 96% of all ECEs in Canada are women [46], this sampling does reflect the population of users intended for this tool. Second, findings may also be limited because this study used a convenience sample. While a random sampling frame would be a better approach, the early childhood educator profession in Canada does not have a central professional community where all members are listed. As a result, there is no mechanism from which to randomly select participants. The provincial government website has a listing of all licensed centres throughout the province by region so emails to centres provide the best option to contact a variety of educators in different cities and towns of various sizes. Thirdly, there is a possibility that participants remembered their responses from the first administration. A recommended interval between administrations appropriate for reliability studies assessing the development of health measurement scales is approximately 2 weeks [23]. Sufficiently short to ensure minimal opportunity for real behaviour change while sufficiently long enough to minimize recall and bias. However, some individuals may have remembered their responses and which may have allowed them to report similar answers in the second administration. Lastly, a limitation is the dropout rate. Thirty-six participants were assigned to arms of the crossover study, however, approximately ten were lost to follow-up resulting in a 27% dropout rate which within the 30% dropout rate [23]. This loss may have been due to characteristics of the survey (time demand) or high job turnover in the early childhood educator workforce [47, 48]. Five paper surveys were lost to lack of postal delivery (lost mail) and two lost due to incomplete data (missed backside of paper-survey). However, the reliability measures of the CHEERS adaptation from paper-based to electronic-based administration were similar the NutriSTEP® and MiniPAQLQ paper-to-electronic adaptations within a similar population and it is reasonable to assume the missing data would not have meaningfully impacted the study outcome.

This study provides evidence of good test-rest reliability of the online-based CHEERS tool relative to the paper-based version. Health interventions are increasingly being made available via multiple platforms (computer or mobile versions) and demonstrating that these digital options reliable improves the reach and immediacy of support for community-based clients through novel technology products. eHealth administration of the CHEERS tool can be used in public health initiatives within community settings and the outcomes can be interpreted in accordance with findings of the paper version of the tool. Future opportunities would be to investigate immediate feedback of the survey to respondents as well as educational outreach.