Measuring early childhood development in multiple contexts: the internal factor structure and reliability of the early Human Capability Index in seven low and middle income countries

Ratified by United Nations member countries in 2015, the Sustainable Development Agenda specifies 17 goals and 169 targets to end poverty, mitigate inequality, and protect the planet for a better future [4]. The fourth year of the SDG era calls for countries to continue to invest not only in interventions and policies that will promote global equity and sustainability, but also in the monitoring systems required to track progress against these targets and thus identify those at risk of falling behind. Of particular relevance to early childhood development, SDG 4.2 states that by 2030, countries must ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education. To track progress against this target, countries are required to monitor (i) the percentage of children under 5 years of age who are developmentally on track in health, learning and psychosocial wellbeing, and (ii) national participation rates in early childhood education.

While many countries monitor national enrolment rates in early childhood education [5], few track the status of children’s early development. Measuring progress toward SDG 4.2 calls for population monitoring of children’s early health and development outcomes. Indeed, child development contributes to a number of SDG targets, including those related to health, gender equity, and poverty reduction, and thus global monitoring of children’s early development is key to supporting progress toward the broader Sustainable Development Agenda [6]. Faced with limitations in terms of the measurement instruments available as well as the resources and capacity required to implement such monitoring systems, tracking children’s health and development in low and middle income countries will be a challenge. In this, however, lies an important opportunity to promote and address the current obstacles associated with measuring children’s early development.

Measurement of children’s development is influenced by culture, language, and theory. What are considered important aspects of and appropriate goals for children’s development, as well as what are deemed suitable assessment techniques to capture this information, can vary considerably across cultures and contexts [7‐9]. Consequently, although tools need to capture aspects of child development that are important to outcomes throughout the life course, they should also be aligned with local culture and early learning and development frameworks, so that they not only accurately reflect children’s capabilities, but also produce information relevant to local policy and practice [1, 8].

A number of measurement initiatives are currently underway to monitor children’s early development at national, regional, and global levels. Some examples include the Early Development Instrument (EDI), the Early Childhood Development Index (ECDI), the Caregiver Reported Early Development Instrument (CREDI), the International Development and Early Learning Assessment (IDELA), the East Asia-Pacific Early Chid Development Scales (EAP-ECDS), the Malawi Developmental Assessment Tool (MDAT), the Measurement of Development and Early Learning (MODEL), and the Regional Project on Child Development Indicators (PRIDI) [3]. Various characteristics of the instruments in use however, including the cost of licensing fees, the level of enumerator training required prior to administration, the time they take to administer, how they are administered, and their applicability and adaptability within different contexts, constitute considerable barriers to their utilization. This is especially the case in contexts where resources and capacity are limited. To overcome these challenges, international leaders in early childhood have called for a more pragmatic and reliable solution to measuring children’s early development in low and middle income countries in particular. It was against this background that the eHCI was developed.

Designed to capture key aspects of holistic development in children aged 3–5 years, the eHCI was developed with the vision of being feasible for use in low resource and capacity settings while having the ability to capture change in children’s development over time [10]. The tool includes approximately 60 items (dependent upon country adaptation) spanning nine developmental domains (Physical Health, Verbal Communication, Cultural Knowledge, Social and Emotional Skills, Perseverance, Approaches to Learning, Numeracy, Reading, and Writing) and can be completed via adult report (e.g. by children’s caregivers, teachers, or early childhood practitioners) in less than 10 min. The eHCI requires minimal resources to be implemented; the tool is available for anyone to use free of charge, little enumerator training is required, and it can be completed quickly and easily by any adult who knows the child. Further, the tool was designed so that it can be easily adapted and utilized within diverse contexts for a range of purposes, including population monitoring, the evaluation of effects of early childhood policies and programs, as well as longitudinal studies seeking to predict children’s future capabilities.

The eHCI was originally developed in 2013 for the purposes of evaluating a program designed to support children and families to be better prepared for school in Tonga [11]. Consultations were undertaken to understand locally, what good child development at school entry looks like, first establishing broad areas (i.e. domains) of development and then identifying specific items within these areas. On the basis of consultations and the child development literature, a draft instrument was developed and independently reviewed by child development experts. Once translated into Tongan, stakeholders originally consulted reviewed the instrument to ensure content and face validity. Piloting was then conducted to determine respondent understanding of items, efficiency of data collection methods, if scale distributions were discriminating between children as theoretically expected, as well as any floor or ceiling effects (i.e. if items were too hard or too easy) for the targeted range of children aged 3–5 years. After revisions based on findings from the pilot as well as a final review by local stakeholders, the eHCI was implemented nationally. Exploration of the psychometric properties of scores produced by the eHCI census in Tonga demonstrated adequate discriminant validity (comparison of mean scores across children grouped by demographic characteristics met theoretical expectations, i.e. older children scored higher than younger children, girls received slightly higher scores than boys, children of more educated mothers received higher scores than those of less educated mothers) and internal scale reliability (through Rasch analysis) [10].

The eHCI has since been adapted and utilized to support a range of early childhood development projects in several low and middle income countries, predominantly across the Asia-Pacific region [12‐17]. Similar to a number of tools designed to measure children’s development [18‐20], development and adaptations of the eHCI in each new context were informed by a combination of both theoretical conceptualization as well as local expert consultation regarding the key aspects of children’s development that are predictive of future capabilities. Through these consultative processes, content and face validity of the instrument were established and adaptations and translations were ensured to be capturing the true intent of each item [21]. The internal factor structure of the instrument, however, is yet to be explored within multiple countries.

Evaluating the psychometric properties of scores produced by a measurement tool is fundamental for its future utilization and effectiveness [22]. An instrument measuring children’s development that lacks in reliability and validity could produce biased scores that lead to ill-informed decisions. With eHCI data now available in multiple countries, work is needed to explore the tool’s validity and reliability. An instrument with the properties of the eHCI that produces scores that are psychometrically robust and appropriate for use within diverse settings has potential for global applicability. Indeed, such a tool could better enable population monitoring of children’s development, as is required for SDG 4.2, particularly in low and middle income countries, with the ultimate goal of shaping services and policy to promote global equity of children’s health and development.

This research is a first step in working to establish the psychometric properties of scores produced by the eHCI within different cultures and contexts. Utilizing data previously collected from seven low and middle income countries, Brazil, China, Kiribati, Lao People’s Democratic Republic (PDR), Samoa, Tonga, and Tuvalu, analyses sought to explore the internal factor structure and reliability of scores produced by the tool within each country. Findings will be used to guide recommendations regarding the reporting of eHCI results moving forward.

Data included in this manuscript were collected from seven countries between 2013 and 17, utilizing different sampling techniques and data collection methods. Contextual information regarding data collection procedures in each country are summarized in Table 2.

Characteristics of each country sample are presented in Table 3. Samples ranged in size from 549 children in Tuvalu to 12,191 in Samoa, with children ranging in age from 2 to 6 years. Though the eHCI was designed to capture the development of children aged 3–5 years, the tool has also been used to collect data on children who fall slightly outside of this age range. This is a result of varied data collection purposes across countries. For instance, in Lao PDR, 2 year olds were included in data collection as this dataset serves as the baseline measure for a randomized control trial; younger children needed to be included at baseline to ensure they also fall into midline and endline data collections in years to come. Each country sample had a relatively even split of males and females; maternal education ranged from the majority of children with mothers who had never attended school (30.2%), started (27.3%), or finished primary school in Lao PDR (29.1%), to the majority of children with mothers who had completed secondary school (42.2%) or tertiary studies in Tonga (17.8%); while the percentage of children who had attended preschool ranged from 23.2% in Lao PDR to 100.0% in Brazil.

First, confirmatory factor analyses (CFAs) were conducted separately for each country to determine the fit of eHCI data to the theoretical structure of the instrument (i.e. nine developmental domains, or eight domains in the case of Lao PDR). Next, CFAs were conducted separately for children aged 3, 4, and 5 years old in each country (as this was the age range consistent across all countries) to explore any variation in fit based on children’s age. Children with missing age data were excluded from this analysis (Brazil n = 2, China n = 56, Kiribati n = 884, Lao PDR n = 852, Tonga n = 53, Tuvalu n = 3). Additional CFAs were conducted for the Lao PDR sample stratified by maternal education, to explore if fit of data to the theoretical structure of the eHCI varied by respondent’s level of education. Specifically, the sample was split into two groups: low maternal education (i.e. no school, started primary, finished primary) and high maternal education (i.e. finished secondary, tertiary) and CFAs were conducted separately for each group. Children with missing maternal education data were excluded from this analysis (n = 5). This analysis was conducted for the Lao PDR sample only, as it was the sole sample for which data were available on the education level of all respondents.

Goodness-of-fit indices including χ² (p > 0.05 indicates good fit; Brown, 2006), root mean square error of approximation (RMSEA; values ≤0.06 indicate good model fit while those between 0.06 and 0.08 indicate satisfactory fit; Hu & Bentler, [24]; Yu, [25]), the Comparative Fit Index and the Tucker-Lewis Fit Index (CFI and TLI, respectively; values ≥0.95 indicate good fit while values between 0.90 and 0.95 indicate satisfactory fit; Hu & Bentler, [24]; Yu, [25]; Schreiber, Nora, Stage, Barlow, & King, [26]), as well as standardized factor loadings (≥ 0.4 considered high and thus deemed a good indicator of the underlying construct; Costello & Osborne, [27]) were used to evaluate model fit. CFAs were conducted in Mplus [28] utilizing polychoric correlation matrices and the weighted least squares mean and variance adjusted (WLSMV) estimation method, both of which are deemed most appropriate for use with binary-type data such as that of the eHCI [29‐31], as well as oblique (geomin) factor rotation which assumes correlations amongst factors [31].

The internal reliability of eHCI domains was also examined for each country, which is often conducted in conjunction with factor analysis to measure how interrelated a set of items are and thus how well they, collectively, measure the underlying construct of focus [32]. Although the majority of similar research assesses internal reliability using Cronbach’s alpha [19, 20], increasingly, ordinal reliability coefficients, specifically ordinal alpha, are deemed to be more appropriate in the case of evaluating the internal reliability of scales including items with binary response options in particular [33]. As such, ordinal alpha coefficients were calculated as well as Cronbach’s alpha (≥ 0.70 deemed acceptable for both coefficients; Bland & Altman, [34]; Gadermann et al., [33]) to allow for comparison with previous research. These analyses were conducted using the package ‘psych’ in R-Studio [35].

Samples utilized in this research differed considerably across countries in terms of children’s demographic backgrounds, data collection methods and purposes, as well as sampling techniques and sizes. Although it might be argued that such differences present a challenge to exploring and comparing the validity and reliability of the eHCI within multiple contexts, this is the pragmatic nature in which the instrument was intended to be used; for a range of purposes and across varied contexts. As Yapa and Bärnighausen [37] discuss, the resource constraints that come with research in low and middle income countries are often the driving force behind creative solutions. As such, we argue that there is strength in that the eHCI was found to demonstrate a common underlying factor structure within the varied contexts in which the instrument has been implemented.

Numeracy, Reading, and Writing domains in particular were found to be working consistently across countries. Items that form these domains had high factor loadings and these scales had high internal reliability across countries. Similar results have been reported for other measures of children’s development, for instance, factor analyses of domains that constitute the EDI, a teacher-completed checklist measuring children’s holistic development in their first year of school, have demonstrated the Language and Cognitive Development domain (which captures children’s literacy and numeracy skills) to have the best fit across multiple countries [38]. Examination of items that form Numeracy, Reading, and Writing domains in the eHCI highlight that little adaptation of these items was required across countries. For instance, the Numeracy domain covers the same concepts of shape, colour, and number recognition, counting ability, and knowledge of numerical concepts such as time and weight, across countries. This might suggest these domains to be the more universal aspects of children’s development, indeed such skills have been demonstrated to be important predictors of outcomes throughout the life course [39, 40], and thus results are consistently strong across countries. Such skills are also arguably more easily observable (i.e. it is likely that a caregiver or teacher knows if a child can read or count, as opposed to whether they know if a child is always wanting to learn new things as measured in the Approaches to Learning domain, or if a child knows good from bad foods as captured in the Physical Health domain), which may also have had an influence on results.

In contrast, results across Physical Health, Verbal Communication, Cultural Knowledge, Social and Emotional Skills, Perseverance, and Approaches to Learning domains tended to demonstrate more variation across country samples. This is unsurprising considering the nature of the skills measured by these domains, including health and hygiene practices, verbal communication abilities, knowledge of culture and culturally acceptable behaviours, social interactions and emotional regulation, as well as how tasks are approached and the ability to complete them, which are aspects of development that would be considered to be more contextually and culturally specific. To illustrate, although Social and Emotional Skills was demonstrated to be one of the distinct domains that the eHCI captures within each country, variation in the strength of factor loadings of items that form this domain between Brazil and Lao PDR might be, in part, explained by cultural differences in social interactions and the expression of emotions between the two countries. Item factor loadings were lower in Lao PDR (on average around 0.65–0.70), an individualistic culture in which emotion is perceived to be experienced internally within an individual, whereas the same item loadings were higher in Brazil (on average around 0.80–0.85), a collectivist culture whereby emotions are thought to occur between people and thus are expressed openly [41]. These results could also be attributed to variation in methodological bias across countries [42]. For instance, acquiescence, the tendency to agree with statements, has been demonstrated to be more common in collectivist cultures [43]. Variation of results across countries in this way highlights the important influence of culture on the measurement of children’s development, and the need for tools to capture not only the aspects of children’s development that are important predictors of later outcomes, but to also be aligned with local culture in order to produce information that both accurately reflects children’s abilities and is relevant to local policy and practice [1, 8].

Reverse-scored items in the Physical Health, Social and Emotional, and Perseverance domains had weak factor loadings in all countries but Brazil, indicating that, compared to other items, they are poorer measures of the underlying constructs being measured by these domains. These results reflect initial findings from analyses conducted in Tonga following the development of the tool, with Rasch analyses indicating that a number of reverse-scored items did not fit the model well relative to other items [10]. It is possible that enumerators and/or respondents had difficulty in understanding and/or responding to these negatively worded items. Previous research has shown that reverse-scored items tend to load onto a factor separate to the construct they are intended to measure, that instead reflects aspects of item method [44, 45]. This was not observed in Brazil however, and this could be due to sample differences in this context relative to other countries. Specifically, Brazil was the only sample for which only preschool teachers completed the eHCI, as opposed to a combination of children’s caregivers and teachers, or caregivers only as in other samples. It might be that children’s caregivers and teachers do not respond to the eHCI in the same way, or that a minimum level of education or literacy is required to understand and respond to items. Indeed, results demonstrated better fit of data to the theoretical structure of the eHCI in Lao PDR amongst more educated caregivers who responded to the tool, compared to those less-educated. However, weak factor loadings for reverse-scored items were maintained when analyses were run separately for caregivers who had low versus high education. Together, results raise important questions regarding respondent reliability that need to be explored by future research.

When considering children’s age in determining how the instrument is operating, results were inconsistent across countries (i.e. in some countries best fit was observed for 3-year-olds and in others for 5-year-olds), and with the exception of Brazil, model fit indices did not vary in magnitude greatly, indicating that the eHCI appears to work relatively consistently across the age range of 3–5 years. Although these results provide some insight into how effective the eHCI is in measuring development across children of different ages, analyses focused on the discriminant validity of the tool, including if items capture a continuum of development for children of different ages, are needed to further explore this question.

Finally, internal reliability results for Physical Health and Perseverance domains indicated that items that form these domains, collectively, are not a good measure of these underlying constructs. It is possible that this is a result of reverse-scored items (which could be measuring constructs separate to that intended) making up a large proportion of these domains (i.e. one in two items in the Perseverance domain are reverse-scored across countries, while for the Physical Health domain this is the case for one in two items in Brazil, one in three items in Samoa and China, one in four items in Tonga, and one in five items in Tuvalu and Kiribati). Nevertheless, local adaptation of the tool in each country deemed all items important to children’s early development in their contexts, and thus it was not the intention of the current research to exclude items on the basis of psychometric results. An example is the first item in the Physical Health domain regarding children being frequently sick. Although we would not naturally assume that this item measures a child’s skills or capabilities (and subsequently it does not work well in the model), in the contexts of countries of focus whereby illness is common, it was deemed important for the eHCI to provide information regarding children’s experience of illness as one aspect of their holistic development.

When interpreting results of the study, it is important to be cognizant of three limitations. First, although the majority of countries studied utilized a census approach to data collection and thus are considered nationally representative, sampling strategies employed in Brazil, China, and Lao PDR (see Table 2) posit that results may differ if eHCI data were to be collected on nationally representative samples in these countries. Next, results indicate that reverse-scored items may not be operating as intended. Beyond analyses presented in this study, however, the information required to be able to explore this further (for example, insight into respondents’ understanding of reverse-scored items) is not currently available. Finally, it is important to reiterate that demonstrating consistent internal factor structure and reliability, as has been done in this study, is not complete evidence of a valid tool and must be considered together with results from additional psychometric analyses.

Relative to other measures of early childhood development currently utilized, the eHCI requires minimal resources to be implemented. Initial psychometric results suggest that this has not come at the cost of the validity and reliability of the instrument. Demonstrating a consistent internal factor structure and reliability is one important aspect of the comprehensive evaluation of an instrument’s validity and reliability. Although not within the scope of the current study, additional work is underway to explore the extent to which eHCI domains can discriminate amongst children’s abilities by a range of demographic and contextual variables, are associated with scores on other measures of child development, show reliability amongst respondents, and are able to predict children’s future outcomes. A low-burden instrument that is both easily adaptable and psychometrically robust within multiple contexts in this way has potential utility internationally. Indeed, such a tool might better enable population monitoring of children’s development, as is required for the tracking of progress towards the Sustainable Development Agenda, particularly in low and middle income countries.

In terms of the reporting of and utilization of data produced by the eHCI, results suggest that eHCI data should continue to be reported across the instrument’s 9 theoretically-based developmental domains, or 8 domains in the case of Lao PDR. Reporting of children’s development across different areas of development in this way enables the identification of areas of both strength and need, and as a result can help to shape more targeted approaches to intervention or policy development. SDG 4.2 however, calls for the monitoring of children who are “developmentally on track”, a concept that, as with children’s development more broadly, is likely to vary across contexts. As such, if the eHCI is to be recommended to track progress against SDG 4.2 in future, research needs to not only work to further validate the instrument, but also determine how “developmentally on track” might be classified utilizing eHCI data.