Psychometric evaluation of the computerized battery for neuropsychological evaluation of children (BENCI) among school aged children in the context of HIV in an urban Kenyan setting

A total of 604 (311 females, 291 males and two with missing gender information) children from Nairobi participated in the study. Nairobi is the capital city of Kenya with a 87.1% literacy level and the language of instruction in the schools is English [25]. We recruited two samples from different study sites. One group of children was sampled from a children HIV outpatient programme. The programme, implemented in seven resource poor settings in Nairobi, included children living with HIV of different ethnic backgrounds who receive home-based care. The sample of children not living with HIV was drawn from three primary schools in Nairobi. The schools were chosen on the basis of their similarity to most schools in Kenya with regards to the mode of education at that time which was the 8.4.4 system with the examining body under the Ministry of Education being the Kenya National Examination Council [26]. These children come from diverse socio-economic settings with most of them from middle-class families. We chose this to rule out the impact of sharp socioeconomic status differences. The study sample size computation was based on data from an earlier study in Africa that found the means on the KABC – 2 to differ between HIV-infected (N = 93) and uninfected (N = 106) [27] by \({\mu }_{1}=184.7 \left(sd=63.72\right)\mathrm{ and}{ \mu }_{2}=200.6 \left(sd=68.72\right)\), respectively, yielding a Cohen’s d of 16.1/66.3 = 0.24. Together with an alpha level of 5% and a power of 80%, these resulted in a total sample size of 544 respondents, thus the target sample size was 272 children living without HIV and 272 children living with HIV, respectively. We slightly oversampled to address any potential loss of data due to missingness.

The BENCI: The existing BENCI test was first developed in Ecuador and offers norms for children aged 6 – 17 years in Ecuador, 7, 9 and 11 years in Morocco and 6—8 years in Palestine [18, 19]. The test can be administered within 75 min with one 10-min break in between the 14 neuropsychological tests. On average, however, the administration takes around 90 min. The test can be administered by skilled psychologists with additional training specific to BENCI.

Paper and Pencil Measures: To test convergent validity of the BENCI, we used paper and pencil tests that are internationally accepted and standardized and have previously been adapted and validated in a rural Kenyan community [28]. This so-called Kilifi Toolkit covers executive functioning, memory, and attention and can be administered within 120 min. The neurocognitive tests have good psychometric properties with split- half reliability between 0.70 and 0.84 while internal consistency is ≥ 0.70 among 7 – 11 year old children in Kenya [28]. Table 1 lists tests in Kilifi toolkit and the BENCI. As part of our study, we also measured age, gender, height, and weight.

In the clinics, a database of children aged 6 – 14 years old was generated and the children were informed to come to the clinic on a certain day of the week when the programme arranged for some fun activities to take place. Most of the time the assessment day fell on a weekend and on the same day as the children were scheduled for their clinical appointments. On the scheduled day, the children and their parents were randomly identified and individually informed about the study with voluntary participation of the children being requested. We included children aged 6–14 years¹ that are HIV-positive and not having any comorbid conditions as reported in their medical reports. We did not include children with comorbid and/or severe medical conditions associated with being HIV-positive as indicated in their medical reports, as well as children who did not meet the age criteria. In the school setting, the children were randomly selected from their classrooms, which ranged from Grades 1 to 5. In this population, we included children aged between 6 – 14 years old and not having any medical condition as reported by the school and the students themselves. Children who did not meet these conditions were excluded from the study. The institutions provided a room where the neurocognitive assessments could be carried out. Relevant subtests in Kilifi toolkit (see Table 1) were administered with paper and pencil by a trained interviewer [28]. For test–retest reliability, 38 HIV negative children (21 females) were re-assessed 2 months after the initial assessment.

Data from the BENCI was automatically captured in the tablet as programmed in the original Spanish version and exported to an Excel sheet. The Kilifi Toolkit data were input into Excel sheets and codes/matching identifications were realigned to ensure correct matching with similar cases in the file with BENCI data. We double-checked the age, gender, and clinic/school groupings to ensure the correct ID matching. Analyses were run in SPSS version 20 and AMOS version 22. We used Alpha = 0.05 as the nominal significance level.

Data were cleaned by first having a visual inspection of a scatter plot and statistical evaluation of each of the subtest scores for outliers. Data with influential outliers were then evaluated through a three-step process to identify if certain scores should be deleted. First, we checked the residuals of the regression of age on the subtests where cases with high standardized residual value, low effect size, and low p-value were noted. Second, we evaluated cases with z scores beyond z =|2| for possible deletion. Third, we conducted a case-by-case check to evaluate whether a certain score would be expected given other subtest scores from these participants. For instance, we discarded scores on the Verbal Memory Immediate hits and Continuous Performance hits subtest that had z-scores below -3 and whose z- scores were not expected for the age groups we were looking at. Through this process, we decided whether certain scores should remain as they are or identify them as missing. We then carried out a missing data pattern analysis where Little's MCAR test statistic was significant (χ² = 2455.2, DF = 1725, p < 0.001), highlighting that scores were not missing completely at random. However, a check on whether the missingness was significantly related to age, HIV status, and date of data collection, uncovered no significant relationship with the missingness pattern in subtest scores. We could, therefore, not identify what the missingness was related to.

Internal consistency in terms of Cronbach’s Alpha (KR-20 for dichotomous items) was determined for all seven tests for which item-level data were available. We opted for Cronbach’s Alpha because it is widely used in testing the internal consistency of the items within a test that reflects the degree to which items covary positively. The test–retest reliability was analyzed using ICC and Pearson’s correlations. We then checked whether the performance within the BENCI subtests aligned with developmental models’ expectation of growth in cognitive performance as children grew in age. Convergent validity was analyzed using Pearson correlation where scores of the BENCI subtests were correlated with the raw scores of corresponding subtests in Kilifi Toolkit. We hypothesized that tests measuring the same cognitive domain would correlate positively. We compared differences between HIV-positive and HIV-negative groups with t-tests and considered possible floor and ceiling effects by checking histograms and outliers by calculating skewness for each subtest.

We run a confirmatory factor analysis in AMOS to assess the construct validity the BENCI using a model of Executive Function proposed by Diamond, in which executive function comprises reasoning, inhibition, flexibility, fluency, and working memory cognitive functions [30]. The model fit was evaluated with the Chi-square tests, Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and the Tucker-Lewis Index (TLI). A model is considered a good fit if the value of RMSEA is below 0.06, and CFI and TLI above 0.90 respectively.

Using correlational and descriptive statistics including histograms, we evaluated attenuation patterns in the BENCI tests. Eight of the BENCI subtests exhibited ceiling and floor effects that tend to suppress correlations and reliabilities. Specifically, on Verbal Comprehension Figures, 30% (N = 181) of the sample scored the highest possible score of 8 hits, while on Verbal Comprehension Images hits, 51% (N = 308) of the sample scored the highest possible score of 8 hits. Other subtests with ceiling effects included Continuous Performance hits, Go No Go hits, Working Memory hits, and Spatial Stroop. Both Verbal Memory Recognition 13.4% (N = 16) and Visual Memory Recognition 16.7% (N = 20) showed some ceiling effects meaning that the number of participants having the highest scores was almost equal to those with average scores. At the same time, floor effects were evident on the Planning Time of First Option and Spatial Stroop errors scores. Semantic Fluency hits 13.4% (N = 16), Phonetic Fluency hits 16% (N = 19), Verbal Memory Delayed hits 16.8% (N = 20), and Planning time total 33.9% (N = 38) showed some floor effects. This meant that the number of participants having the lowest scores was almost as equal to those with average scores. The floor and ceiling effects highlighted that these subtests psychometric functioning could be improved by adding easier and more difficult items, respectively, in any future revisions of the BENCI. The remaining BENCI subtests showed no such attenuation effects.

We computed the Cronbach’s Alphas (KR-20 s) for seven of the subtests with dichotomous item scores. The internal consistency of the BENCI subtests varied from poor to excellent reliability. As shown in Table 3, the Language Comprehension tests, Verbal Comprehension Images, and Figures, had the fewest items (N = 8) and Cronbach Alpha 0.49 < α < 0.68 which was the lowest among the other BENCI subtests. Low Cronbach Alphas tend to suppress correlations, but most of the BENCI subtests had high Alphas. The Abstract reasoning, Planning, Go No Go, Spatial Stroop, and Processing Speed tests correlated well with themselves (0.75 < α < 0.97 or alpha range from 0.75 to 0.97) hence showing that there was little random measurement error.

Possibly due to the ceiling effects being less severe because of lower mean scores, we found Verbal Comprehension Figures and Images tests to show higher internal consistencies among children living with HIV (0.57 < α < 0.68) than among children not living with HIV (0.35 < α < 0.56), whose scores were more affected by the ceiling effect. In the Abstract reasoning, Planning, Go No Go, Spatial Stroop, and Processing Speed sub-tests the items had acceptable and excellent (0.76 < α < 0.97, or alpha range from 0.76 to 0.97) internal consistency showing that the tests are reliable for both children living with HIV and those not living with HIV, as shown in Table 4. The Alphas in the latter tests were higher in the lower-scoring sample of children living with HIV than in children not living with HIV due to less severe attenuation effects in the former group.

Table 4 presents the Intraclass Correlation (ICC) of the test and retest scores of the BENCI and the Pearson correlations between the repeated measurements among the 38 children not living with HIV. The Intraclass correlation for specific tests ranged from -0.34 to 0.81. The coefficients were rather high in Sustained Attention RT, Immediate Visual Memory, and Alternate Visuo-motor Coordination (ICC range from 0.74 to 0.81, r = 0.68—0.62). Moderate correlations were found in Immediate Verbal Memory, Delayed Visual Memory, and Visual Recognition Memory (ICC range from 0.52 to 0.58, r = 0.39—0.38). Test retest reliability was poor for Go/No-Go (RT), Sustained Attention CA, and Reasoning (ICC range from 0.14 to -0.34, r = 0.08—-0.15).

The test–retest reliability results showed that most of the tests were consistent on the two occasions (2 months in between t1 and t2). With clear significant gains in performance as expected by increasing test familiarity and maturation for fifteen out of nineteen subtests, except for Sustained Attention CPT, Verbal Recognition Memory (CA), Reasoning (CA), and Go/No-Go (RT) that showed no clear improvements in mean performance.

Table 5 presents the correlations between corresponding BENCI and Kilifi toolkit tests. The attention, memory, inhibition/planning, reasoning, and flexibility tests in the BENCI and Kilifi were expected to correlate. However, some of these tests did not correlate as expected due attenuation effects, while others correlated as expected despite the attenuation effects.

In domains of reasoning, several inhibition, and a few memory-related tests in the BENCI were positively correlated with tests in Kilifi toolkit, supporting convergent validity across these domains. The BENCI’s Working Memory test was expected to correlate with Kilifi’s Self-Ordered Pointing Test (SOPT) because they both measure working memory. However, the BENCI Working Memory test did not have a significant correlation with Kilifi’s working memory test, Self-Ordered Pointing Test (SOPT). This could be because the BENCI Working Memory test showed ceiling effects and might have been too easy for most test takers.

Kilifi’s Verbal List Learning Test and Nonverbal Selective Reminding Memory test were expected to correlate with the BENCI’s Verbal Memory and Visual Memory tests because they all measure memory. However, none of the BENCI’s memory tests had a significant correlation with Kilifi’s Nonverbal Selective Reminding Memory Test (NVSRT). Moreover, the BENCI’s Verbal Memory Recognition and Visual Memory Recognition tests had no significant correlation to any of Kilifi’s memory tests. This outcome could be because the BENCI’s Verbal Memory Recognition and Visual Memory Recognition tests had some ceiling effects while Kilifi’s NVSRT had floor effects. However, the BENCI’s Verbal Memory Immediate hits had a significant correlation with Kilifi’s Verbal List Learning’s (VLL) Immediate Memory Span (r = 0.37), Level of Learning (r = 0.40) and Total correct answers (r = 0.41). In addition, the BENCI’s Verbal Memory Delayed Trial was also significantly correlated with Kilifi’s Verbal List Learning’s Immediate Memory Span (r = 0.21). Moreover, the BENCI’s Visual Memory Immediate hits had a significant correlation with Kilifi’s Verbal List Learning’s (VLL) Immediate Memory Span (r = 0.23), Level of Learning (r = 0.34) and Total correct answers (r = 0.32). In addition, BENCI’s Visual Memory Delayed Trial was also significantly correlated with Kilifi’s Verbal List Learning’s (VLL) Level of Learning (r = 0.23) and Total correct answers (r = 0.25). The significance was found despite the BENCI’s Verbal Memory Delayed showing some floor effects. The rest of the memory tests in the BENCI and Kilifi had no ceiling or floor effects. The correlation between Kilifi’s Verbal List Learning’s (VLL) Level of Learning and Total correct answers and the BENCI’s Reasoning test was not expected. As expected, the BENCI Abstract Reasoning Test significantly correlated with Kilifi’s Raven’s Progressive Matrix (RPM) (r = 0.21). Both reasoning tests had no attenuation effects.

Kilifi’s People Search test and FNRT test were expected to correlate with BENCI’s Continuous Performance test and Spatial Stroop Attention test because they all measure attention. Among the attention tests, the BENCI sustained attention test, Continuous Performance hits and reaction time test, did not have a significant correlation with Kilifi’s visual sustained and selective attention—People Search test (r = -0.10; r = 0.12), as well as auditory sustained and selective attention test—Forward Digit Span total score (r = -0.14; r = 0.07). People Search test had floor effects while Continuous Performance hits had ceiling effects. Moreover, the BENCI tests that contain an attention component, Reasoning (r = -0.37) and Working Memory (r = 0.19) were also significantly correlated to Kilifi’s People Search. Kilifi’s People Search and its correlation with the BENCI’s Reasoning and Working Memory tests was unexpected as these BENCI tests are not primarily meant to measure attention.

Kilifi’s Tower Test was expected to correlate with the BENCI’s planning test because they both measure inhibition. This is indeed the case, as the BENCI Planning Total Time test had a significant association with Kilifi’s Tower test (r = -0.21). However, BENCI’s Planning Time of First Option test had no significant association with Kilifi’s Tower test (r = -0.11). This results should be interpreted cautiously because the BENCI’s Planning Total Time test had some floor effects while the Planning Time of First Option had floor effects indicating that items were relatively difficult for our test takers.

Overall, in the reasoning domain, much convergence between the BENCI and Kilifi Toolkit was supported, whereas in the memory and inhibition domains there was only partial convergence. Subtests in the flexibility, attention, and working memory domains showed little convergent validity with the Kilifi mostly because of attenuation effects.

As can be seen in Table 6, children not living with HIV outperformed those living with HIV on all BENCI tests. However, the mean group difference was significant in all subtests except Continuous Performance Test hits and reaction time, Go No Go hits, Verbal Memory Recognition hits, and Planning total time.

We checked whether the performance within the BENCI subtests aligned with developmental models’ expectation of growth in cognitive performance as children aged, and report Pearson correlations between age in years and the BENCI subtest performance for the children living with HIV- and those not living with HIV separately in Table 7. We hypothesized that children not living with HIV would significantly outperform those living with HIV. Among the children living with HIV, there were significant associations in the expected direction between age and Verbal Comprehension Images hits, Verbal Memory hits, Verbal Memory Recognition hits, planning total time, Planning Time of First Option, Abstract Reasoning hits, Visual Memory Immediate hits, Visual Memory Recognition hits and Spatial Stroop omission errors. Among children not living with HIV, there was a significant association between age and Continuous Performance reaction time, Processing Speed reaction time, Verbal Memory hits, Abstract Reasoning hits, and Visual Memory Delayed hits. The lack of significant correlations between some cognitive indicators and age could be because of attenuation effects, but might also relate to sampling issues (e.g., older participants appearing in the sample because of delayed development and the repeating of grades in school).

We tested the construct validity of Executive Functioning as proposed by Diamond for normal development [30]. According to his model, the subtests that measure inhibition, flexibility, reasoning, memory, and fluency together constitute executive functioning [30]. These are tests that evaluate the ability to make decisions, exercise self-control, pay attention, be creative, solve problems, and plan towards having good health and success in life. These are considered core functions in the brain hence the name executive functions. We fitted a confirmatory factor analysis model previously fitted successfully in the Arabic version of the BENCI [20] and sought to adjust the model slightly to improve fit if necessary.

A second-order model with Executive Functioning as a second-order latent factor and five first-order latent factors (i.e., Fluency, Reasoning, Memory, Inhibition and Flexibility) measured by the specific the BENCI subtests (Fig. 2) was specified and tested with the pooled sample including missingness handled by Full Information Maximum Likelihood. The model fit indexes suggested a good fitting model (χ2 (100, N = 604) = 245.55, p < 001, RMSEA = 0.049, CFI = 0.908, TLI = 0.875). However, this model had several issues. First, the Fluency factor was estimated to have a negative residual variance that we fixed at zero. Second, in this revised model, the Verbal memory factor also yielded an estimate negative residual variance that we treated similarly by fixing it at zero. Third, in the third model, the residual variance of the Alternate Visuo-motor total time also needed to be fixed to zero. Next, we considered modification indices and found that the model could be improved if we included a covariance between the residuals of Reasoning and Flexibility and between the residuals of Semantic Fluency correct answers and Verbal Memory Recognition correct answers. This further modified model showed an acceptable fit (χ² (101, N = 604) = 205.73, p < 0.001, RMSEA = 0.041, CFI = 0.934, TLI = 0.911). Figure 2 presents the standardized factor loadings. An inspection of this model showed that not all indicators of Inhibition (Go No Go RT = λ -0.46; Go No Go CA = λ 0.74) had significant loadings on their respective factor, indicating that these specific tests did not measure Inhibition as intended (Fig. 2). It also showed that the latent factor of Inhibition did not load on the Executive Functioning factor. Therefore, we removed the Inhibition factor together with its indicators and tested a second-order factor with only four factors. This model fitted well (χ² (51, N = 604) = 135.57, p < 0.001, RMSEA = 0.052, CFI = 0.944, TLI = 0.914). Figure 2 presents the factor loadings of this model. Therefore, the five components of Executive Functioning as validated before did not all show up in the Kenyan sample, while Executive functioning comprised of fluency, reasoning, verbal memory, and flexibility was found to fit well in the Kenyan sample. The final model with four factors each measuring executive functioning supports the construct validity for the BENCI battery, despite Heywood cases on the Alternative Visuo-motor subtest.

AMOS treats missing data using full information maximum likelihood, which is considered a robust method for treating missing data. However, we checked whether model fit would be affected when using a dataset with no missing data. On running the model with no missing data, the model fit was excellent (χ² (51, N = 327) = 64.07, p > 0.05, RMSEA = 0.028, CFI = 0.968, TLI = 0.958). This shows that the BENCI does have good construct validity though some changes in some test items and instructions are needed in future revisions of some subtests.

Some subtests were found to have floor effects due to having too many difficult items while others had ceiling effects due to having too many easy items. Too few and easy items resulted in ceiling effects for the language tests. The BENCI’s subtests showed poor to excellent internal consistency with most subtests showing higher alpha values for the HIV-positive group than the school sample. This was likely caused by smaller attenuation effects in the subtests with ceiling effects or the HIV-positive group showed more variation in true scores leading to higher Alphas as seen in the N-back working memory test [31]. The internal consistencies in our study were similar albeit slightly lower than those found in the Moroccan sample [20], possibly because the level of difficulty of the test items suited the younger cohort in the Moroccan sample better than in our data. This points to the need to develop age-appropriate norms and to add items with age-suitable difficulties in future revisions.

Our results for the BENCI test–retest reliability were fairly similar to a previous study conducted in Morocco [20]. The Arabic adaptation of the same tool reported Intraclass correlation to range from -0.23 to 0.81, similar to our study [32]. However, the poor test–retest reliability of the reasoning test could be due to the relatively long-time interval between the two assessments in our study as a longer interval may create changes in the construct [33]. It is possible that the respondents were thinking about the test items more often than before the first administration [33]. The latter is more likely with children who have high mental imagery skills meaning they are likely to think about the test items quite often and grow familiar with them and forthwith give different responses in the second assessment [34]. A child may respond substantially different in a language test whose retest is one year compared to verbal memory because their language ability has improved well past their memory ability. Studies on cognitive tests have had a re-test time interval of 15 to 60 days though there were recommendations for within a 14-day lapse of time especially for tests such as visual memory which would lose reliability over longer durations [20, 33, 35‐37]. However, some studies have shown that for verbal memory and visual motor speed tests, the test–retest reliability with a one-year time-lapse remains stable while for language tests a recommendation for not less than 14 days has been made [36, 38]. The mixed results in our study suggest that the test domains and time lapses play a role here [37]. Our test–retest results in attention tests are also similar to those of other studies that show higher reliability in attention speed tests compared to accuracy tests [39]. Tests that call for speed over accuracy have been found to have high reliability than those that call for accuracy over speed [39].

BENCI attention tests do not correlate with Kilifi’s People Search and Forward Digit Span as expected, but they showed convergence with tests that had attention components. Studies have cited the tendency of attention tests to confound with other cognitive functions [40‐42]. In our study, similar administration processes between tests with attentional components could have contributed to convergence as seen in the BENCI’s Working Memory test with Kilifi’s People Search test. These two are attentional control tasks as they call for a response to correct stimuli during incorrect stimulus thereby inhibiting a response. Correlations between attention tests have been found to support convergent validity with a range from low- to-moderate. Speed measures have higher significant correlations compared to accuracy attention measures [39, 43]. In our study, however, the BENCI attention accuracy tests showed moderate convergent validity while attention speed measures showed weak convergent validity. Poor convergent validity between some attention tests has been documented in other studies [40]. In the memory domain, BENCI’s working memory and Kilifi’s people search correlated well, a finding that has also been found in other studies comparing working memory tests to attention tests [37, 41, 42].

BENCI’s Visual Memory test showed a weak correlation with Kilifi Toolkit’s Nonverbal Selective Reminding Memory Test (NVRST). The administration is similar between these two tests. An explanation for this could be found in studies showing the impact of familiarity with the tools on scoring. In our study, the NVRST test involved memorizing the shape formed by a set of 8 dots and then replicating the shape by placing a marble on a set of dots. In the BENCI version, the child was supposed to memorize several images and then correctly point them out when shown amidst a set of other pictures; a task that would involve other cognitive functions such as visual-motor coordination. Pointing out pictures is a familiar learning concept in the Kenyan context. This is because among the methods used in teaching preschoolers is by pointing out images and encouraging the children to read and memorize their names. The administration was fairly similar but their scores in terms of correct answers were not highly correlated. Probably other psychological processes are involved in the BENCI subtest that are not in the Kilifi subtest. There are some studies that have found a similar lack of correlation between tests. In a study done in Zambia, a non-verbal test called draw-a-person was locally adapted and the two tests, the original and adapted one, were compared and found to not be correlated [44]. However, when the ratings were done by adults and correlated to educational outcomes, the two tests had significant correlations. Further research can explore similar comparisons between uncorrelated tests to find out if other psychological processes are involved. Such an evaluation could be similar to the one conducted in the Zambian study. This is in trying to find out whether the BENCI visual memory test expectations do truly reflect the cultural indicators for non-verbal memory. However, the NVRST in the form of Children’s Memory Scale (CMS) dot location subtest has also been found not to have significant correlations with the Lebby-Asbell Neurocognitive Screening Examination (LANSE) visual memory test [42]. In addition, NVRST administration involves visuomotor coordination and other cognitive functions in addition to memory.

Computerized assessments are preferred due to ease of administration and scoring as well as precision [37]. However, Kenyan children are not very used to computerized assessments and a lack of familiarity may introduce variance in test scores that are not related to the construct being measured. Some of the factors that have been known to introduce construct irrelevant variance with computerized assessments include proficiency with the computer-based tests, ease of interaction with the platform, speediness of the tests and test-taker’s anxiety [45]. Some administration processes, such as tasks calling for inhibitory control, within the tablet may affect some domains more than others [46]. The lack of familiarity and some administration processes associated with tablet-based testing could affect convergence validity when compared to some paper-based tests. However, there are some studies that have shown no significant differences in test performance between tests using computer-based platforms and those using paper-based ones meaning that variation in convergence may apply to some tests more than others [46]. To reduce variation in some of these tests, studies have suggested several approaches including reducing the difficulty level of computer-based tests as well as clarifying the relationship between tasks and the expected test takers performance [45, 47]. It is however, beyond the objectives of this study to investigate approaches that would have worked best in reducing validity variance between the BENCI and Kilifi toolkit. These are next level questions to consider.

The lack of convergence in some tests may also be contributed to by lack of a common construct between some of the BENCI and Kilifi toolkit tests. Since the latter is the gold standard, comparing it to a test that that does not capture the same constructs may give us erroneous findings. Differences in correlations between measures have been found to increase when comparisons are made to alternate measures with low convergence validity [47]. Improvements and adaptations of some of the BENCI tests may improve convergence with the Kilifi toolkit tests.

The BENCI highlighted clear mean differences between the HIV-positive and HIV-negative groups. Just as indicated in the BENCI results, tests can have mean differences but the score differences between the groups may not be significantly different as seen in the scores for correct answers in the inhibition test and time taken in the planning test. An earlier study showed that certain tests like inhibition and planning can have the ability to differentiate healthy from unhealthy populations but the difference in scoring within the tests may not be significantly different [48]. However, the BENCI did affirm what other studies have found that children living with disease score lower than children living without disease in tests of working memory, inhibition, memory, and planning among other cognitive functions [42, 48‐50]. Moreover, taking more time when doing a test has been associated with taking more mental effort to achieve a desired outcome, in this case a correct response, entails a healthy approach to inhibitory tasks [30]. Better performance in correct answers is denoted by higher scores while in reaction time it is denoted by lower scores. Therefore, for children having high reaction time, performance will be regarded as poor. Overall, this is true when the dependent variable is time but not when it is accuracy. For example, higher reaction time is worse than a lower one in Selective Attention, Sustained Attention and Go/No-Go tests. These findings add to the body of literature on the significance of testing for cognitive deficiencies among unhealthy children.

The planning test did not have significant loading on the inhibition factor in the pooled sample and subsequently, this factor did not load well onto executive function. This has not been the case in another Sub-Saharan African study that supported the construct validity of a planning test [51]. Inhibitory control has been found to be higher in children within settings that emphasize obedience and self-control such as East Asian countries and been found to be lower, to a point where there are no significant age differences, among children in developing countries and communities [52]. The study also reported cross-national differences in inhibition, shifting, and updating. We would then expect the children in this study to have the BENCI inhibition tests to load onto executive function just like other western adapted tests have done in a sub-Saharan setting. This is more so since inhibition tends to develop rapidly among younger children hence, we would not expect a lack of this cognitive function among 6- to 14-year-olds even though inhibitory control tends to mature at adolescence [30, 53]. However, studies looking into whether maturity of inhibitory control affects how well the function can load into an executive function model may clarify the results we found in this study. Observations of the school and home executive function stimulation activities give a broader picture of the activities emphasized and how they encourage inhibitory control development. These observations could be integrated in further research with the BENCI. Flexibility on the other hand builds developmentally onto inhibition and loaded well on executive function. This finding does not reflect the arguments pointed out earlier on inhibition. Inhibition is a first-order component that appears around 6–8 years and flexibility is a second-order component that appears later in development [54]. Since flexibility loaded well onto executive function, the lack of significant loadings in the inhibition construct could potentially be because of the lack of culturally aligned items in the inhibition tests or a problem with instructions. The findings in the construct validity indicators call for a developmental approach when interpreting scores and the need to norm the BENCI for age groups.

The BENCI also showed support for metric and partial scalar invariance as opposed to strict scalar invariance. This means that the BENCI items are loaded onto the latent factors similarly across groups, hence can be compared across the groups. The same applies to items per subtest. However, comparability of means between the latent factors was not supported in its entirety meaning that we cannot compare the means of fluency, flexibility, verbal memory, and reasoning across the groups. We can choose to create separate norms for HIV + and HIV- groups since the tests behave differently in the two groups, but this will not give us an opportunity to compare performance. One of the options that can enable performance comparison is to create norms with the healthy and optimally functioning group but caution should be integrated when norming for Verbal Comprehension Figures and Visual Memory Delayed tests. We may underestimate or overestimate between-group abilities due to miscalibration of the tests and the results may be marred with measurement bias. This means that we may not have true between-groups construct differences due to other construct irrelevant variables causing differences in test scores. In this case, we may choose to correct for intercept differences during norming by estimating their effect sizes and relating this to effects on the norm scores [55]. As an alternative, we can choose to carry out a study on why the two tests are biased and correct for any item level (attenuation effects in Verbal Comprehension Figure). We are yet to come across a study that investigates measurement invariance of a neurocognitive tool in Kenya and its regions. Children studies that we have come across are based in high income countries [56] and cannot be compared to our setting due to different group dynamics and cultural dynamics that underlie cognitive performance and developed test items [24].

In this study, one drawback was that the results could only be generalized in a community setting and not a clinical one. We could not find comparison tests for some domains due to the limited availability of validated tools within the Kenyan culture.

The study also noted that some subtests had floor and ceiling effects, which compromised the interpretation of other findings. In this case, any results pertaining to the subtests having ceiling and floor effects should be interpreted with caution. Moreover, further studies may revise the tests by perhaps adding more items to the tests with ceiling effects and decreasing the difficulty of the items in the tests with floor effects so as to match the difficulty to ability level and reduce attenuation effects. In addition, age-appropriate norms for the subtests should be considered.

The methods used to capture reaction time and total time may not have been completely accurate because the paper–pencil tests used a stopwatch that is prone to administration errors while the IPad-based tests used an internally configured watch. In the paper-based tests, errors may be integrated when timing is not stopped immediately a task is completed or when an administrator gives more time for task completion than would be required. These can create systematic or random measurement errors where the latter could suppress correlations. This may have been the case in convergent validity where random measurement could have suppressed some correlations. Nevertheless, the possibility of errors in paper-based tools is another reason to prefer automated computerized tests with internalized and consistent timing across participants.