Validation of cross-cultural child mental health and psychosocial research instruments: adapting the Depression Self-Rating Scale and Child PTSD Symptom Scale in Nepal

The dearth of mental health and psychosocial support (MHPS) research in low- and middle-income countries (LAMIC) is a barrier to providing evidence-based care to children and youth. Of the more than two billion children in the world, the majority lives in LAMIC. Half of these children live in poverty, two-thirds are affected by armed conflict, and a third is underweight or stunted [1], yet these vulnerable populations comprise only five percent of published child mental health literature [2].

The adaptation and validation of instruments to assess MHPS for use in LAMIC is crucial to eliminate this gap in child global mental health research and service provision [2, 3]. Hospital records, structured diagnostic interviews in clinical settings, and school-based psychological testing have been used extensively in high-income countries to assess both prevalence of childhood mental illness and the response of children to intervention. However, hospital records and clinical diagnostic interviews are typically lacking in LAMIC because of the absence of child psychiatric services. Therefore, the use of instruments that can be administered to teachers, parents, and children are a helpful alternative to gain information until clinical services are more established. Without validated instruments, resources are easily misallocated through either providing care to children not requiring services or depriving care to children who desperately need it. Furthermore, the psychometric properties of validated instruments can be used to estimate costs of service provision or deprivation.

However, instruments developed and validated with children in high income countries with Western cultural settings cannot simply be translated with the expectation they will have the same psychometric properties in other cultural contexts. Cutoff scores established with Western child populations are not necessarily comparable in other settings and may lead to misclassification and distortion of prevalence rates [4, 5]. Moreover, the instruments may not capture the constructs they are intended to measure in other cultural contexts where the meaning, clustering, and experience of symptoms often differs [6‐8]. In humanitarian emergencies in particular, data from unvalidated instruments can be worse than no data at all because it may lead to inappropriate and potentially harmful intervention [9, 10]. Without the use of validated screening instruments, psychosocial interventions unintentionally may divert resources from the children most in need of mental health services. Without validated screening instruments, it is not possible to evaluate the effectiveness of an intervention, thus risking potential failure of programs to improve children's lives and simultaneously wasting scarce human and economic resources.

Questionnaires, therefore, need to be validated in any new socio-cultural setting. However, the process and interpretation of validation procedures are not straight forward, especially in cross-cultural context. The definition, determination, terminology, and interpretation of validity vary by discipline, available resources, and type of problem studied. Cross-cultural validation techniques have been developed for adults [11‐15], and there are examples for children and adolescents, as well [16, 17]. However, there has not been agreement on a single validation method most appropriate for global mental health research with adults or children. The sole consensus has been that only translating and back-translating falls short of producing valid tools [4, 18]. With increased emphasis on task-shifting in mental health care and training [19], it may be possible to have experienced non-psychiatrists conduct validation evaluations using structured interviews. Task-shifting refers to the process of having midlevel professionals, such as nurses and physician's assistants, and paraprofessionals, such as community psychosocial workers, take on responsibilities assumed by psychiatrists and psychologists in high-income settings. Task-shifting is warranted because of the dearth of high-level professionals in LAMIC. Ultimately, an array of validation approaches may prove most useful because the variation in types and intended uses of measures in global mental health and psychosocial research.

We proposed six questions in the introduction that can be used by mental health researchers and psychosocial interventionists working with children in cross-cultural settings. Our goal was to use these six questions to evaluate our process of adapting and validating instruments for mental health and psychosocial research with children affected by armed conflict in Nepal. For the first question "What is the purpose of the instrument?", our goals were evaluating treatment, estimating prevalence, and detecting MHPS-related disability. For the second question "What is the construct to be measured?", the purpose dictated employing cross-cultural constructs that were locally salient and sufficiently resembling the psychiatric categories of depression and PTSD. Therefore, we required a validation against an external criterion related to diagnosis and impairment, which we accomplished through ratings on structured interviews with the GAPD and K-SADS completed by a psychosocial counselor. However, before the validation we needed to assure appropriate transcultural translation by answering questions three, four, and five.

For the third question, "What are the contents of the construct?", somatic symptoms stood out as lacking content equivalence between Western populations and Nepali populations. Qualitatively, children did not associate DSRS items six and eight (appetite loss and stomachaches) with sadness or depression. Quantitatively, these items had no significant discriminant validity and the lowest inter-item correlations. Similarly, during validation of the Beck Depression Inventory and Beck Anxiety Inventory for adult populations in Nepal, gastrointestinal complaints did not differentiate between persons with and without psychological distress [58‐60]. For future research, we would recommend exploring the discriminant validity of other somatic complaints such as headaches or paresthesia, in place of gastrointestinal complaints. In place of an appetite question, more concrete items about changes in food eaten regardless of food availability may be more effective, e.g. "Have you been eating less food than usual over the past week even when food was available?" or "Have family members said that you are not eating enough food?" Alternatively, these items could be dropped from the DSRS. Based on current evidence, questions of abdominal complaints and appetite changes do not help identify depression among children or adults in Nepal.

In the CPSS, children in focus groups identified two items as common responses to trauma that were not associated with distress. These items were avoiding activities and people related to the event (CPSS.7) and less interest in activities (CPSS.9). In a conflict zone, it would be appropriate to avoid a place where an attack or bombing occurred or avoid people in uniform who may incite violence. Similarly, children added the common cultural explanation that ghosts and spirits haunt places where bad events happened. Quantitatively, these items also had poor discriminant validity. In settings of recent conflict, items related to avoidance and changed activities may not reflect pathology and could erroneously inflate PTSD prevalence estimates. However, these items may be more salient in detecting MHPS problems with greater time after cessation of political violence.

A surprising finding was the disjoint between the focus group qualitative findings and the validation study in regards to traumatic amnesia (CPSS.8). Children in multiple focus groups stated that traumatic amnesia does not occur. Moreover, they stated that their goal was to forget the event, and forgetting leads to feeling better. In the Nepali context, forgetting does not literally refer to being unable to remember but rather refers to not having intrusive memories [36]. The denial of traumatic amnesia is consistent with qualitative work among adult trauma survivors in Nepal who claimed that they remember all the details of their traumatic events but wish they could not [35]. However, when the 162 children completed the CPSS, traumatic amnesia had greater mean endorsement than ten of the other items. Traumatic amnesia also had the greatest inter-item correlation and had significant individual item discriminant validity. This raises questions about why these qualitative and quantitative responses appear to be at odds. To resolve this, it would be helpful to do more qualitative work to find out why some children endorsed the traumatic amnesia item. The issue of traumatic brain injury may be salient here because many child soldiers with PTSD reported exposure to bomb blasts, which could affect trauma recall [27].

For the fourth question "What are the idioms used to identify specific items?", the wording was changed on numerous items. To improve acceptability of the instruments, children suggested the removal of language that appeared to blame and stigmatize respondents. In Nepal, it is common to view traumatic experiences as the result of bad karma [35]. Therefore, children suggested removing the mention of guilt in the CPSS because it could be perceived as reinforcing blame among trauma survivors.

For the fifth question, "How should questions and responses be structured?", it was culturally unfamiliar to present children with declarative statements to endorse the degree of veracity. It was more understandable to administer the items as questions. In addition, the order of the response set on the DSRS was counter-intuitive. A striking finding was children's interpretation of locally developed pictographic representation of emotional gradations. The dhoko-basket scale had a different meaning to the children than that intended by the researchers. An intended physical to emotional abstraction was interpreted instead as an economic to emotional abstraction. This illustrates how attempts at cultural-adaptation can lead to even greater confusion or misrepresentation. It is as important to do focus groups about locally-developed items and responses sets as it is to assess Western-developed tools.

The CPSS performs similarly: 72.2% of children are correctly classified. However, nearly one quarter are misclassified with high CPSS scores but lack counselor rated psychosocial disability (false positive). Of the total sample, only 5.6% of children have psychosocial disability but are misclassified with low CPSS scores (false negatives). Low prevalence of trauma-related disability also contributed to the large difference between the negative and positive predictive values. If psychosocial disability were more prevalent in the sample, the instrument would have shown greater positive predictive value. Ultimately, both the DSRS and CPSS perform well to include the majority of children in need of services. However, the instruments, if used as screening tools, would include a large number of children who do not have psychosocial disability, thus reducing the cost effectiveness of a resource-intensive intervention.

Using an adapted validation procedure with a psychosocial counselor who received extra training was a useful alternative to clinician-rated validation procedures in a setting without child mental health specialists. Our procedure did not require that the few psychiatrists or psychologists in Nepal leave their obligations of providing care. It did not incur the high cost of purchasing expert clinician's time. It also is replicable for other validation procedures. In settings similar to Nepal, highly trained psychosocial workers with multiple years of experience may be best positioned to make assessments on indication for psychosocial treatment because they are the individuals with the greatest training and experience in this setting, and they know the cultural context. Moreover, the emphasis on psychosocial disability using a structured modification of the GAPD assured that the validated instruments captured children with functioning problems and not only presence of symptoms. Validation of the Child Psychosocial Distress Screener in Burundi employed a similar approach [16]. Ultimately, our alternative procedure produced instruments with acceptable psychometric properties. When compared with Birleson's [45] original validation of the DSRS, the Nepali DSRS has similar sensitivity (English 69% vs. Nepali 71%) but better specificity (English 57% vs. Nepali 81%).

We were surprised to find that item discriminant validity varied significantly between the Nepali CPSS and the English CPSS psychometrics established in the U.S. [50]. In the original U.S. sample, the six items with lowest discriminant validity included traumatic amnesia and foreshortened future--items that showed the strongest validity in this Nepali sample. Moreover, the three items that showed the strongest validity in the American youth sample performed poorly in the Nepali sample: distress with reminders, less interest in activities, and overly careful. It is unclear whether this is due to the nature of trauma studied--a single earthquake in California versus a decade of war in Nepal--or other cultural differences. In another U.S. sample, the irritability/anger item was associated strongly with disability, which is in keeping with the Nepali findings [61].

The study also highlights items that could be selected to produce brief screening versions of the DSRS and CPSS. From the DSRS, the three items with significant discriminant validity for indication-to-treat were having energy to complete daily activities (DSRS.7), feeling that life's not worth living (DSRS.10), and feeling lonely (DSRS.15). The "life not worth living" item is important to include also because affirmative responses should trigger a suicide screening and referral for services. Five items on the CPSS had significant discriminant validity: nightmares (CPSS.2), flashbacks (CPSS.3), traumatic amnesia (CPSS.8), feelings of a foreshortened future (CPSS.12), and easily irritated/angered at small matters (CPSS.14). Interestingly, these five items comprise two items from criterion B (re-experiencing), two items from criterion C (avoidance/numbing), and one from criterion D (increased arousal) of the DSM-IV PTSD diagnostic criteria.

Future studies and transcultural translation/validation studies in other setting could improve upon the work described here. After development of an instrument, it would be helpful to do cognitive interviewing with a subset of children. Cognitive interviewing is a qualitative research method in which questionnaire respondents are asked how they interpret a question and why they provide a specific response [62]. This is a more individualistic approach to complement similar information obtained through focus groups. This would help to elucidate contradictory findings such as that related to traumatic amnesia. Furthermore, a larger sample size for the validation study would have increased the power for individual item discrimination. All of the psychosocial functioning assessments were done by one psychosocial counselor. With more assessors, inter-rater reliability could have been assessed and idiosyncrasies of individual raters may be revealed.

Before concluding, it is helpful to consider how validated instruments can be used to shed light on cost-effectiveness of an intervention. Psychosocial practitioners and researchers increasingly have argued against providing interventions purely based on membership in a vulnerable group, such as being a former child soldier or a victim of child trafficking [10, 63, 64]. Rather, evaluation of a child's MHPS is needed to determine which children may need intervention including children who are not members of a target group. Providing interventions to only children in a specific group can worsen stigma and decrease community support for a program and its beneficiaries.

Screening with validated instruments is an alternative to group-based selection for an intervention [65]. Based on our findings with the DSRS and CPSS, we can compare how the instruments would perform in a cost-based analysis using a screening based strategy. If a specific intervention in LAMIC cost $20 per child [65] and the DSRS were used to screen children for the intervention, the actual cost per child would be higher because of the number of false positives. With the DSRS, there were 26 false positive and 20 true positives who scored above the DSRS cutoff. Therefore, 1.3 healthy children would be treated for every child with depression identified, resulting in 2.3 children (1.3 healthy children + 1 depressed child) entering an intervention. The intervention that cost $20 would then cost $46 (2.3 × $20) to treat one depressed child. For the CPSS, there were 19 true positives and 36 false positives; 1.9 healthy children would be recruited into an intervention for every one psychologically traumatized child at the CPSS cutoff of 20. Therefore, a psychosocial intervention costing $20 per child would cost $58 (2.9 × $20) in programmatic expenses because of the need to include 1.9 healthy children in addition to every traumatized child. Additional calculations are required to estimate the costs to society of not including children with MHPS problems in an intervention. For example, if a child does not receive the intervention, what are the reductions in productive labor, increases in crime, and increases in other healthcare costs. Both the DSRS and CPSS have good psychometric properties (high negative predictive value) to minimize the number of children mistakenly excluded from an intervention, and therefore minimize the costs to society of not treating an individual. The more expensive the intervention, the more crucial it is to have instruments that have strong psychometric properties to properly include and exclude children in psychosocial programs. With the majority of child MHPS programs in LAMIC not employing validated instruments, there is substantial risk of economic inefficiency and financial waste through the inappropriate inclusion or exclusion of beneficiaries.

Validation of instruments in LAMIC, where the majority of the world's children live, is crucial for the advancement of research and intervention for children. Our findings highlight the potential pitfalls of assuming that only translation and back-translation can capture cultural differences in performance of mental health instruments. While the specific process of transcultural translation and validation will vary based on objectives and local resources, a critical evaluation of the translation and validation process is indispensible. The six questions outlined here provide a framework for researchers and interventionists to systematically do such an evaluation of tools. In addition, the study demonstrated that task-shifting the validation process to trained non-psychiatrists using a structured interview can produce acceptable psychometric properties. Validated instruments additionally are crucial to maximize the financial benefit for improving child psychosocial wellbeing and minimize the misapplication of economic resources in LAMIC. Only through the appropriate development and interpretation of instrument-derived findings do we reduce the risk of producing misleading conclusions and generalizations about the mental health of the majority of the world's children.