Comparing the validity of the self reporting questionnaire and the Afghan symptom checklist: dysphoria, aggression, and gender in transcultural assessment of mental health

Expert guidelines advocate the inclusion of local perspectives of distress in the assessment of mental health in conflict-affected populations [1‐3] to address issues of cultural relevance for psychosocial support programming [4‐8]. Context-insensitive questionnaires may lead to inaccurate inferences concerning the needs of conflict-affected populations, whose members may or may not respond to mental health instruments in ways comparable to populations living in relatively secure communities [9]. Simply translating questionnaires into relevant languages is insufficient to address local conceptual issues [10‐13]. Developing contextually valid emic (i.e., culturally-specific) instruments involves combining qualitative and quantitative approaches, marrying the ethnographic concern for local meanings with the epidemiological focus on replicable surveys [7, 13, 14]. In postconflict settings, mixed-methods techniques to develop and implement local measures have been applied to genocide-affected communities in Rwanda [15, 16], refugees from Darfur in Chad [17], postpartum women in the Democratic Republic of Congo [18], violence-affected adults in Timor-Leste [19, 20], internally displaced persons in Sri Lanka [21], and war-affected youth in several settings [22, 23].

The Afghan Symptom Checklist (ASCL) was developed using ethnographic interviews and thematic coding of wellness and illness narratives to provide a culturally-grounded assessment of psychosocial wellbeing in Afghanistan [24]. In their study, Miller and colleagues described several Afghan idioms of distress: asabi, an anxiety-like state; fishar, or blood pressure abnormalities; and jigar khun, literally “liver blood,” an extreme and persistent dysphoria explained in humoral medicine as resulting from an overabundance of blood in the liver. The ASCL had good psychometric properties in a sample of 324 adults in Kabul.

While developing local measures like the ASCL has clear advantages in serving specific linguistic and cultural groups, it is not without costs. Considerable time and resources are needed to complete the essential steps of transcultural measure development: translate and back-translate content, evaluate content validity with international and local experts, and pilot instruments to evaluate psychometric properties [25]. Even when such steps are taken, there remain important questions concerning conceptual comparability. Results derived from measures specifically designed for one setting are often difficult to compare to results from others. In the face of such challenges, psychosocial programs often select screening instruments that have been previously implemented and proved reliable in other international settings. Such etic (i.e., culturally neutral) global mental health instruments are usually deployed to assess a range of symptoms indicative of common mental disorders, positing common factors in psychological distress. Typically these measures are used as screening instruments, and are not designed to be diagnostic of mental health disorders [26, 27] or to provide within-population prevalence rates [9].

The Self-Reporting Questionnaire (SRQ-20) is one such instrument. The SRQ-20 was developed by the World Health Organization [3] to measure symptoms of common mental disorders across different cultures in low-income settings [28, 29]. Designed for ease of administration, it consists of 20 items covering symptoms of anxiety and mood disorders with a binary (yes/no) response scale. The SRQ-20 has been assessed across a wide range of contexts, including conflict and refugee settings in Colombia [30], Guinea Bissau [31], Mozambique [32], Rwanda [33], Uganda [34], Iran [35], Pakistan [36], and Afghanistan [37, 38]. In Afghanistan, the psychometric properties of a Pashto SRQ-20 were examined by Ventevogel and colleagues [39], who reported reasonable psychometric properties but large differences across gender. Some concern has been voiced that internationally-developed instruments such as the SRQ-20 may be insensitive to culturally-salient idioms of distress and thus may not differentiate between clinically-relevant and non-clinical distress [40].

We compared the ASCL and the SRQ-20 in a dataset from Afghanistan to examine the value of using a locally-developed emic measure versus using an internationally-developed etic measure. We approached this question methodologically in two stages: first we examined the construct validity of each measure by comparing items’ face validity and using factor analysis. Second, we compared the two measures’ external validity using regression models. In the latter we asked whether the measures were comparably associated with two third variables that have a well-known impact on mental health in low-income and conflict settings: exposure to traumatic events and socioeconomic status. Trauma and poverty are reliably associated with anxiety disorders, depression, and general emotional distress in conflict-affected adults [41‐43]. Comparing how the ASCL and SRQ-20 were associated with these variables would provide an estimate of their external validity, a critical component of measurement validity [44]. A similar approach was used by Jayawickreme and colleagues to compare the PTSD Symptom Scale-Self Report [2] and the Beck Depression Inventory (BDI) to a locally-developed measure in a conflict affected sample in Sri Lanka [13]; the latter was found to explain additional variance in functional impairment.

In order to address the conceptual ambiguity inherent in using total scores calculated from multidimensional questionnaires, we combined construct and external validity goals to examine how empirically identified subscales on the ASCL and the SRQ-20 were differentially associated with trauma exposure and socioeconomic status. Our hypotheses were that certain subscales would draw on similar variance, but that the ASCL would include dimensions that the SRQ-20 did not, and these dimensions would account for more variance in external criteria. However, given that there has been little examination of the factor structure of either measure, we did not have specific hypotheses about the content of specific subscales or which subscale would be more associated with which criteria. Given the known gender segregation and disparities in mental health in Afghanistan [45‐48], we tested hypotheses that these effects would be moderated by gender.

Strengths of the current study include an explicit focus on the cultural relevance of instruments, two years of preparation and pilot surveys, recruitment across multiple sites, random sampling, and gender balance. We feel that because of these strengths this study avoids many of the limitations common in the field. However, like all research projects, this study is not without its limitations. These include not measuring functional impairment, relatively low levels of associations between distress scores and external criterion variables, double-loading items in EFA models, and suboptimal RMSEA statistics in CFA models for the ASCL. Nevertheless, we believe our analyses comparing the external validity of the SRQ-20 and the ASCL are clinically meaningful, and we encourage researchers to build upon these findings to improve mental health assessment in Afghanistan and other conflict-affected settings.

Our findings suggest that the ASCL was a better measure of distress than the SRQ-20 for women, while the two measures were similar for men. Given the strict gender demarcation of Afghan society [24, 48, 51, 55, 56], we expected to find striking gender differences. Men and women inhabit different social and emotional spaces in Afghanistan, with downstream implications for psychosocial [37, 46‐48] and physiological [50] wellbeing. The large mean differences across ASCL and SRQ-20 subscale scores are thus consistent with the literature. Why the cultural appropriateness of the ASCL seemed more salient for women is less clear. Being involved in the public sphere, men may be more likely to recognize the more generalized modes of expression codified in the SRQ-20, while women, who have limited social interactions outside the home, may respond to more culturally-grounded expressions of distress. It may also be that responses follow the script of gender-differentiated display rules (i.e., culture of emotions). Permissible ways to express distress are gendered, such that Afghan men may publicly express anger, jealousy, and hate, but not fear, grief, or doubt – emotions that might bring shame to families’ honor [57]. Thus in our data, men’s scores may have been constrained by a ceiling effect on particular item responses.

The hypothesis that using total scores would mask important variability with implications for external validity was supported by our findings. The internal structures we found for the SRQ-20 and the ASCL were somewhat distinct from previous descriptions. The latent structure of the SRQ-20 included three factors: somatic complaints, negative affect, and emotional numbing. Although different from the two-factor solution identified by Ventevogel and colleagues [39] using a different Afghan dataset, this structure still largely mirrors symptom clusters of common mental disorders – i.e., anxiety and depression. Unlike the factor structure reported by Miller and colleagues [24], the ASCL in our data was comprised of a latent variable structure consisting of jigar khun, aggression, and fishar. Jigar khun was by far the dominant latent variable in exploratory and confirmatory models of ASCL scores, suggesting that it represents a specific cultural syndrome rather than simply an idiom of distress. Idioms of distress are more general ways of experiencing and expressing distress within specific local contexts [58‐60], whereas cultural syndromes (as described by Nichter) are “widely recognized prototypical cultural ailment[s] that encompasses a fuzzy set of associations coalescing around one or more core cultural symbols” (p.407) [58]. In prior development of the ASCL [24], the item jigar khun consisted primarily as grief. In our study, the inclusion of asabi and other items representing states of arousal within the jigar khun subscale seems to suggest that jigar khun is a multidimensional construct, combining several symptoms of common mental disorders. In other words jigar khun is more than a general expression of distress, and we feel that it meets criteria as an Afghan cultural syndrome.

Jigar khun was not associated with variance in trauma events beyond negative affect and emotional numbing as measured by the SRQ-20. This suggests that jigar khun pulls on similar sources of variance as somatic complaints, negative affect, and emotional numbing for Afghans with respect to trauma. With respect to socioeconomic status, however, the models for jigar khun and SRQ-20 somatic subscale suggest a different conclusion – namely, that jigar khun is associated with household wealth above and beyond somatic complaints for women. That dysphoria is associated with poverty in populations affected by armed conflict is well-known [41, 42]. What our findings seem to suggest is that this association may be gender-dependent. Further research should examine the gendered nature of features of distress, especially in regions with strict gender segregation such as Afghanistan.

In contrast to jigar khun, we found that aggression was associated with both trauma and socioeconomic status above and beyond the SRQ-20 subscales, especially for women. This suggests that the ASCL is a better measure of Afghan distress in large part because it includes aggression items. Aggression has been for the most part ignored in standard North American and European psychiatric concepts of common mental health disorders. Its prominence here is consistent with reports that behavioral disturbances are common in other taxonomies of mental distress. Explosive anger is a critical element of psychosocial distress among West Papuan refugees in Australia [61]. In Timor-Leste, explosive anger is associated with ongoing socio-economic disadvantage [43] and past trauma exposure [20]. Rees and colleagues [62] have emphasized the risk of explosive anger for women in particular, who experience the disproportionate impact of poverty hardships and human rights abuses. In Afghanistan, Catani, Schauer and Neuner [63] have linked war exposure for adults and increased levels of abusive behavior toward their children. Our findings show that aggression is an important dimension of Afghan expressions of distress, and that it can take the form of self-directed as well as interpersonal violence. In content analyses of qualitative interview data collected concurrently with the present study [48] and in other work conducted in Afghanistan [45], women with intense emotional distress reported hitting themselves, beating their children, and causing injury to others. Indeed, the connection between aggressive behavior and psychological distress in Afghanistan is reflected in language used to describe distress. In Pashto and Dari, the word khapa means angry and annoyed as well as sad, anxious, and worried. The word khapgan (lit: hardship) refers to either sadness or anger, depending on social context [64].

Our findings are specific to comparing measurement across constructs, and do not imply equivalence of meaning. In other words, that the subscales of the SRQ-20 and the jigar khun subscale of the ASCL drew upon a similar pool of variance psychometrically does not necessarily imply that the SRQ-20 subscales represent culturally meaningful concepts. Being associated with similar variance in external criteria may imply comparable content and external validity, but does not imply similar construct validity. This is not to undermine those who would argue that the SRQ-20 is useful for screening in humanitarian response efforts; indeed, our findings suggest that it is, insofar as it is associated with important variance in Afghan emotional distress. However, presenting evidence of psychometric association is not the same as presenting evidence of culturally-grounded parallel meanings. Symptom checklists and screening inventories are situated within the cultures of their development; useful in comparative monitoring and evaluation, they are rarely designed to capture the subtleties of local explanatory models.

Studies that have compared locally-developed, emic mental health measures to globally-deployed, etic measures suggest considerable psychometric overlap between instruments, but highlight that local development does provide some additional value in terms of both content and external validity [12, 21]. Findings from the current study echo this conclusion. Although the body of literature is still relatively small, the consistent advantage of locally-developed measures suggests that efforts to develop them are indeed worthwhile. However, the equally consistent finding regarding the large overlap in content and external validity suggests that decisions not to develop culturally-grounded instruments may be justified, depending on project scope and resources. We are reminded of the emphasis placed by Kohrt and colleagues’ [8] on the consideration of research purpose as key to tool selection. Where the goals of psychosocial programming in emergency contexts are short-term and relatively limited in scope, resources dedicated to developing local measures to screen for mental health are perhaps better spent elsewhere. Over the long term, however, developing culturally-grounded and locally validated emic measures will significantly improve the quality of psychosocial programming, as the small but consistent advantages of emic measures observed in our findings and others’ are likely magnified over time. For sustainable programs it is particularly important that case-identification and monitoring and evaluation tools capture salient expressions of distress [65]. In terms of research, the development of emic instruments is critical because the goals of methodological excellence and cultural specificity overshadow many other possible concerns. Using such scales alongside etic scales may open up a range of strategies to examine multiple health outcomes with multiple, complementary tools. Psychosocial programs deciding whether or not to devote the resources necessary to develop local measures of distress where there are none should weigh the purposes of the tool and the potential for long-term engagement.

Comparing the ASCL and SRQ-20 revealed substantial overlap in construct and external validity, but the ASCL captured more variance associated with trauma and poverty for women. These findings highlight the advantages of locally-developed measures of mental health and the importance of considering gender in transcultural validation. In addition, practitioners should not ignore the role of aggression in emotional distress in cross-cultural setting. Culturally relevant measures are worth developing for long-term psychosocial programming.