Explaining differential item functioning focusing on the crucial role of external information – an example from the measurement of adolescent mental health

In a consensus paper published 20 years ago the International Epidemiological Association European Questionnaire Group brought up to front the deficiencies in measurement that were characterising epidemiology:

“Epidemiological findings are often based partly or completely on responses to questionnaires. Yet the attention given to questionnaire development is often inadequate, compared with the amount of time and resources devoted to study design, population selection and analysing data.” [1].

Clearly, progress has been made since then, but there are still many examples of epidemiological and public health studies conducted without sufficient examinations of the operating characteristics of the items of questionnaires that the analyses are based on. It follows that questions are frequently raised about the extent to which trends in self-reported mental health problems are distorted by deficiencies in measurement. Similarly, while there is overwhelming evidence of higher rates of internalising self-reported mental health problems among women than men, there are yet uncertainties to what extent these differences are affected by the methodology used for collecting and analysing the data.

These uncertainties and question marks are captured by the concept of invariance, which constitutes an inevitable part of comparative research across groups and time. To make valid comparisons across genders, a requirement for measurement is that the instruments work in an equivalent way for men and women. Violations of that requirement may distort the person measures and make the comparisons between genders invalid. Such requirements of invariant measurement were stated by Thurstone [2] already in the late 1920s and Guttman [3] in the 1950s, and formally specified in a mathematical model by Rasch [4] in the 1960s.

Lack of measurement invariance across sample groups was previously called item bias [5] but is nowadays commonly called Differential Item Functioning (DIF). The scale subjected to an analysis in the present paper consists of eight items. The persons are located along a variable (also referred to as trait or construct) according to their degree of psychosomatic problems measured as a summation of the responses across all items. DIF means that one or more items in a set of items, and for an equivalent psychosomatic problem, are functioning differently for different members of a sample group, e.g. for boys and girls. DIF may appear as uniform or non-uniform. If the magnitude of DIF is constant along the entire variable, i.e. regardless of the location of the person measures on the variable, then there is evidence of uniform DIF. In contrast, if the magnitude of DIF varies along the variable there is evidence of non-uniform DIF.

Over the years a large body of literature has been published on DIF, mainly about how to identify DIF but also on how to deal post-hoc with evidence of DIF. Different procedures available for detecting DIF are described by Osterlind and Everson [6] in a monograph exclusively dedicated to DIF. Their review covers the Mantel–Haenszel procedure, methods based on Item Response Theory and logistic regression, as well as some other methods [6]. Different methods to detect DIF seem to generate similar results. In a study examining DIF in two mental health scales all three methods applied (logistic regression, the Mantel-Haenszel procedure and Rasch analysis), showed consistent results [7].

In their DIF monograph Osterlind & Everson [6] highlighted a pitfall in conducting DIF analyses:

“Sometimes, for reasons unknown, calculations of a DIF detection strategy may suggest DIF, where none truly exists” (p. 21).

While similar observations were reported previously by other researchers, no explanations were given until the last few years when the concept of artificial DIF was introduced and artificial DIF was shown to be an artefact of the procedure for identifying DIF [8] whereby real DIF favouring one group in one item induces DIF in other items favouring the other group. In order to correctly interpret a DIF analysis, real DIF has to be distinguished from artificial DIF. If DIF items are misidentified, artificial DIF items may be wrongly deleted or resolved as if they were real DIF items. This in turn may negatively affect the properties of measurement.

In a paper Hagquist and Andrich [9] summarised recent advances of analysis of DIF and suggested a unified methodology, including the distinction and identification between real and artificial DIF.

A DIF analysis starts with examining the original item set for DIF across sample groups that are to be compared, followed by a sequential procedure for distinguishing real and artificial DIF items. Among the original items and for each sample group, the item showing the greatest DIF is considered to be a real DIF item and therefore resolved by splitting the item into group specific items (e.g. one for boys and one for girls if gender is subjected to the DIF analysis). The resulting item set including two group specific items is analysed. Following the resolution of an item, it cannot induce artificial DIF in other items, and once again the item, if any, showing the greatest magnitude of DIF is considered to be a real DIF item. If necessary, this sequential procedure continues in additional steps. Having identified the item(s) showing greatest real DIF, a decision has to be made whether to retain, resolve or remove the real DIF items in further analyses. Such a decision has to be accompanied and guided by external information, which is the topic of the present paper.

Ideally, when data fit the model of analysis, multiple items which assess different aspects of the same variable increase both the reliability and validity of measurement. In practice fit and invariance, reliability and validity, do not always coincide, implying situations where a trade-off between them needs to be considered. Improved fit is dealt with by resolving the DIF item into group-specific items implying that the fit of the data to the Rasch model may be improved and the reliability retained. The invariance of the item’s parameters is, however, compromised and the validity of the variable may be affected negatively. Therefore, Andrich and Hagquist [10] suggested a rationale for deciding whether or not resolving items which improves fit and retains reliability, but violates invariance while retaining validity, may be justified:

“If the source of DIF can be understood as a result of an aspect irrelevant to the content of the variable and therefore deemed dispensable, then resolving the item and accounting for the DIF seem legitimate. However, if the source of DIF involves an aspect of the item relevant to the content of the variable and thus deemed indispensable, then resolving the item in a way that reduces the difference between the group means may seem dubious.” (p. 202).

To illustrate the complexity of the trade-off between model fit and validity Hagquist and Andrich [9] provided an example of gender DIF from a psychosomatic problems scale for adolescents:

“…But does this reasoning also apply when the gender DIF is likely to be caused by biological factors? For example, the DIF shown for the item Stomach ache in the present analyses may reflect abdominal pain because of the girls’ menstrual periods. It turns out that in dealing with this DIF a critical issue is whether this potential source of the DIF should be considered relevant or irrelevant for the conceptualisation of psychosomatic problems and its applications.” (p. 7).

They concluded that without external information about the sources of the DIF it is difficult to decide whether to resolve and take account of a DIF item or not.

The present paper is capitalising on the previous work by Andrich and Hagquist [10] and Hagquist and Andrich [9] which was referred to above. Their suggested rationale for deciding whether or not resolving DIF items assumes that the source of DIF is known. Based on a tentative Rasch analysis of a psychosomatic symptoms scale used among adolescents they hypothesised that the DIF shown for the item Stomach ache may reflect abdominal pain because of the girls’ menstrual periods. This hypothesis will be further elaborated and tested empirically in the present paper using the Rasch model analysis of DIF. The purpose of the paper is to provide an illustrative and concrete example of how external information can be used for investigations of the sources of DIF.

The study is based on data collected in the Health Behaviour in School-aged Children (HBSC) study among students 11, 13 and 15 years old [11]. The HBSC study currently includes 49 countries and regions across Europe and North America and is conducted in collaboration with the WHO Regional Office for Europe. Repeated data collections have taken place every fourth year since the 1980s. Data are collected in schools with a questionnaire which is completed anonymously in the classroom.

For the purpose of this study data from Sweden collected 2005/06, 2009/10 and 2013/14 are used. The entire data set used for this study comprises a total of 18,983 students; 6518 in grade 5, 6002 in grade 7 and 6463 in grade 9, corresponding to 11, 13 and 15 years old students. Across years, boys and girls as well as students in different grades were evenly distributed. At each year of investigation and grade the groups of students were homogeneous with respect to year of birth. The proportion of students belonging to the dominating year of birth varied between 93 and 97%.

The outcome measure subjected to the analysis in this paper is the HBSC checklist on psychosomatic symptoms, commonly called the HBSC-SCL, and frequently used in public health studies among adolescents [12]. This composite measure includes somatic as well as psychological complaints that are shown to reflect a common dimension. The term psychosomatic is used in a general sense, without making any presumptions about etiology [13].

The HBSC-SCL consists of eight items that follow the question “In the last 6 months, how often have you had the following complaints?”: Headache, Stomach ache, Backache, Feeling low, Irritability or bad temper, Feeling nervous, Difficulties in getting to sleep, Feeling dizzy.

The response categories for all of these eight items are ‘About every day’, ‘More than once a week’, ‘About once a week’, ‘About once a month’ and ‘Seldom or never’. The categories are ordered in terms of implied frequency and the higher frequency, the higher the degree of psychosomatic symptoms.

All DIF analyses of the psychosomatic scale were conducted using complete data, which reduced the sample size with 6.6%.

The following question about the menarche was included in the HBSC questionnaire:

“Have you begun to menstruate (have periods)?” with the response options yes or no.

DIF was analysed with respect to gender and the first menstrual period (menarche) among girls. Analyses of gender DIF conducted separately for each grade are briefly summarised. The period DIF is analysed at a finer level and reported in detail. Because the distribution of girls having had or not having had their first period was unevenly distributed among grades with some small sample groups within grades 5 and 9 the period DIF analyses were not conducted separately for each grade but based on the entire sample. The period DIF was analysed based on three sample groups: boys, girls not having had a first period and girls having had their first period. Because there were big differences in the sample size between these groups the sample size was randomly set to the value of 3500 for each group which was close to the original value for the smallest group. In the gender DIF analyses no adjustments were required because the groups of boys and girls were almost equally sized.

DIF-analysis is an indispensable part of psychometric analyses aiming to investigations of invariant properties of a measurement instrument. In order to decide whether or not to resolve items showing evidence of DIF, information about the source of DIF is required. The purpose of the present paper was methodological: to provide an illustrative and concrete example of how external information, in this case about the menarche, can be used for investigations of the sources of DIF. The results presented in this paper support the hypothesis that the source of the gender DIF for the item Stomach ache included in a psychosomatic problems scale is likely to be girl’s menstruation, i.e. a gender specific biological factor. Both the gender DIF and the period DIF approaches used in this study point to the same conclusion. Separate DIF analyses in each of the three grades show that there is only a small gender DIF for the item Stomach ache in grade 5, i.e. at the age about 11 when only a small proportion of the girls have had their first period. In contrast, such DIF is more evident in grade 7 (about 13 years old) when a vast majority of the girls have experienced their first period and in grade 9 (15 years old) when almost all girls have had it. Hence, consistent with the hypothesis about the menstruation as a source of the DIF, the magnitude of the DIF is greater in grade 9 than in grade 7. While the gender DIF in grade 7 appears to be mainly uniform, the DIF in grade 9 tends to be non-uniform which means that the magnitude of the DIF varies along the trait. Among grade 9 students with lower degrees of psychosomatic problems the gender DIF is more pronounced than among students with higher degrees of such problems. The analysis including all three grades simultaneously clearly shows a period DIF, with girls who have had their first period reporting stomach ache most frequently. Among those students with lower load of overall psychosomatic problems the DIF between boys and girls not having had a first period is smaller than between girls who have had their first period and those who have not had it.

From a content perspective the DIF evident for the item Stomach ache seems explainable given the available knowledge about the age for the menarche. Given that, it’s not surprising that there is only a small gender DIF in grade 5 and that the strongest evidence of gender DIF is to be found in grade 9. The indicated interaction in grades 7 and 9 between period and gender respectively, and the class intervals along the trait, i.e. the non-uniform DIF, is also tentatively explainable. Among those students experiencing higher degrees of psychosomatic problems the relative impact of Stomach ache caused by the periods may be minor because of being part of an overall pattern of more frequent complaints.

The analysis of DIF reported in this paper also shows that additional information, which is external to the measurement instrument and outside the analysis which identified it, is required in order to investigate in depth the sources of the DIF. The access to data on the menarche enabled a direct test of a specific hypothesis, in addition to the descriptive DIF-analysis across grades. Because the hypothesis was supported, it follows that in a next step a decision has to be made whether the item Stomach ache should be resolved for gender DIF or not. Following the rationale quoted above which was suggested by Andrich and Hagquist [10], such a decision should not only be guided by statistical criteria but also by a consideration about the implications for the content of the measure. It should be noted that in group comparisons, resolving an item has the same consequences as removing an item, i.e. the impact of the resolved item on the group comparisons is taken away. Because the gender specific items for Stomach ache have different location and/or slope values, the estimates for the groups are no longer invariant.

While resolving an item may improve the fit of the data to the model and retain reliability it destroys invariance of the item parameters among the groups. Resolving an item may either improve or worsen the validity, depending on whether the source of the DIF is relevant for the content of the variable or not. If the source of the DIF is relevant to the measure, resolving an item may worsen the validity and vice versa.

If a measure subjected to a DIF analysis is clearly conceptualised, insights about the sources of the DIF may be sufficient in order to judge whether the source of the DIF is relevant to content of the variable in question and to decide whether the DIF should be resolved and taken account of or not. Because the concept of psychosomatic problems is not fully defined and there is not any presumption about etiology, knowledge about the sources of the DIF is necessary but not sufficient. It turns out that individual conceptualisations of the measure will govern whether the item showing DIF should be resolved or not. In the current example it needs to be considered if abdominal pain reflecting gender specific biological conditions should be a part of the measure of psychosomatic problems. If the measure conceptually is intended to tap information about Stomach ache regardless of causes, resolving the gender DIF, or removing the item, would negatively change the content of the variable. In contrast, if complaints caused by gender specific biological factors are not considered to be relevant for the measure of psychosomatic problems, resolving the item would not just retain the reliability and improve the fit of the data to the model but also improve the content validity of the variable. In the judgement of different options to handle the DIF, the available data may turn out to be insufficient to make a decision, requiring additional external information. It follows that the current paper doesn’t end up with a suggestion about which option would be best to choose in order to handle the DIF.

While the impact on the person measurement of the gender DIF for the item Stomach ache may seem relatively small, if there are other items showing DIF in the same direction, the overall effect on person measurement will certainly increase. Another item showing evidence of gender and period DIF of about the same magnitude and in the same direction as Stomach ache is the item Felt low.

In conclusion, external information about the menarche was used to carry out an in depth analysis of the source of the gender DIF indicating that a biological factor was causing the gender DIF evident for the “item frequency Stomach ache”. The present paper also shows that in order to decide whether to resolve the DIF or not additional information may be required if the concept in question is not clearly defined. However, it may also be considered that regardless of the source of DIF, that invariance should take precedence over fit and that the item should not be resolved.

Finally, some remarks are in order. As stated above, the purpose of the present paper was methodological. The available external data on the menarche worked properly to provide an illustrative and concrete example of how external information can be used for investigations of the sources of DIF. While the psychosomatic problems scale is considered causal (reflective) [17‐19] in that the degree of problems governs the probability of certain responses to each item and the analysis support the hypothesis about the menarche as a cause of the gender DIF, further quantitative and qualitative information is required in order to make firm conclusions about the causes of the gender DIF and to exclude alternative hypotheses. Also, additional data are required to examine possible interactions between period DIF and other variables, e.g. grades (age) as well as to identify possible interactions with year of investigations.