Validity of the school setting interview for students with special educational needs in regular high school – a Rasch analysis

The concept of special educational needs is internationally used to describe students who temporarily or permanently experience difficulties in their learning. The concept covers students with and without a diagnosis, and includes up to 20% of school-aged youth [1]. Common diagnoses among students with special educational needs are Attention Deficit Hyperactivity Disorder (ADHD), Asperger’s syndrome and dyslexia [2]. Symptoms of these diagnoses often include difficulties to concentrate, follow instructions, organize and conduct tasks, and/or read and write [2‐4]. Struggling to complete education successfully may lead to lower self-esteem and poor overall mental health [2, 5], along with higher rates of school-dropout and unemployment due to low academic achievements [2, 6]. Compared with students with physical disorders or without disabilities, students with developmental disorders (ADHD, autism or dyslexia) rate their perceived quality of life significantly lower [7, 8]. Adolescents’ participation in education at school is a vital component of their everyday life and a determinant of health, development and well-being as well as future opportunities in adult life [9]. To achieve full potential for occupational participation in education, students’ opportunities as regards their learning conditions and engagement in school activities are critical [10]. Students should be provided with appropriate adjustments and support [11] in areas such as writing, reading, knowledge gathering, and practical tasks, as well as support in initiating and organizing school activities and reminders to perform tasks at appropriate times [12]. Focusing on the social environment and the learning context, e.g. communication and participation within the classroom, is also important to emphasize [13]. Student’s unique abilities and the characteristics of the specific school environment must be considered and should form the basis for adjustments to give students with different types of disabilities equal opportunities for participation in education [14].

By assessing the student-environment fit, the interaction between the student and the school environment, information reflecting the student’s occupational performance is generated. The School Setting Interview (SSI) [15] is an assessment instrument that assesses student-environment fit and is theoretically based on the Model of Human Occupation [10]. The SSI consists of a student-centered interview and an accompanying rating scale including 16 items of everyday school activities. The assessment takes about 40 min to complete and identifies possibilities, hindrances and potential needs for adjustments concerning students’ participation in school activities [15].

The SSI was initially developed for students with physical disabilities [16] and a psychometric study supported evidence of construct validity. However, the study revealed a need for more challenging items and a refinement of the scoring in the rating scale [17], which thereafter was developed from a three-step to a four-step rating scale in version 2 [18]. Findings and input from professionals with experience of the SSI were used to develop the SSI to make it applicable to students with other difficulties than just physical [19]. Even though the SSI, version 3.1 [15], has been successfully used for students with different difficulties, the construct validity has not yet been evaluated for students with special educational needs. This calls for psychometric evaluation of the internal construct validity of the scale. Depending on the characteristics of the student, different school activities and environmental factors are experienced as more or less challenging. It is hypothesized that students with special educational needs, often involving difficulties with concentration, organization and finishing tasks, will experience more challenges, i.e. low student-environment fit, in school activities such as “Remember things”, “Read”, “Take exams” and “Write”. On the other hand, they may experience high student-environment fit regarding “Access the school” and “Go on field trips”.

The Rasch measurement model is a mathematical approach belonging to modern test theory [20], and has become established as the standard for modern psychometric evaluations of outcome scales [21]. The analysis addresses several measurement issues and aspects in addition to those in classical test theory. It transforms ordinal data to an interval-level variable for detailed investigation of the structure and operation of rating scales [21, 22].

The aim of the present study was to investigate the psychometric properties of the School Setting Interview (SSI) for students with special educational needs in regular high school. More specifically, the aim was to examine whether the SSI items are valid for this group of students, with additional consideration on bias of items by gender and diagnosis, the measurement properties of the SSI rating scale, and whether the SSI measures a unidimensional construct.

This study was conducted to examine the measurement properties of the SSI among students with special educational needs in regular high school, a previously unevaluated field of use of the SSI. Data were found, in general, to be in accordance with the Rasch model, providing support for the internal construct validity of the SSI. However, areas for potential improvement were also identified where the SSI rating categories with the disordered thresholds are of most concern.

The findings indicated that the SSI captures three different levels of student-environment fit. The minimum requirement of this kind of assessment is to separate people into two groups (high and low level of attribute) [32]. Since the SSI is able to separate about three levels of performance, the sample may be classified into meaningful categories of high, medium and low experienced student-environment fit. The SSI three-step rating scale in earlier versions also separated students into three levels of student-environment fit and the new, four-step, rating scale was proposed to increase the separation [17]. This was not confirmed in the present study, perhaps because this new group of students had a different type of difficulties than previously investigated. Positively, the sample presented a good, overall, person-fit and the overall item-fit was acceptable. In clinical practice, it is important that the assessment is appropriately targeted to the population being assessed for adequate measurement [22, 25, 26] and satisfyingly, the targeting of the SSI items was good for this new group of students. Item 4 was the most difficult item for the sample, and with respect to the existing difficulties among the students, this was in accordance with the hypothesis. However, some more difficult items would probably increase the psychometric properties of the assessment [23] since the present sample exhibited a higher mean location than included items. Item 2, “Read”, showed signs of misfit, though these were non-significant. This observation may not convert into other populations, but if so, the misfit should be thoroughly investigated since reading is a highly important activity in school, and thus an essential item in the SSI.

Support for unidimensionality and local independence was satisfying since both multidimensionality and response dependency are serious threats of the psychometric properties of an assessment and implies that responses to an item depends on responses to other items or that the scale reflects more than one latent trait [26, 31].

The SSI rating categories did not work as intended for this group of students since all items had, more or less, disordered thresholds. Investigation of the response distribution confirmed that category 1 (score of 2 - partial fit, student has adjustment but additional adjustments are needed) and 2 (score of 3 - good fit, student has necessary adjustments) were rarely used. Both categories imply that adjustments have already been made in the environment to increase the student-environment fit. Thus, this indicates that students had not received needed adjustments, which was also demonstrated earlier for this group of students in Swedish schools [33] as well as internationally [19]. Even though laws and legislation [9, 34] entitle all students to participate in and acquire education, students with special educational needs do not seem to sufficiently obtain individualized support which would enhance and improve their performance and participation in school activities as a result of improved student-environment fit. Item 14 “Get assistance”, had the best category functioning and the most equal distribution of scores among the different categories. This indicates that if all assigned categories are not used, in this study due to absence of environmental adjustments, it might be a reason for disordered thresholds [35, 36]. Despite issues with the categorization of the scale, it is essential for the clinical utility of the SSI to keep all four categories since it is used both as an assessment of level of student-environment fit and as an evaluation of implemented adjustments [15]. Category 1 (score of 2) and 2 (score of 3) are of great value to see whether implemented adjustments fulfilled their purpose or whether the students are in need of further adjustments. The present study was based on data from an initial SSI assessment and further investigation of the category functioning should include data from SSI evaluation of implemented adjustments. Thus, further studies are necessary to investigate the underlying cause of why two of the four categories are not used as intended. If the reason for non-use of category 1 (score of 2) and 2 (score of 3) is not due to absent adjustments, the scale needs to be revised.

Item 4 “Remember things” showed uniform DIF regarding gender and item 6 “Homework” showed DIF among students with and without diagnosis. When DIF is present the observed group differences, at least partially, reflect something other than the latent trait [21, 37, 38] and comparison between boys and girls or between student with and without diagnosis is not completely reliable in the specific items. ANOVA statistics and the ICC curve were used in the present study since DIF analyses’ actual power is affected by sample size, the distribution of persons in relation to the location of items and the distribution of residuals [38]. Additionally, an item split was performed in order to investigate the DIF further [23, 38]. DIF does not necessarily imply clinical significance, and theoretical and practical issues should be considered before adjustments are made [39]. Removal of item 4 was not considered an option since much school activity, as well as everyday life, involves executive functions such as planning and remembering. The same applies for item 6 as homework is an integral part of schooling. Since the item split confirmed that the DIF did not affect the overall psychometric properties, the decision to keep the original SSI items was supported. Item Remember things might have different meanings for girls and boys, which might be reflected in the scoring of the item. Girls are generally more committed in their schoolwork [39] and spend more time doing school-related activities than boys [40]. This may indicate that they more often have self-initiated strategies for planning and remembering details related to school activities, which might have caused the detected DIF. What caused the DIF in item Homework might be related to that having a medical diagnosis is associated with increased odds of receiving support in Swedish schools [41]. The support may include specific assistance regarding the whole school activity of Homework, such as planning and modifications of tasks or instructions. This assistance might have influenced the higher scoring among students with a diagnosis, even though the students had the same level of student-environment fit as those without a diagnosis. The detected DIF should be investigated further to evaluate its clinical effect and whether they are also present in other samples.

The findings provide support of construct validity of the SSI for use among regular high school students with special educational needs in order to provide and evaluate environmental adjustments. Assessing student-environment fit to be able to provide environmental adjustments for students is essential, and for this evaluation the SSI could be used as a valuable tool by personnel at schools and student health units. However, the items with the detected DIF and the SSI rating scale with its disordered thresholds needs to be further scrutinized.