Introduction
In the last decade, sport scientific research became more and more interested in a construct called executive functions (EF). EF refer to a broad construct (Etnier & Chang,
2009) that encompasses a set of higher-level functions, such as mental set shifting, information updating and monitoring, as well as inhibition of prepotent responses (Miyake et al.,
2000). On the one hand, sport science investigated the effect of chronic or acute exercise on EF (Chang, Labban, Gapin, & Etnier,
2012; Lambourne & Tomporowski,
2010; McMorris & Hale,
2012; Verburgh, Königs, Scherder, & Oosterlaan,
2013), whereas on the other hand, recent research questioned the role of EF on athletic performances and sport expertise (Jacobson & Matthaeus,
2014; Krenn, Finkenzeller, Würth, & Amesberger,
2018).
In the latter case, seminal findings were published by Vestberg, Gustafson, Maurex, Ingvar, & Petrovic (
2012), who found soccer player’s EF to be significantly predictive for the number of goals and the number of assists players scored two years later. In addition, they detected higher EF in players of the highest Swedish national league in comparison to soccer players of the second and third Swedish national division and that all soccer performance level groups scored significantly above population norm. As a consequence, EF were considered as key indicator of athletic performances in team sports (Lundgren, Högman, Näslund, & Parling,
2016; Vestberg et al.,
2012). Subsequent research was able to corroborate this significant role of EF mainly in soccer: It was found that elite soccer players showed higher EF than subelite players (Vestberg et al.,
2012; Vestberg, Jafari, Almeida, Maurex, Ingvar, & Petrovic
2020), that ambitious soccer players showed higher EFs than the population norm (Vestberg et al.,
2012) and that player’s EF correlated significantly with their scored assists (Vestberg et al.,
2012,
2020) and scored goals (Huijgen et al.,
2015). Hence, these studies provide alleged evidence that above-average cognitive performance is associated with soccer expertise. Further, EF were suggested to represent a cognitive measure of game intelligence in team sports like ice hockey (Lundgren et al.,
2016) and soccer (Vestberg et al.,
2020).
The above described findings signifying the role of EF in team sports, predominantly in soccer, mainly are based on the Design Fluency Test (DFT) claiming to measure EF (Huijgen et al.,
2015; Lundgren et al.,
2016; Vestberg et al.,
2012; Vestberg, Reinebo, Maurex, Ingvar, & Petrovic
2017). The DFT was originally developed for the assessment of fundamental skills and higher-level executive functions in clinical populations of children, adolescents, and adults and is one element of the Delis–Kaplan Executive Function System (D-KEFS) (Swanson,
2005). Rows of boxes, consisting of the same array of five dots, are presented. Within each box, different designs have to be generated by connecting the dots using four straight lines. The participants are required to draw as many different designs as possible within 60 s. There are three conditions which differ in the properties of the dots. In condition 1 (C1) all dots are filled, in condition 2 (C2) the dots are empty, and in condition 3 (C3) the dots are alternately filled and empty. According to the D‑KEFS manual (Delis, Kaplan, & Kramer,
2001b), C1 provides a basic test of design fluency. C2 requires design fluency and response inhibition caused by the change from filled to empty dots. C3 is developed to assess design fluency and cognitive flexibility through switching between filled and empty dots. Suchy, Kraybill, and Larson (
2010) emphasized that C3 captures a separate construct that needs to be considered when interpreting DFT results. In general, it is stated that the DFT measures the “initiation of problem-solving behavior, fluency in generating visual patterns, creativity in drawing new designs, simultaneous processing in drawing the designs while observing the rules and restrictions of the task, and inhibiting previously drawn responses” (Swanson,
2005, p. 122). It is claimed by many researchers that such skills are crucial for success in several team sports (Furley & Memmert,
2010b; Tillman & Wiens,
2011). In soccer, for instance, “a successful player must constantly assess the situation, compare it to past experiences, create new possibilities, make quick decisions to actions, but also quickly inhibit planned behavior” (Vestberg et al.,
2012, p. 4). Huijgen et al. (
2015) emphasized that “a soccer player must be able to quickly anticipate and react to fast changing situations that occur during a soccer match” (p. 2). Based on these soccer-specific demands, Vestberg et al. (
2012) argued that the DFT is appropriate for the assessment of EFs associated with success in soccer because it challenges similar EFs as in typical soccer game situations. However, taking DFT’s clinical origin into consideration its forthright application in the sample of elite athletes seems challenging and makes the highest demands on its psychometric properties.
In contrast to the findings on the DFT and soccer expertise (Vestberg et al.,
2012,
2020,
2017), Furley, Schul, and Memmert (
2017) pointed out that findings of improved cognitive performance in expert soccer players is anything but consistent. Several studies failed to provide evidence of superior executive functions in experts (e.g. Furley & Memmert,
2010a,
2015) or showed equivocal findings (e.g. Verburgh, Scherder, van Lange, & Oosterlaan,
2014). These inconsistencies are discussed against the background of confounding variables, sample sizes, expectation of the researchers, and definition of expert (Furley et al.,
2017), as well as under the perspective of the need for reliable measurements (Schweizer, Furley, Rost, & Barth,
2020).
In previous studies in sport, the DFT was used based on the test criteria provided by the D‑KEFS manual (Delis, Kaplan, & Kramer,
2001a). However, according to Homack, Lee, and Riccio (
2005), much research has to be done in order to fully determine the psychometric properties of the DFT. Likewise Shunk, Davis, and Dean (
2006) concluded that the psychometric properties of the DFT were not well established. Although the D‑KEFS manual provides data on reliability of the DFT, these data refer to a heterogeneous sample in terms of age and other demographic characteristics. Furthermore, the period between test and retest was not kept constant within a time range of 9 to 74 days. Additionally, the D‑KEFS manual consists of information on validity that incorporates intercorrelations of measures, and differences between Alzheimer and Huntington disease patients (Delis et al.,
2001a). Therefore, there are open questions on short-term and long-term test–retest reliability and practice effects in general, as well as questions concerning the use of the DFT in athletes in particular. Results on the stability of the rank order of participants in short- and long-term intervals are necessary to be able to evaluate findings on the prospective value of DFT performance. Further questions concern the differential and prospective value of the DFT in team sports in order to expand the existing knowledge on the diagnostic power of the DFT for application in team sports.
As the DFT was designed to detect dysfunctional EF in clinical populations, its application in elite athletes asks for strong evidence for a reliable and valid assessment of EF in this highly skilled sample. So far, research has failed to provide this clear evidence of reliability and validity. The current study aimed to contribute to this target and to enhance the evidence about the assessment of the DFT in sports for practitioners and researchers. We assembled different data sets collected in the applied field of sport psychology to enable a broad and differential analysis of the psychometric properties of the DFT.
The first aim of the present study was to determine reliability of DFT scores. Short-term test–retest reliability of DFT scores were assessed in three samples having different activities of varying duration between test and retest. Thus, reliability was examined in varying contextual situations to enable estimation of the effect of different sources of bias between measurements. Using a correlational coefficient approach, the relationship between test and retest from individual values was evaluated to show how well the rank order of participants in the first test was replicated in the retest (Hopkins,
2000). Additionally, changes between test and retest were examined to assess non-random effects, resulting from i.e. activities between measures, motivational and learning processes (Hopkins,
2000). Long-term test–retest correlation and systematic change of DFT scores between measurements were evaluated in a sample of national volleyball team players who were tested twice within a year.
The second aim was to determine the differential value of the DFT. Previous research (Vestberg et al.,
2012,
2017) showed higher DFT scores in the sum of correct designs in soccer players compared to normative data (Delis et al.,
2001b). This study focused on differences between adolescent elite soccer players and high-school students. In contrast to previous studies (Huijgen et al.,
2015; Vestberg et al.,
2012,
2017), all single and composite DFT performance scores were considered in order to reflect DFT performance in a complex manner. Based on the findings by Vestberg et al. (
2017), we expected that elite athletes would show a higher total sum of correct designs in the DFT, compared to the student group.
The third aim addressed the prospective value of the DFT in national team volleyball players in order to determine the extent to which the results transfer across different types of ball sports (Vestberg et al.,
2012,
2017). Past research suggested that playing soccer attaches high demands towards EF (Vestberg et al.,
2012,
2017). However, also volleyball as open-skill sports and strategic sport disciplines seem to make high demands on EF (Alves et al.,
2013; Jacobson & Matthaeus,
2014; Krenn et al.,
2018; Montuori et al.,
2019). Taking several cognitive similarities between both sports into account (e.g. focusing on the ball and movement patterns of team players and opposing players; keeping tactical information and experiences about team members and opponents in mind; adapting to continuously changing situations and creating new ways to solve upcoming problems on the court; cf. Alves et al.,
2013), we assumed significant correlations between DFT scores and performance parameters in volleyball. In addition, we expected higher correlations between DFT scores and more compounded and broader performance parameters (e.g. total points scored, attack errors), which should rely more heavily on the EF concepts of inhibition, working memory and cognitive flexibility than more specific performance parameters (e.g. serve aces and serve errors).