Structural validation of three German versions of behavioral and motivational scales in high-risk sports

Following institutional approval by the Board for Ethical Questions in accordance with the Declaration of Helsinki, we collected the data using a web-based questionnaire in a cross-sectional design. We recruited participants from a number of different high-risk sports via emails to students and employees of the University of the first author and local sports clubs (e.g., paragliding association). All participants completed the survey online. Participants not finishing the survey were excluded from analyses.

The final number of participants was 719 (25% female), with the highest numbers of individuals performing various disciplines in paragliding (59%). Further, the sample consisted of freeride skiers (14%), mountain trail runners² (16%), freestyle skiers/snowboarders (7%), as well as mountaineering athletes (4%). The participants had a mean age of 35.4 (±11.6) years and reported an average of 7.3 (±6.1) years of experience. Age and experience varied between sport activities, with the youngest age and lowest years of experience in freestyle skiing and snowboarding (age: 23.0 ± 3.8 years; years of experience: 6.0 ± 3.6 years) and the oldest age and most years of experience in paragliding (age: 38.9 ± 11.4 years; years of experience: 7.9 ± 6.5 years). Female participation ranged from 19% in freestyle skiing/snowboarding to 32% in mountain trail running.

We translated the items following the guidelines of Guillemin, Bombardier, and Beaton (1993). The items were translated into German by a group of Sports Science Masters students, who were all fluent in German, and they were asked to note any remarks and questions while translating. The group met to discuss items and phrases until full consent was reached. In the next step translated and original items were sent to three experts from the field of health psychology and sport science who were equally fluent in both English and German language. They were asked if the German items were an accurate translation of the original English items, when they identified problematic items we modified them and the process was repeated until the experts agreed that all of the German items accurately represented the meaning of the original English ones.

We tested the hypothesized factor structure using Bayesian structural equation modelling (BSEM) in Mplus 7 (Muthén & Muthén, 2012), estimating BSEMs with weakly informative priors for approximate zero cross-loadings and residual correlations, as recommended by Muthén and Asparouhov (2012), for each inventory. Each BSEM was estimated using the Markov chain Monte Carlo (MCMC) simulation procedure, using the Gibbs sampler over 200,000 iterations across two MCMC chains in order to assess model convergence and stability of estimates.

Model convergence can be assessed in a number of ways (Kaplan & Depaoli, 2012). In this study, we used the Gelman-Rubin convergence diagnostic (potential scale reduction factor; PSR) and Kolmogorov-Smirnov (K-S) tests to assess model convergence in addition to visual inspection of all parameter trace plots. Model convergence is evident when the PSR value approaches one following the warm-up period (i.e., the first half of the iterations) (Gelman et al., 2014), when K‑S tests show no significant differences between the estimates of parameter distribution across the chains, and when parameter values in each MCMC chain mix well (i.e., have stable means and variances (van de Schoot & Depaoli Sarah, 2014)).

We assessed model fit by examining: factor loadings, the posterior predictive p (PPp) value and the results of likelihood χ² tests, which examine differences between the model generated and the observed data. Excellent model fit is indicated by PPp values of approximately 0.50 and with a symmetric 95% credibility intervals (CI) centered around zero (Muthén & Asparouhov, 2012). We assessed internal consistency using the composite reliability coefficient (Fornell & Larcker, 1981) and discriminant validity using the latent variable correlations obtained from the BSEMs.

Where good model fit or convergence was not reached we re-examined the items in each scale to identify which items were problematic. To improve model fits, we removed items from each inventory; items were considered for removal based on the following criteria: having low factor loadings (< 0.6); theoretical (ir)relevance; having substantive cross-loadings or correlated residual variances (> ± 0.2); parameters with significant K‑S tests; the highest PSR; or trace plots which showed poor mixing of the MCMC chains. Once identified, we removed these items and re-estimated the BSEM. We repeated this process until the final model for each inventory was both statistically and theoretically sound; it was important that item removal could be justified both statistically and theoretically to avoid making modifications based on sample specific, chance characteristics of the data, which then may not represent the relationships among variables in the wider population (Biddle, Markland, Gilbourne, Chatzisarantis, & Sparkes, 2001).

Given that Barlow et al. (2013) found a two-factor structure to be most appropriate for the SEAS After inventory, we analyzed our data with two- and three-factor models in order to see if the translated version confirmed these initial findings. In addition to the criteria mentioned above, we used the Deviance Information Criteria (DIC) to compare the two- and three-factor G‑SEAS After inventory as recommended by Asparouhov, Muthén, and Morin (2015).

Once the final models were established we performed a sensitivity analysis as the choice of priors can affect the parameter estimates (Muthén & Asparouhov, 2012; Stenling, Ivarsson, Johnson, & Lindwall, 2015). To do so, we re-ran the final models for each inventory with smaller (0.05) and larger (0.015) priors for the cross-loadings, before comparing the parameter estimates for discrepancies between these models and those estimates with a prior variance of 0.01. We also weighted items with their respective factor loadings and then calculated Pearson’s correlation coefficients to examine the relationships between subscale means.

Visual inspection of trace plots for all parameters supported convergence (i.e., showed good mixing and no upward or downward trends). In addition to this the PSR value for all models fell below 1.1 during the warm-up phase of the simulations and remained below 1.1 for the remaining iterations (Table 1), and there were no significant K‑S tests for any of the five models.

For each G‑SEAS inventory, the initial 18-item BSEM models with non-informative priors achieved adequate convergence and all items had significant loadings on their intended factors only. However, there were a myriad of other problems that meant the models were not deemed to be suitable. In the Between inventory, two items had problematic correlated residuals and one low major factor loading (0.49). In the While inventory, there were ten problematic correlated residuals and three major factor loadings were low (0.53–0.58). The three-factor After inventory had five problematic correlated residuals and one low major factor loading (0.56). The two-factor After model had one low major factor loading (0.56) and had 17 correlated residuals that exceeded their a priori limits. Through an iterative process, four items were removed from each G‑SEAS inventory using the criteria outlined earlier (the remaining items can be seen in Tables 2, 3 and 4).

The BSEMs for the three 14-item G‑SEAS inventories, the G‑ACCSI, and the G‑RTI with informative small variance priors for cross-loadings and residual correlations have excellent fit with PPp values of approximately 0.5 and having symmetric 95% posterior predictive CIs centered on zero (Table 1). The major factor loadings in each inventory were significant, acceptable, and in the hypothesized direction. Furthermore, there were no cross-loadings that exceeded their a priori limits (Tables 2, 3, 4, 5 and 6). There were no correlated residuals that exceeded their a priori bounds in the Between, While, three-factor After participation G‑SEAS, G‑RTI, or G‑ACCSI. However, there were four correlated residuals that exceeded their a priori limits in the two-factor After inventory all in the Agentic Emotion Regulation factor; correlated residuals within a factor indicates that there is shared variance that is unaccounted for by the model (e.g., there is another latent factor influencing the data).

The DIC for the 14-item two-factor After model was 19,097.742 and the DIC for the 14-item, three-factor After model was 19,096.132, lower than that of the two-factor model despite having 16 more parameters. Despite the differences in DIC being small, the three-factor After model is a better fit to the data as there are no correlated residuals that exceed their a priori limits.

Sensitivity analyses for each inventory showed that the factor loadings and cross-loadings were relatively stable when specifying smaller (0.005) and larger (0.015) a priori variance priors. Using both smaller and larger prior variances 100% of discrepancies across the three inventories were within ±0.05, the maximum discrepancy was −0.043 with smaller prior variances set; and the maximum discrepancy was −0.029 with larger prior variances set.

Table 7 shows the latent factor subscale means, standard deviations, composite reliabilities and latent factor intercorrelations for the three G‑SEAS inventories, the G‑RTI, and the G‑ACCSI. The composite reliability of each subscale resulted in r > 0.8 across each inventory and the subscales within each inventory were all positively correlated. The G‑RTI subscales had a weak inverse relationship and the Accidents and Close Call subscales were positively correlated (0.61). None of the 95% CIs for interfactor correlations encompassed 1.00, thus, supporting the discriminant validity of the subscales within each inventory (Anderson & Gerbing, 1988).

Table 8 shows correlations between subscale means, where the items were weighted with factor loadings. Sensation seeking of the Between participation G‑SEAS was positively correlated with DRT, accidents and close calls and negatively with PB and age. Accidents and close calls were positively correlated with sensation seeking, which was also negatively related to both PB and age. Accidents and close calls were both positively correlated with DRT and negatively with PB. Age displayed negative correlations with all subscales except PB, which resulted in a weak positive relation.

First, one might mention a lack of validation with established measures (e.g., the Sensation Seeking Scale). However, the concurrent validity of the inventories has been well established in the original since we validated constructs which already put those inventories in relation to established measures (Barlow et al., 2013; Woodman et al., 2013), validating the three scales with one another was seen as sufficient. Second, both the exclusion of SEAS items and structural validation was done in the same sample. This approach might result in biased estimates for structural validation. Future studies using the G‑SEAS might consider calculating indices for internal consistency and factorial structure. As discussed there are several strengths resulting from the differences in the factor structure of the G‑SEAS; nevertheless, a limiting factor is that because of those differences, the G‑SEAS and the original SEAS cannot be used to directly compare German- and English-speaking populations. This study has provided evidence for a three-factor structure for each of the G‑SEAS scales. Without collecting new data in both German- and English-speaking populations to carry out a measurement invariance analysis, it is not possible to make direct comparisons between data collected using the SEAS and G‑SEAS. We would recommend that this analysis be carried out using BSEM as the differences in factor structures of the SEAS and G‑SEAS may be due to differences in analytic methodologies. Therefore, we recommend that G‑SEAS should be used in German-speaking populations and should not be used to make direct comparisons with data collected using the SEAS. Interscale correlations do not differ between the 14-item and the 18-item German version; this suggests that despite missing 4 items, the G‑SEAS still measures identical constructs. Another limitation which is not necessarily tied to this research only is the in our opinion problematic operationalization of high-risk sports. Despite the definition we cited at the beginning of the introduction, we think future research might benefit from a more detailed definition including additional components related to skills and experience. Though this is beyond the scope of this article, it is worth mentioning that researchers in high-risk sports need to address this problem in future studies.

This study has a number of strengths, including the variation within the population; in addition to being of sufficient size, the sample included a number of different high-risk sports, participants from a variety of experience levels and of different ages and sexes, thus, ensuring a heterogenic sample population. The analytical methods used in this study should also be seen as another strength of this article. It has been shown that BSEM is a more appropriate method than traditional CFA methods as it better reflects the complexities of reality (Niven & Markland, 2016; Stenling et al., 2015).