The nutrition for sport knowledge questionnaire (NSKQ): development and validation using classical test theory and Rasch analysis

The research was approved (S16/267) by the La Trobe University’s SHE College Human Ethics Sub-Committee (SHE CHESC). Participants in the expert panel review and think-out-loud focus groups read the Participant Information Statement and signed the Consent Form. Participants who were involved in steps six to eight read the Participant Information Statement and provided consent electronically.

A novel eight-step validation method for the development of a nutrition knowledge questionnaire was designed based on an extensive review of the literature and used to validate this questionnaire [35]. The steps include: (1) Definition of Sports Nutrition Knowledge (2) Generation of items to represent sports nutrition knowledge (3) Choice of scoring system (4) Assessment of content validity by panel of experts (5) Assessment of face validity by student athletes (6) CTT analysis: Removal of items on the basis of item difficulty, item discrimination and distractor utility (7) Rasch analysis: Assessment of dimensionality and removal of item on the basis of not meeting assumptions that difficult questions are less likely to be answered correctly, and well-scoring participants are more likely to answer individual items correctly (8) Assessment of construct validity by comparing nutrition and non-nutrition students; assessment of test-retest reliability (consistency over time) by assessing correlation of test on two attempts; and re-checking of steps six and seven. The steps that make this methodology novel are the quantitative assessment of content validity, the assessment of distractor utility (how feasible incorrect multiple choice options are) and the inclusion of Rasch analysis.

Figure 1 provides a summary of the methods and results.

Experts (step four) were recruited using purposive sampling between April and May 2016 and student-athletes (step five) were recruited using convenience sampling in June 2016. For steps six and seven, Australian Football League (AFL) Victoria community football players, La Trobe University student-athletes and other recreational athletes were invited to complete the questionnaire via email, Facebook groups, and online athlete forums. Data collection occurred between July 2016 and October 2016. For step eight, La Trobe University undergraduate and postgraduate business, health science and nutrition students were recruited via email and their Learning Management System (LMS) notice boards and recreational athletes were invited to participate via email and their team Facebook pages. Data collection took place from November 2016 until January 2017.

To calculate a CVI (step four) three to 10 experts are needed [36]. The ideal number of participants for focus groups (step five) are six to 10 [37]. Parmenter et al. [38] recommend that to carry out CTT analysis at least one more person than the number of items are required. Pallant [23] recommends that to carry out Rasch analysis, 240 participants are ideal. At steps 6 and 7, the questionnaire had 178 items; a target of 200 participants was set to account for both the CTT and Rasch estimates. For step eight, a power analysis for an independent sample t-test was conducted in G-POWER to determine a sufficient sample size using an alpha of 0.05, a power of 0.80, a large effect size (d = 0.8), and two tails. Based on these assumptions, the desired sample size for each group (nutrition versus non-nutrition students) was 51 [39, 40].

Missing values were assessed for normality using Kolmogorov-Smirnov statistic and compared across participant characteristics using independent t-test and ANOVA or Mann–Whitney and Kruskal-Walis, as appropriate (see Additional file 1: Supplementary Material S1). Total and sub-section scores were assessed for normality using Kolmogorov-Smirnov statistic. Parametric tests (independent t-test; ANOVA) were used for normal data. Where assumptions of normality were violated, non-parametric tests (Mann–Whitney U test; Kruskal-Wallis) were used. Normal results were reported as mean ± SD, non-normal results were reported as median and IQR. For correlations, Pearson’s r was used for parametric data, and Spearman’s r was used for non-parametric data. Individuals with and without nutrition education were compared across participant characteristics using chi-square test, to account for potential confounding. Differences in scores based on age, gender, country of birth, level of education, and history of playing sports were also assessed.

Ten experts were invited to be involved in the content validity assessment. Six sports dietitians agreed to participate and three of these returned feedback forms; two respondents were Australian and the other was Swiss. They worked in private consultancy, research and education, and industry. Eight students participated in the retrospective think-out-loud focus group. One student who was unable to attend the session met with the researcher on a separate occasion. For the item analysis based on CTT and Rasch scaling (steps six and seven), 462 athletes started the questionnaire; after excluding data with more than 11% missing values (n = 259) and participants who did not meet the eligibility criteria for age (n = 15) there were 188 usable responses (Table 1). For the assessment of construct validity, test-retest reliability and re-evaluation of CTT and Rasch parameters (step eight), 287 students and athletes started the questionnaire and there were 181 usable responses, including 28 responses from individuals who had completed the questionnaire on two occasions (Table 2).

There did not appear to be any differences between individuals who completed and did not complete the questionnaire (See Additional file 1: Supplementary Material S2).

Individuals who studied nutrition were more likely to be female, have a tertiary education and were younger (See Additional file 1: Supplementary Material S3).

Sports nutrition knowledge was defined as “Knowledge of concepts and processes related to nutrition for optimal athletic performance including knowledge of weight management; hydration and fuelling strategies for before, during and after training/performance; supplementation and alcohol use”. The original test plan (see Additional file 1: Supplementary Material S3) follows logically from this. Each correct answer was awarded a point. The final questionnaire had 89 items and six unidimensional sub-sections; weight management (n = 13); macronutrients (n = 30); micronutrients (n = 13); sports nutrition (n = 13); supplements (n = 12) and alcohol (n = 8). The sports nutrition section covers hydration, the pre-completion meal, nutrition during exercise and recovery nutrition. The item response formats include agree/disagree/not sure, multiple choice, and effective/not effective/not sure. The questionnaire is designed to be administered online and includes pictures to reduce responder fatigue. The number of items at each stage of validation is represented in Fig. 1.

The experts reported that the questionnaire covered all relevant topics and therefore content validity was confirmed (step four). Most items that were rated poorly for accuracy, relevance, appropriateness and clarity were removed (n = 25) although some exceptions were made. The number of items that did not meet the CVI and examples of changes made based on expert’s feedback are included as Additional file 1: Supplementary Material S4.

Student athletes were able to identify what each section was attempting to capture. Where there were issues with wording of questions, items were removed (n = 17) or modified (n = 16), so face validity was confirmed (step five). Examples of changes made based on students’ feedback are included as Additional file 1: Supplementary Material S5.

Construct validity was demonstrated (step eight) because there was a significant difference in the total scores for those who had nutrition education and those who did not have nutrition education (P < 0.001) (Table 3). Individuals with nutrition education also scored better across all sub-sections except alcohol (Table 3). No other participant characteristics were found to have a significant effect on scores (See Additional file 1: Supplementary Material S6).

Internal reliability of sub-sections ranged from 0.51 to 0.78 (Table 3). All sections except alcohol achieved or approached adequate values for internal reliability given the number of items in each sub-section. Test re-test reliability ranged from 0.35 to 0.81; all sections except supplementation achieved or approached adequate values for test-retest reliability (Table 3).

Item analysis using CTT was carried out at steps 7 and 8. Most items that did not meet the criteria for item difficulty, item discrimination and distractor utility were removed or modified, although some exceptions were made, and the final tool included 15 items that did not meet these criteria. Examples of changes made on the basis of item analysis are included as Additional file 1: Supplementary Materials S7 and S8.

Based on the Rasch analysis conducted on the tool administered at step eight, the questionnaire as a whole was multidimensional. Eleven items were removed because they were shown to be poor discriminators and/or were answered differently (demonstrated DIF) on the basis of age or gender. With these items removed, the questionnaire as a whole was still mutli-dimensional, but the six sub-sections were shown to be unidimensional (Table 3). Each sub-section also had an adequate mean, SD for the overall item/trait interaction statistics, meaning that the expectation that difficult items were more likely to be answered incorrectly and well scoring participants were more likely to answer questions correctly was met. The chi-square probability for macronutrients, micronutrients and alcohol was significant (Table 3); this occurred because some items that did not meet the Rasch indicator requirements were kept because they were deemed important to assess in an SNKQ.

Due to the limitations with existing SNKQs, researchers have raised concerns regarding the accuracy of previous reports of athletes’ nutrition knowledge, and have postulated that the relationship between nutrition knowledge and dietary intake may have been misjudged [8, 14, 41, 42]. The aim of this study was to create an SNKQ that tested awareness of current consensus recommendations, was adequately validated and could be used with athletes from a range of sports. The newly developed tool, the Nutrition for Sport Knowledge Questionnaire (NSKQ), has 89 individual items (44 questions, with some having multiple parts) covering six distinct subsections The questionnaire takes around 25 min to complete and is comparable in length to the GNKQ [40], which has 113 items and the SNKQ developed by Zinn et al. [43], which has 88 items. Since the questions are based on consensus guidelines, several items assess theoretical knowledge. Questions that assess practical knowledge have also been included. The tool is more comprehensive than existing measures as it includes an alcohol sub-section and adheres to a detailed test plan. The questions are based on current guidelines. For example, rather than ask about carbohydrate requirements as % total calorie intake, we have included a question on requirements in g/kg/day and specified the type (‘endurance’) and intensity (‘moderate to high’) of activity. Likewise, in contrast to other existing tools, our hydration question reflects findings that using thirst to judge fluid needs can maximise performance and current recommendations that hydration plans should be tailored to the individual [31, 44]. Moreover, questions on the timing of the pre-completion meal and recovery snack have intentionally been omitted. It is too difficult to assess the correct answer to these questions given the increasing evidence to support the positive benefits of periodizing nutrition based on the goals of individual sessions and overall training schedule [4, 28].

The questionnaire has demonstrated face and content validity based on student-athletes and sports dietitians’ judgements. In contrast to previous tools, the content validity has been assessed quantitatively, using a CVI [36]. Individuals who reported undertaking studies in human nutrition achieved higher scores across all sections, except alcohol, indicating that the questionnaire has good construct validity. The group with a nutrition education were younger and were more likely to be female and tertiary educated. In contrast to previous reports, there was no significant difference in performance based on age or gender [41, 40]. Therefore, the variations in knowledge between groups are unlikely to be due to underlying differences in participant characteristics. Test-retest reliability was assessed based on the correlation between individuals’ (who repeated the questionnaire) test scores. A limitation of this method is that motivated individuals may upskill between attempts. The average total score was higher for attempt two. Nevertheless, overall test-retest reliability was high, and all sub-sections, except ‘supplements’, achieved (or approached) adequate test-retest reliability. Participants performed most poorly on the supplement section. Therefore, it is feasible that the supplement test-retest result occurred due to participants guessing answers. The overall internal reliability was very high, and the internal reliability of most sub-sections (except alcohol) reached or approached the requisite 0.7 value. As expected, there appeared to be a relationship between the number of items and KR-20. Streiner [18] recommends that KR-20 be interpreted with caution if there are more than 20 items; the overall scale and macronutrient sub-section exceeded this value.

Based on the Rasch analysis, the overall Item/person interaction statistics were adequate, indicating compliance with the expectations that difficult questions were less likely to be answered correctly, and individuals who performed well overall were more likely to answer individual questions correctly. The questionnaire as a whole was multi-dimensional, but with problematic items removed, each section was shown to be unidimensional. Therefore, sections can be used independently, as required. Where the whole tool is used, sub-sections rather than total score should be reported.

All items were written so that units and food names were generic and likely to be understood by individuals of varying cultural backgrounds; however, additional evaluation is required to confirm the functionality of the tool in groups who differ from the present cohort. The fact that country of birth did not influence scores, and the use of Rasch analysis, which produces questionnaires that are independent of the sample used for validation, give some indication that the tool is likely to also be valid in other groups.

A limitation of this study is that we were unable to calculate response rates because we distributed the questionnaire using Facebook groups and online athlete forums, making total exposure unclear. The completion rates for step 7 (where item analysis and Rasch analysis was undertaken) were relatively low (~45%), but there did not appear to be any relationship between participant characteristics (other than sport played, country of birth) and completion rate (S2). The completion rate for step eight (~66%) was adequate [45]. The sample size is another potential limitation. For step seven, there were 188 responses for a 176 item measure; for step eight, there were 181 responses for a 100 item measure. The target sample size for both studies was 200. However, there is some evidence to recommend that samples as small as 30–50 are appropriate for CTT [46]. Similarly, Chen et al. [47] modelled Rasch with varying sample sizes and found that stable results can be achieved with samples of around 100.

A limitation of the questionnaire itself is that the length may be prohibitive, especially for athletes balancing training and work/study who are often time-poor. In addition, some items were poor discriminators. This was reflected by low item discrimination in CTT and the significant chi-square probability of the micronutrient, macronutrient and alcohol sections. For several questions, the poor item discrimination can be explained by the item’s relatively high or low difficulty index. That is, when a question is answered correctly (or incorrectly) by a large proportion of individuals, the overall range of responses is minimal, and therefore it is hard to achieve a meaningful difference between high and low scoring individuals. Many of these items were kept because they tested important concepts, providing valuable feedback on gaps in knowledge. Item discrimination is worth re-evaluating using larger samples of predominantly athletes (not including nutrition students). Likewise, future studies may focus on creating a short-form tool that can be used for rapid assessment of nutrition knowledge. A short-form tool would be useful in research settings where the correlation between knowledge and other factors is being assessed. A short-form tool may also have utility in the elite setting as a ‘screening’ tool for professionals working with athletes, i.e. to identify individuals who need nutrition education and extra support.

At present the NSKQ has only been validated in an Australian population. Future studies could focus on validation to confirm reliability and validity in other regions.

A key advantage of the questionnaire is that it has been validated using a robust methodology. To our knowledge, this is one of very few NKQ to be assessed against the Rasch Model. Likewise, it is the only tool to assess content validity qualitatively and to assess distractor utility - a distractor that is too obviously wrong will significantly increase the chances of respondents guessing a correct answer; this type of analysis is valuable. Importantly, the authors have considered the limitations of the statistics and accordingly made decisions that focused on the quality of the overall tool. In addition, the questions (and their correct answers) are based on the most recent evidence and recommendations with regards to sports nutrition; they are generalizable to most sports and enable comparison across disciplines. The tool uses food terms and measurement units that are likely to be understood by athletes from a range of countries. Moreover, the tool is detailed and therefore can assess gaps in knowledge. The NSKQ has been designed to be administered online and can provide participants with immediate feedback with regards to correct answers to questions. This is likely to be especially helpful for athletes who do not have access to professional support. The online format provides unique opportunity to direct participants to reputable and relevant resources based on their outcomes.