Reproducibility and validity of food group intake in a short food frequency questionnaire for the middle-aged Japanese population

In our study, the sample size was determined to show that the FFQ is reasonably valid for nutrient intakes instead of food group consumption. This is because the primary measure is the validity for nutrient intakes; the validity for food group consumption is a secondary measure. The sample size (n = 285) was calculated so that the Pearson correlation coefficients between nutrient intake as estimated by the FFQ and those derived from DRs would be significantly different from 0.30 based on the assumptions of a coefficient of 0.45 [2] (a = 0.05 and b = 0.20) and dropout rate of 10% (n = 28–30) [13].

We recruited 308 individuals from 9 areas in the Japan Multi-Institutional Collaborative Cohort Study (J-MICC study), Yamagata Molecular Epidemiological Cohort, and Tsuruoka Metabolomics Cohort to participate in a dietary survey from September 2011 to October 2013. The details regarding the study areas and the number, age, and sex of the participants enrolled from each of the study areas are summarized in Table 1. To be eligible for participation in the present study, the respondents had to be 35–69 years of age at baseline and live in the study area of each cohort.

Eight participants failed to complete the DRs owing to their busy schedule and other difficulties. Of the remaining 300 individuals, we further excluded one who retracted her consent, three who recorded their diet for more than 12 days, and one who did not fill in both the first (FFQ1) and the second (FFQ2) questionnaires. In addition, seven participants skipped either the FFQ1 or FFQ2. Eventually, 143 men and 145 women who completed the 12-day DR, the FFQ1, and the FFQ2 were included in the analysis for reproducibility and validity. The response rate was not calculated because we did not record the number of individuals who were asked to participate in this dietary survey.

In the validation study, we collected data on the participants’ age, sex, height, weight, and physical activity level using a self-administered questionnaire at the beginning of the survey. The validation study was scheduled as illustrated in Additional file 1. The participants fulfilled the first FFQ1. Four 3-day DRs (DR1–DR4) were then conducted at 3-month intervals, and the respondents were asked to answer the FFQ2 at 2 months after the last DR. The responses to the FFQ2 were compared with those to the FFQ1 to assess reproducibility. The FFQ was validated by comparing the food group intake estimated by both the FFQ1 and FFQ2 with the intake derived from the DRs as a reference. Further details of the FFQs and DRs are described below.

The self-administered FFQ includes question items on the average frequency of consumption during the past year with the following eight possible responses: almost never, 1–3 times per month, 1–2 times per week, 3–4 times per week, 5–6 times per week, once per day, twice per day, and ≥ 3 times per day. These responses are then converted into intake scores of 0, 0.1, 0.2, 0.5, 0.8, 1, 2, and 3, respectively, to approximate the intake frequency per day. The consumption of foods is tabulated in grams for 20 food groups based on the Standard Tables of Food Composition in Japan (seventh revised edition) [14]. The food groups (shown as the number of items: description) were rice (1), bread (1), noodles (1), potatoes (1), soybean products (4: tofu in miso soup, tofu dishes, fermented soybeans [nattō], and fried tofu [ganmodoki]), green vegetables (5: pumpkin, carrot, broccoli, green-leaf vegetables, and other green-yellow vegetables), other vegetables (5: cabbage, radish, dried radish [kiriboshi-daikon], bamboo shoots, and other vegetables), fruit (2: citrus fruits and other fruits), mushrooms (1), seaweeds (1), fish (7: fish, bone-edible small fish, canned tuna, octopus/shrimp/crab, shellfish, fish eggs, and fish paste products), meat (4: chicken, beef/pork, liver, and ham/sausage/bacon), eggs (1), milk (2: milk and yogurt), oils (6: margarine, butter, mayonnaise, deep-fried dishes, light-fried dishes/sauté, and peanuts/almonds), confectionery (2: Western- and Japanese-style confectioneries), green tea (1), coffee (1), alcoholic beverages (1), and soybean paste (1).

The FFQ contains no question items on usual portion size for 43 food items, so we applied the standard portion sizes based on DRs in a population from Aichi Prefecture [3]. However, portion sizes are requested for three kinds of staple foods in Japan (rice, bread, and noodles). The daily consumption of each food item was computed by multiplying the portion size by the intake score. For alcoholic beverages, the amount and frequency per week or month were asked for the following 10 items: sake, Japanese liquor (shōchū), shōchū highball, large bottle of beer (633 mL), medium-sized bottle of beer (500 mL), 350 mL of canned beer, 250 mL of canned beer, single whiskey, double whiskey, and wine. Sugar-sweetened beverages (SSBs) were not included in the short FFQ.

Nonconsecutive 3-day DRs including one weekend day at 3-month intervals over four seasons, that is, 12-day DRs, were collected. Prior to the dietary survey, the participants were guided individually or in small groups on how to complete the DRs by their district dietitians. Participants were asked to record their intake of all foods, dishes, and drinks using a notebook and pictures. A digital scale with a maximum weighing capacity of 2 kg was recommended for measuring dishes with a large portion size (e.g., noodle and rice bowls). No special products were specified. Pictures were taken using checkered-pattern luncheon mats as a scale whenever possible. The managers of each study area were trained by research dietitians (N. I. and C. G.) and the explanations given to the study participants about the DRs were standardized between areas using a common DR manual. Pictures taken with mobile phones, smartphones, and digital cameras were not adopted as the gold standard, but were requested to help supplement the written records, obtain the product names of the confectionery and processed foods, and estimate the portion sizes and volumes. Since 2011, a research dietician (N.I.) has conducted quality control on weighing, recording, and taking photographs of meals in all areas [15‐17]. Between 2012 and 2014, staff dietitians were monitored for compliance with the participants’ instructions. Compliance was rated on a 5-point scale, with “5” indicating almost complete compliance and “1” almost none; these scores were provided as feedback to the staff. The main items checked were the use of luncheon mats; taking pictures of breakfast, lunch, and dinner; taking pictures of snacks; and recording the names of the food ingredients, the amount of food in approximate amounts, and the weight of the food according to the food scales. For each season, the registered dietitians confirmed the details of the DRs by phone or email (or in an interview for the first DR in some cases) when the descriptions in the DRs were unclear. Data from all regions were retrieved using a standardized data checking algorithm, and suspicious data (such as outliers and missing seasoning data) were checked and corrected.

Body mass index (BMI) was calculated using the following formula: body weight [kg] / (height [m])². Energy consumption according to the FFQs and DRs was estimated using the Standard Tables of Food Composition in Japan (fifth edition) [18]. The reason for using the fifth edition was so that we could match the editions adopted in the development of the FFQ. We confirmed that the energy values of the foods appearing in the dietary survey in the fifth edition were identical to those in the seventh and most recent edition.

The median of daily energy-adjusted intake (adjusted by the residual method) of rice, which is a major energy contributor in the FFQ1, FFQ2, and DRs, were 453, 434, and 364 g for men and 272, 267, and 226 g for women, respectively (Table 2). There were no food groups with a robust Z score outside ± 0.5 between the FFQ1 and FFQ2 in men and women.

When comparing the FFQs with the DRs, rice intakes on the FFQs were higher than those in the DRs for both men and women. In particular, robust Z scores below − 1.5 between the FFQ1 and DRs were found for other vegetables (− 2.3), meat (− 1.8), and oils (− 1.6) in men, and other vegetables (− 1.6) in women; robust Z scores between the FFQ2 and DRs showed similar differences. The lowest robust Z scores were found for other vegetables in both men and women. In men, the median of daily energy-adjusted intake of other vegetables was 173 g in the DRs and 48 g in the FFQ1, which was underestimated by 72%. Similarly, in women, the median intake of other vegetables in the FFQs was approximately 54% lower than that in the DRs.

Crude and energy-adjusted SRs by FFQ1 vs. FFQ2 are markers for the reproducibility for food group intake. In men, energy-adjusted SRs were distributed from 0.38 (seaweeds) to 0.86 (alcoholic beverages), with a median of 0.61 (Table 3). The percent of exact agreement between the FFQ1 and FFQ2 according to quintile categorization was 42% as the median, with a range from 31% (seaweeds) to 55% (alcoholic beverages) for men. The agreement for the same and adjacent category was 81% (range 70–94%) as the median for men (Table 4). Extreme disagreement rates (median) were 1% for men. In women, energy-adjusted SRs by FFQ1 vs. FFQ2 were distributed from 0.45 (seaweeds) to 0.84 (alcoholic beverages), with a median of 0.66 (Table 3). The agreement rate for women was 41% as the median, with a range from 32% (seaweeds) to 62% (coffee). Agreement for the same and adjacent category was 80% (range 69–92%) as the median (Table 4). Extreme disagreement rates (median) were 1% for women.

For validity in men, energy-adjusted SRs for FFQ1 vs. DRs were distributed from 0.11 (potatoes) to 0.71 (milk), with a median of 0.44. De-attenuated SRs were distributed from 0.17 (potatoes) to 0.76 (bread and milk), with a median of 0.51 (Table 3). In women, energy-adjusted SRs for FFQ1 vs. DRs were distributed from 0.17 (seaweeds) to 0.72 (alcoholic beverages), with a median of 0.39. De-attenuated SRs were distributed from 0.23 (seaweeds) to 0.77 (alcoholic beverages), with a median of 0.47 (Table 3). The SRs by FFQ1 or FFQ2 vs. DRs can be indices for validity. For both the energy-adjusted SRs and the de-attenuated SRs, the median correlation coefficients by the FFQ2 were 0.04 higher than those by the FFQ1 for men. For women, the median of energy-adjusted SR by FFQ2 was 0.03 higher than that of FFQ1 and the median of de-attenuated SR was 0.05 higher. When miso was included as a soy product, the de-attenuated SRs were 0.61 (FFQ1 vs. DR) and 0.61 (FFQ2 vs. DR) for men, and 0.47 (FFQ1 vs. DR) and 0.52 (FFQ2 vs. DR) for women.

The median agreement rates between the FFQ1 and DRs in men and women were 29% and 28%, respectively (Table 4). The median agreement rates for the same and adjacent categories were 67% (range 58% (potatoes) to 86% (bread)) for men and 65% (range 57% (other vegetables) to 86% (alcoholic beverages)) for women. The agreement rates between the FFQ2 and DRs showed almost the same median value in men and women. The agreement rates for six food groups for men (rice, bread, milk, green tea, alcoholic beverages, and soybean paste) and six food groups for women (rice, bread, milk, green tea, alcoholic beverages, and soybean paste) were greater than or equal to 75% for both the FFQ1 and FFQ2. Extreme disagreement rates (median) for FFQ1 in men and women were 1 and 2%, respectively.

Bland–Altman plot graphs for the consumption of rice (a), fish (b), meat (c), milk (d), other vegetables (e), and fruit (f) among men and women are shown in Additional file 3. Rice (a) showed a wide distribution on the x axis for men; positive and negative errors occurred randomly in individuals with intermediate mean consumption (200–500 g/day). Fish (b) and meat (c) were widely distributed in terms of average intake (x-axis), and the difference, which was FFQ1 − DR (y-axis), tended to become negative as the intake increased. Due to differences in the average reflecting − 34 g (fish) and − 67 g (meat) for men, these food groups were underestimated in the FFQ1. Regarding milk intake (d), the mean intake in the FFQ1 and DRs was concentrated close to the intersection of the x- and y-axes for both men and women. For individuals with low mean levels of consumption, the differences between the errors in the FFQ1 and DRs tended to be less on the y-axis. Greater underestimation was observed in men than in women for the intake of other vegetables (e) and fruit (f) on the FFQ1.

Based on the robust Z scores, the amounts of food group intake were relatively underestimated in this 47-item short FFQ compared with the DRs. Previous research has reported that most FFQs with over 100 items often overestimate the absolute dietary intake because the reported amount of foods will increase overall when many items are asked [26, 27]. This 47-item short FFQ consists of 20 food groups, 11 of which contain only one food item; thus, the small underestimation should be acceptable. In addition, judging from the robust Z scores and the Bland–Altman plot graphs, the estimated intakes on the FFQ1 were severely underestimated for meat for men and for other vegetables for both men and women.

A previous study that evaluated 15 food groups using the 47-item FFQ has already confirmed its reproducibility in the central area of Japan (Aichi Prefecture) [12]. That study showed that energy-adjusted SRs were 0.65 as the median (range 0.59–0.80) for 844 men and 0.60 (range 0.56–0.69) for 1074 women. Although the minimum SRs were lower than those in previous reports in both men and women, the present results showed similar SRs. Thus, the reproducibility of the 47-item FFQ is considered generalizable throughout Japan.

Regarding short FFQs (40–66 items) developed in Japan—Ogawa (40 questions) [28], Date (40 questions) [29], Maruyama (55 questions) [30], Kobayashi (58 questions) [31], and Yokoyama (66 questions) [25]—the median SRs ranged from 0.50 to 0.60 in Ogawa [28], Date [29], and Maruyama [30]. Because the median SRs of the 47-item short FFQ were 0.61 for men and 0.66 for women, the reproducibility of the weight by food group was slightly better in the present study. This short FFQ was developed using multiple regression analysis (MRA) of 102 items from a semiquantitative FFQ [32]. The food list on the 102-item semiquantitative FFQ included foods with a high supply rate of 21 nutrients and energy by contribution analysis (CA). The MRA is based on the variance of nutrient intake. The cumulative R² estimated by MRA can generally be explained by a smaller number of foods compared with the cumulative percent CA. Additionally, because the foods listed in short FFQs are commonly consumed and easy to recognize, the 47-item FFQ might have higher reproducibility.

We assessed the validity of intake by 20 food groups estimated by the present 47-item short FFQ. The number of food groups in previous studies involving short FFQs developed in Japan ranged from 10 to 30 [25, 28, 30, 31]. Generally, the SR for validity was higher when the number of food groups was small, with medians ranging from 0.51 to 0.60 in 10 food groups (Ogawa [28], Maruyama [30]) and from 0.44 to 0.48 in over 30 food groups (Kobayashi [31], Yokoyama [25]). The validity of current short FFQs is comparable to that of latter SRs. In addition, when the participants’ intakes were classified into quintiles, the evaluation based on agreement rates showed that the medians were at the same levels as those in previous studies (Yokoyama 70% vs. our FFQ2 69% for men, and Yokoyama 64% vs. our FFQ2 69% for women). Therefore, our FFQ is considered to be reasonably valid.

The reproducibility and validity of staple foods were relatively higher, especially for rice and bread; milk and alcoholic beverages also showed higher reproducibility and validity for men and women. However, caution is needed when interpreting the intake of some food groups with relatively low SRs. For men, the reproducibility and validity of potatoes were considerably low. Although the reproducibility of mushrooms was acceptable, the validity with both the de-attenuated SR and the cross-classification rate was relatively low. The intakes of meat and other vegetables in men were underestimated, but the categorization power was sustained. For women, the validity of other vegetables, mushrooms, seaweeds, and confectionery were relatively low with de-attenuated SRs in the 0.20s. Especially for seaweeds, the SRs for reproducibility were also low for women. The intake of other vegetables in women was underestimated. In previous FFQs developed in Japan, similar findings were also observed for these food groups [2, 12, 33]. Several reasons could explain these findings. First, it may be easy to observe between-person variation in drinks because drink intakes are widely distributed. However, the ranges of portion sizes were very small (1.0–3.0 g) for dried foods (e.g., seaweeds, dried mushrooms). In addition, because the amount of dried foods and added water can be a systematic error in dietary assessment studies, the correlation coefficient may be underestimated. Another reason is the recognition and memory of individual dietary intakes. In other words, drinks are easier to remember because they are taken alone, while foods are often consumed as a mixture, which makes it more difficult to remember their frequency. Staple foods are also easy to remember as a single dish. In addition, regarding Japanese dietary habits, those who consume bread instead of rice as a staple food are likely to drink milk at the same time [34], which would lead to high reproducibility and validity for both milk and bread [31].

The classification of food groups resulted in 15 groups in a previous reproducibility study [12]. In the present study, this has been expanded to 20 groups based on the Standard Tables of Food Composition in Japan (seventh revised edition). Miso (soybean paste) was classified as a soybean product in the past edition, but this was changed to a seasoning in the revised edition; therefore, soybean products and miso were evaluated separately in this study. By increasing the number of food groups to 20, it was possible to compare the present FFQs with other FFQs and evaluate the relationship between foods and diseases in more detail.

Whether DRs should be compared with the FFQ1 or FFQ2 when designing a validation study remains controversial [1, 35]. Our previous studies on the same FFQ used the FFQ1 for validation [10, 12]. The present study found that the median de-attenuated SRs between the FFQ2 and DRs were slightly higher than those for the FFQ1. According to Willett [1], the correlation coefficient between the FFQ1 and DRs often underestimates the true correlation, whereas that between the FFQ2 and DRs provides an optimistic correlation. Since the FFQ2 was administered after the DR survey, the participants may have been able to provide the real frequency. Previous validity studies by Yokoyama [25], Ogawa [28], Date [29], and Willett [36] treated the FFQ2 as a comparison, and Maruyama [30] assessed validity through a comparison between the FFQ1 and FFQ3 (the third administration of FFQ). Kobayashi [31] used the average value from the FFQ1 and four FFQs. It is difficult to conclude which value of the FFQ is valid for evaluation, as previous studies have used a variety of methods. However, the results from this study for both the FFQ1 and FFQ2 may provide important evidence for future research on more appropriate validation methods.

In this study, we recruited a sufficient number of participants throughout Japan; however, some limitations should be noted. First, although we defined food group intake as the usual dietary intake based on 12-day DRs, this did not reflect the actual daily food intake. According to Fukumoto et al. [37], the number of days required to assess the mean intake of nutrients with 95% confidence intervals within 5% deviation of an individual’s mean from the usual (“true”) intake using DR method is 2–4 weeks for energy, carbohydrates, and protein, and 7–40 weeks for fat, vitamins, and minerals. Since within-individual variations for food group intake are generally larger than those for nutrition intake [38], the 12-day DRs used in the present study may have been rather short. However, longer DRs could increase the burden on participants and result in more dropouts, leading to selection bias. Therefore, we prioritized the feasibility of dietary surveys over the number of survey days statistically required. Second, the influence of selection bias should be noted because the volunteer population participating in a year-long DR survey will be more health conscious than the general Japanese population. Third, SSBs were not included in the short FFQ because the ability of the 47-item FFQ to estimate nutrients semiquantitatively is limited. The effects of SSB consumption on cardiovascular disease (CVD) morbidity and mortality and risk factors have been reported [39, 40]. The effects on CVD incidence and risk factors have also been studied in Japanese populations, but further studies including mortality risk would be needed to accumulate evidence [41, 42]. Fourth, although we evaluated the reproducibility and validity of the FFQ developed in Aichi prefecture over a wide area of Japan, we were unable to evaluate the reproducibility and validity in each area. Thus, we may need to examine between-region differences in the reproducibility and validity of this FFQ. Finally, our survey was set to cover a non-consecutive 3-day period, but some high-calorie foods, such as cakes and sweetbreads, may not have been included in the usual daily diet, as these foods are often consumed only on special occasions (e.g., birthdays, parties); therefore, some participants may have underreported these foods.