Reproducibility of physical activity recall over fifteen years: longitudinal evidence from the CARDIA study

Abundant research demonstrates inverse associations between physical activity (PA) level and chronic disease risks, including cardiovascular disease, diabetes, hypertension and certain cancers [1]. Although clinical manifestations of diseases such as breast cancer do not typically become evident until middle age or older, the pathological etiology may be associated with PA during young adulthood or childhood [2]. It is therefore important to reliably and validly assess participation in PA over the lifespan to fully examine relationships between PA and chronic disease development.

In epidemiological studies that follow participants from late adulthood, PA participation over the lifespan may be assessed by recall of historical PA. Historical PA is defined as activity engaged in more than one year before the assessment [3‐6]. Studies have correlated historical PA with: cardiorespiratory fitness and health markers documented in medical records [3]; personal reports to a physician at a prior point in time [7], and current health markers [8]. However, time constraints and lack of criterion measures are challenges to assessing the validity of historical PA recall.

This analysis extends previous research from the Coronary Artery Risk Development in Young Adults (CARDIA) study [9] by analyzing the reproducibility of historical PA recall for the entire CARDIA sample, and examining differences in reproducibility by age, sex, and race subgroups. The time period between historical PA recalls was 15 years, which may approximate a period of time relevant to studies of cancer and other chronic diseases. Specific objectives were to: (1) examine the reproducibility of reporting PA occurring during adolescence and young adulthood; (2) assess reproducibility differences for reporting PA frequency, type, and intensity; (3) identify demographic characteristics associated with reproducibility differences.

Table 1 summarizes sample characteristics. The distribution of sex, age, education, BMI, and PA scores was similar between the analytic sample at Year 0 and the entire enrolled CARDIA sample; there were slightly more white participants in the analytic sample than in the enrolled sample. By Year 15, educational attainment, overweight, and obesity increased and PA scores decreased in the analytic sample. PA scores at Year 15 were lower than Baseline PA scores for all race and sex groups.

Table 2 presents 15-Year reproducibility of PA levels in adolescence and early adulthood. Weighted kappa and percent agreement indicated consistency between Baseline and Historical PA levels during HS, and lower consistency for PA engaged in either before HS or during the year before Baseline. To examine whether PA recall was influenced by current PA level, Historical PA category was compared to Current PA category. Kappa statistics and percentage agreement for Historical and Current PA categories were low, indicating that Current PA category was less associated with Historical PA category than was Baseline PA.

Fifteen-year reproducibility of reported types of activities was higher for vigorous-intensity activities such as sports and running, and for activities with very low participation rates such as bowling or golf, than for more moderate activities (Table 3). For all activities, differences between Baseline and Historical recall indicated that respondents were more likely to omit activities that were reported in Year 0 than to report activities that were not originally reported, illustrated by the consistently higher numbers in the “Yes/No column” than the No/Yes column.

To explore 15-year reproducibility of reported months of PA participation we plotted the distribution of Historical months by number of months reported at Baseline for each activity. Figure 1 depicts ideal response distributions if participants are able to accurately recall their reported Baseline activity. These idealized data show 100% of those who originally reported no activity (0 months at Baseline) recalling 0 months. For those who reported 4, 5, or 6 months at Baseline, ideal recall is distributed across 4–6 months of activity, with no recalls of other than 4, 5, or 6 months. In contrast, Figure 2 depicts the plots of actual data for months of walking. An interesting pattern emerged: regardless of the number of months reported at Baseline, the majority of participants recalled doing either no activity (0 months) or activity during all 12 months, with some suggestion of clustering at 6 months. We observed a similar pattern for other moderate- and vigorous-intensity activities (data not shown).

Examination of moderate- and vigorous-intensity PA scores reproducibility indicated a modest Spearman correlation between Baseline and Historical moderate-intensity scores (r = 0.29). The correlation for vigorous-intensity scores was stronger (r = 0.50). However, the spread of scores was extensive (see Figure 3), and represents a substantial amount of both under-and over-reporting at the individual level. Historical scores were modestly associated with Current scores for both moderate-intensity (r = 0.36) and vigorous-intensity scores (r = 0.37).

Finally, we applied multiple regression analysis to identify sample characteristics that may explain the differences between Baseline and Historical moderate- and vigorous-intensity scores. Both regression models were significant: model statistics for moderate-intensity activity recall were: F = 76.60, R ²  = 0.26, p < 0.0001 and for vigorous-intensity activity recall were: F = 34.35, R ²  = 0.14, p < 0.0001. Factors associated with the adjusted mean differences between PA scores reported at Baseline and those recalled Historically, are presented in Table 4. Overall, reported differences were larger and more likely to be positive (reflective of over-reporting) for vigorous than for moderate activity. Baseline PA was a significant predictor, in that the lower the Baseline PA level, the more likely participants were to over-report their Historical PA. For moderate intensity, the pattern of differences appeared to show regression to the mean, with the lower scores at Baseline showing over-reporting and higher scores showing under-reporting. For vigorous-intensity scores, participants were more likely to over-report, with the exception of the most active individuals. Regression analysis also indicated significant associations between several demographic variables and the difference in PA recalled scores, including interactions and main effects of sex and race, as well as education and race. Men over-reported vigorous-intensity Baseline levels to a greater extent than women; black men over-reported their activity levels to the greatest extent. Among black participants, higher education was associated with greater over-estimation. BMI level was also a significant predictor of moderate-intensity difference scores, with over-reporting of Historical PA increasing as BMI increased. Although there were several significant predictors in our models, associations between PA recall discrepancy and both season of the year and susceptibility to social desirability (measured by the Marlowe-Crowne Social Desirability Scale) [13] were explored, but neither was found to be statistically significant (data not shown). Overall, the regression models explained little of the variance in the difference scores for either moderate-intensity (R² = 0.26) or vigorous-intensity scores (R² = 0.14).

Accurate and reliable PA assessment is essential for epidemiologists, exercise scientists, clinicians, and behavioral researchers. Recently, objective measures of PA, such as accelerometers, have dominated the public health literature [1]. Objective devices may assess current behavior well; however they cannot provide information about PA from the distant past. To understand the influence of historical PA on chronic disease risk, it is important to know whether PA can be reasonably recalled over the long-term and to assess demographic, social, and behavioral factors that may affect recall.

Results of our comparison of PA recalled over 15 years to original reports varied by both the type of survey question and the type of information obtained. Respondents were able to reproduce categorical ratings of overall PA level (e.g., ‘very active’) reasonably well, particularly for a well-defined and significant time such as during HS (percent agreement >50% and Kappa = 0.43). As displayed in Table 2, percent agreement and Kappa were more modest for periods that were likely to be less memorable, such as before HS or the year before entry into the CARDIA study. It is also possible that HS activity was recalled well because it is a period that may be recalled and reported more often than less well-defined periods. Agreement for reports of whether respondents participated in specific activities was also reasonably reliable (see Table 3), particularly for vigorous-intensity activities (percent agreement 64-79% and Kappa 0.28-0.48). However, in some cases, high agreement may be due to the low participation rate for these activities (e.g., bowling/golf), so that one agreement category (No/No) is very large. As a result, it is not possible to determine which specific activities can be recalled best. When more quantification was involved, such as estimating the number of participation months for an activity, agreement was negatively affected by a combination of a propensity to exclude activities in the Historical report, as well as response clustering at 0 and 12 months of participation, as shown in Figures 1 and 2. It appeared that long-term recall led to a loss of time resolution, with a tendency toward all-or-none estimation, and perhaps some “splitting the difference” by estimating a value of 6 months.

In general, our data showed that PA recall consistency over fifteen years among young and middle-aged adults was generally modest, but comparable to studies of similar and longer duration [14‐19]. However, an important observation is that even when overall agreement was reasonably good for long term recall studies of this type -- such as the correlation of 0.50 for the vigorous activity score -- error at the individual level was quite large. As shown in Figure 3, for a vigorous-intensity score of 500 at Baseline, the Historical scores ranged from 0 to 1500. This substantial error at the individual level will likely reduce the researcher’s ability to detect relationships between historical activity and outcomes within individuals. However, on the positive side, in contrast to Falkner et al. [20], historical recall in this study appeared to reflect actual recall, rather than current activity. With the exception of the Moderate Activity Score, for which agreement of Historical scores with Current score was similar to agreement with Baseline score, Historical reports were more similar to Baseline than they were to Current reports of PA.

The large CARDIA cohort allowed examination of demographic and other predictors of reporting discrepancy. Generally, the Historical score was higher than Baseline, particularly for vigorous-intensity scores. These results indicate over-reporting of recalled physical activity and are consistent with previous studies [16, 18]. Demographic characteristics including BMI, race and education were significantly associated with discrepancies in recall. A race by education interaction reflected the unexpected finding that over-reporting increased with higher levels of education among black participants. Our results also indicated an interaction of sex and race with the Baseline activity level that highlighted over-reporting by men, particularly black men. These results indicate that demographic factors need to be taken into consideration when pursuing studies of physical activity recall, and specific examination by subgroups should be considered. However, in our study, the recall discrepancy was also a function of Baseline activity scores, with the most active participants at Baseline likely to produce Historical activity scores lower than Baseline. Because true activity at the recalled time period is usually not available, it may be difficult to account for this source of error in studies that use recalled activity.

Overall, the current study adds to a growing body of research on long-term PA recall. These studies are important, as researchers increasingly use historical and lifetime measures to examine exposure to PA over the life course in relation to health outcomes [5, 6, 8, 16]. A major limitation of methodological studies of long-term recall PA instruments is that validity can rarely be established due to the lack of criterion measures at the period(s) being reported. That leaves reliability or reproducibility of reports as the primary indicator of instrument quality. Reliability has generally not been examined relative to the actual period of interest. Many studies look at reproducibility of lifetime reports over 3–10 week periods [4, 8, 21], or up to one year [22]. These studies show that the reproducibility of self-reported lifetime activity recalled over short periods range from r = 0.53 to 0.85. Studies of specific activities recalled over longer periods (10–36 years) have shown weaker associations; correlations and kappa statistics range from 0.09-0.52 [14‐19]. However, recall accuracy may have been affected by participants’ age, which varied from middle to older age.

This study has several notable strengths. Participants recalled activity over a long period of time (15 years) with the same instrument that was originally used. Other studies have used different instruments at two points in time, which has made interpretation and comparison of findings difficult [19]. The CARDIA cohort provided a large and diverse sample that allowed examination of factors related to the difference between Historical and Baseline reports. The current study focused on early adulthood and included questions about activity during adolescence. The Historical recall in CARDIA provides relevant data for studies of PA exposure in early life and later health outcomes.

There were also limitations to our study. We cannot determine whether PA recall reflected what participants were actually doing at the time of the Baseline exam, but rather what they reported doing. There are also potential limitations of generalizability. This study relied upon a single questionnaire in a cohort of black and white young and middle aged adults. Long term reproducibility is, at least in part, a function of the reliability of the questionnaire. The CARDIA questionnaire has been shown to have good reliability over two week retest (r ~ 0.80) [11]. It is not clear how well our results will generalize to other ethnic groups, older age groups, or to studies that use different questionnaires. Additionally, we cannot estimate the effect of participant dropout from CARDIA between Years 0 and 15.

However, our results provide an important step in understanding historical PA recall and have implications for future studies. For example, for investigators assembling retrospective cohort studies in which PA is used as an exposure variable, our data suggest that individuals do well at classifying their activity level with categorical questions, particularly for memorable life periods. Categorical responses can be important indicators; for example, a five-category single-item general health question has been shown to be related to health outcomes [23]. Seeking more quantitative precision in Historical recalls, on the other hand, may not be productive; our data showed that there was not much precision in participants’ ability to recall the amount of time over the course of a year for a given activity. These results provide important information when considering the kinds of questions that may be reasonably asked. In future research, investigators may want to consider whether the additional participant burden of asking about details regarding specific components of activity such as duration and frequency adds predictive value.

The current study was able to expand on previous research by examining different components of PA recall. This study characterized novel, systematic patterns, such as the clustering of number of months of the year of reported PA. It also showed that participants were better able to recall vigorous-intensity activities, and could accurately reproduce their activity levels during salient times of life, such as HS. Overall, these data suggest that historical PA recall over 15 years is only modestly reproducible and poor at the individual level among young and middle aged adults. Researchers should consider these limitations when undertaking studies that require assessment of PA in the distant past.