Educational inequalities in Global Activity Limitation Indicator disability in 28 European Countries: Does the choice of survey matter?

Table 1 shows the age-standardized GALI prevalence and the ARRs by survey for each country, stratified by gender. Confidence intervals can be found in Online Resource 3.

There is substantial variation in prevalence between surveys. For men, the ARRs using as reference EU-SILC indicate that EHIS provides statistically significantly higher prevalence estimates for 11 of the 15 countries, while ESS yields lower prevalence for 2 countries (Belgium, Cyprus) and no significant difference for the 2 remaining countries (Greece, Romania). For women, EHIS estimates higher prevalence than EU-SILC in 12 countries, with the 3 remaining countries showing no statistically significant differences between the two surveys. When comparing ESS with EU-SILC, the results are diverse. For men in 11 of the 27 countries, ESS yields significantly higher prevalence estimates; in 10 lower and in 6 not statistically different. Women display the same pattern.

When comparing ESS and EHIS, for men, ESS produces higher prevalence estimates for one country (Belgium), lower prevalence for 9 countries and no statistically significant difference for 4 countries. Women display a similar pattern, with EHIS estimating higher prevalence also for Poland.

Table 2 (Model 1) presents the average across all countries of the patterns described above for men and women. For men, as compared to EU-SILC, EHIS estimates on average 17% higher prevalence (ARR = 1.17, CI 1.09, 1.25), and ESS a non-statistically significant 6% lower prevalence (ARR = 0.94, CI 0.84, 1.05). For women, EHIS estimates 23% higher prevalence (ARR = 1.23, CI 1.06, 1.30) and a non-significant 5% lower prevalence (ARR = 0.95, CI 0.85, 1.06). The average difference in prevalence for ESS is not statistically different from EHIS.

The stratified analysis by education shows that ESS estimates statistically significantly lower prevalence than EU-SILC and EHIS for the low educated group (Table 3). The difference as compared to EU-SILC is 12% (ARR = 0.88, 95% CI 0.81, 0.97) for men and 9% (ARR = 0.91, 95% CI 0.83, 0.99) for women. The results for other educational levels are consistent with the results from the model with all educational levels showing higher prevalence for EHIS than EU-SILC, although the difference is larger for the high educated (ARR = 1.31, 95% CI 1.19, 1.42) than for the low educated (ARR = 1.07, 95% CI 1.01, 1.13). Women display a similar pattern.

Figure 1 shows educational differences in GALI prevalence by country, survey and gender (CIs are presented in Online Resource 5). For both genders and most countries, the ARRs are substantially higher than 1, indicating a higher prevalence of GALI disability among the low educated as compared to the high educated, although several exceptions exist. These include Czech Republic (EHIS men and women), Slovenia (EHIS men), Slovakia (EHIS and ESS men), Italy (ESS men; EHIS women), Portugal (ESS men), Cyprus (ESS men), Greece (ESS men; EHIS women), Romania (ESS men; EHIS women) and Croatia (EHIS women).

The average ARR for men for all countries combined for EU-SILC is 1.93 (95% CI 1.79–2.07) as compared to 1.61 (95% CI 1.46–1.66) for EHIS (Table 4, Model 1). This corresponds to 52.5% higher educational differences in EU-SILC than in EHIS. When comparing EU-SILC and ESS, estimates of ARR are 63.2% higher than in ESS (1.93/1.57)/(1.57–1.00). For women in EU-SILC, the differences were 28.1% higher for EU-SILC (1.73–1.57)/(1.57–1.00) as compared to EHIS and 32.7% higher as compared to ESS (1.73–1.55)/(1.55–1.00).

We find a statistically significant association between collection mode and GALI prevalence only; none of the other survey characteristics is associated with GALI prevalence. Relative to present interviewer (PAPI and CAPI), remote interviewer (CATI) is associated with a lower GALI prevalence for men (ARR = 0.81, 95% CI 0.68–0.99) and women (ARR = 0.83, 95% CI 0.70–0.96). Controlling for collection mode does not change the difference between EHIS and EU-SILC as can be seen by comparing the ARR for survey according to Model 1 (ARR = 1.17, 95% CI 1.09, 1.25) with that of Model 5 (ARR = 1.18, 95% CI 1.10, 1.26) for men and by comparing Model 1 (ARR = 1.23, 95% CI 1.06, 1.30) with Model 5 (ARR = 1.25, 95% CI 1.16, 1.31) for women.

For the educational inequalities, only the inclusion of collection mode has a modest impact on the survey-specific inequalities (Table 4). Comparing the ARRs between the models with and without adjustment for this survey characteristic shows no reduction for EU-SILC and ESS and a small reduction for EHIS (ARR: 158 vs. 1.61) for men. For women, adjusting for survey characteristics shows small reduction for EHIS (ARR 1.51 vs. 1.57) and ESS (ARR 1.52 vs. AR 1.55), but a small increase for EU-SILC (1.77 vs. 1.73).

Table 5 shows that the ranks of both prevalence and inequalities given by the surveys do not agree, with correlations close to 0 in most cases. For the prevalence, only the rank comparison between EHIS and ESS for men is close to being statistically significant (Tau = 0.36 and p value = 0.07). For the compassion of educational inequalities, the exception is the rank comparison between ESS and EHIS which is statistically significantly correlated at the 5% level, though with relatively low Kendall’s tau of 0.40.

EHIS estimates around 17% (men) and 23% (women) higher average prevalence of GALI disability than EU-SILC; 24% (men) and 29% (women) than ESS, whereas prevalence is not statistically significantly different between EU-SILC and ESS. The analyses stratified by education show that ESS estimates lower prevalence relative to EU-SILC only for the low educated; and that EHIS estimates higher prevalence across all educational groups, but more marked for the high educated than for the low educated. There is no agreement between surveys in ranking of countries by average prevalence.

On average, EU-SILC estimates the highest educational inequalities in GALI disability (ARR = 1.93 for men; ARR = 1.73 for women), followed by EHIS (ARR = 1.61 for men; ARR = 1.57 for women) and ESS (ARR = 1.57 for men; ARR = 1.55 for women). Educational inequalities are statistically significantly different between surveys for several countries.

There is no agreement between surveys in ranking of countries by educational inequalities in GALI prevalence, with the exception of a small positive correlation between EHIS and ESS for men (Kendall’s Tau = 0.40).

We observe a statistically significant association of GALI disability with collection mode of the survey, with remote interviewer (CATI) associated with lower GALI prevalence relative to present interviewer (PAPI and CAPI). However, the inclusion of survey characteristics does not account for the observed differences between surveys in prevalence or inequalities.

This is the first systematic analysis of the agreement of 3 European surveys in their estimates of GALI prevalence and educational inequalities. Unlike previous studies, we used micro-level data to explore variations both in relative (ARRs) and absolute (ARDs) terms. This is desirable considering that odds ratios (ORs) tend to be artificially high in the case of non-rare conditions (Tajeu et al. 2012) and that risk ratios are preferred over ORs as measures in epidemiologic studies. Additionally, we have used a structured framework to compare country-specific and average differences in GALI prevalence and inequalities as well as their association with survey characteristics, and have explicitly compared country rankings for these outcomes.

Limitations of the study include our inability to study the effect of differences in phrasing of the GALI question. We could not include GALI question differences in the pooled analyses with all surveys because there was no variation in GALI phrasing within EHIS (uses GALI standard phrasing throughout) and ESS (omits time reference throughout). We examined whether GALI phrasing significantly explained variation in GALI disability within EU-SILC, but we were unable to detect a statistically significant association (Online Resource 6). Omission of dimensions of the GALI question (being limited, in activities people usually do, because of health problems, for at least the past 6 months), as well as changes in wording and separation of the dimensions into several questions, has been shown to have an important effect on how individuals respond to self-reported questions (Cambois et al. 2016a; EHLEIS 2011; McClendon and O’Brien 1988).

There are important differences in the prevalence and the educational inequalities of GALI disability between the surveys included in the analysis. These differences have not been explained by the survey characteristics included in our models. There are other factors that are hard to capture that could explain the observed differences in prevalence and inequalities of GALI disability. For instance, the nature of the surveys is different from one another. ESS has extensive information on beliefs, attitudes and behaviors of Europeans, while EHIS is rich in health-related questions and EU-SILC focuses more on socioeconomic and income variables. This means that the context of the survey where the GALI question is being asked varies across surveys, with respondents being primed with other types of questions that could alter their response to the GALI question. The context in which the survey takes place, the wording and format of the question and even adjacent questions have been shown to matter in the responses individuals provide to self-reports (Schwarz 1999).

Although survey characteristics did not significantly explain the reported differences, we found a significant association between prevalence of GALI disability and mode of data collection. The results indicate that surveys conducted by a remote interviewer (CATI) are associated with lower prevalence when compared to present interviewers (PAPI, CAPI). Prior research has shown that collection mode has an impact on data quality (Bowling 2005), as well as on response rate: Response rates are higher in face-to-face interviews (Demarest et al. 2013), and lower in telephone interviews (Sykes and Collins 1988).

Our results are consistent with previous studies that used SRH. Croezen et al. find that prevalence of SRH is significantly different between the three surveys they compare and find associations with several survey characteristics (response rate, sample size, collection mode). Toch-Marquardt (2017) finds something similar for prevalence of SRH and for occupational inequalities, and finds no consistency in regional patterns. The choice of survey has a major impact on the conclusions we draw both about prevalence and health inequalities. This is the case when looking at both educational and occupational inequalities in health. Furthermore, our analyses of the ranking of countries by survey also indicate that the conclusions we draw of best and worst performers are also affected by the choice of survey. This has important implications for the monitoring of health and cross-national comparisons.

Further research is necessary to identify factors that explain the differences between surveys. One promising approach is to exploit changes in the implementation of a survey (e.g., collection mode, sampling design, phasing of the question) within a given country to establish how these affect the measurement of important health indicators. As more years of data become available, changes in survey implementation toward harmonization will provide opportunities to better understand the (lack of) agreement of health measurements across surveys.

The implications of these findings for health monitoring are important. At the national level, it is difficult to make reliable assessments of the prevalence of disability since the agreement between different surveys is lacking and there is no gold standard among the three surveys. This is also the case for the educational inequalities. For monitoring purposes at the country level, it is perhaps best to look at the trends over time for GALI disability and inequalities, and assess if the surveys agree in the trends. This would inform whether a country is consistently improving or worsening. International comparisons are even harder to perform reliably. Which countries are best and worst performers in terms of prevalence or inequalities in limitations depends strongly on the survey. As long as we have no way of knowing which survey represents reality, our only option is to combine all available data sources and search for patterns that are consistent between surveys. Although prevalences do not agree in magnitude, all three surveys estimate a higher prevalence for Latvia and Slovenia, and a lower prevalence for Cyprus. For educational inequalities, both ESS and EU-SILC estimate high risk ratios for Norway for men and Portugal and Slovenia for women.

Although we were unable to detect statistically significant effects of sample size and response rate, it is objectively desirable that both are optimized given financial constraints. This increases the accuracy of population health measurements and minimizes the risk of bias. From this perspective, it is perhaps legitimate to put more confidence in larger sample surveys with higher response rates like EU-SILC or EHIS. ESS has smaller sample sizes which also complicates working at subpopulation levels (Robine et al. 2003), particularly when stratifying by country, gender and education. However, ESS has other advantages, like a higher degree of ex ante harmonization than EU-SILC, whereas EHIS is conducted infrequently.

It is still unclear whether population levels of disability can be reliably measured with self-reports. The lack of agreement in prevalence and inequalities in disability and self-reported health calls for caution when using these surveys for cross-country comparisons.

We find that both prevalence and educational inequalities of GALI disability are significantly affected by the choice of survey. We arrive to different interpretations of the health status of a country, its relative position to other countries and the size of educational inequalities depending on what survey is used for the measurement. This has important implications for population health monitoring, as well as developing valid comparisons across countries.

Our findings add to existing literature that investigated the comparability of SRH and has determined that this self-reported measure also varies significantly across other widely used European surveys. Further study is necessary to elucidate the causes of these discrepancies, and further harmonization of wording of the GALI question is necessary. Meanwhile, one should be very cautious in using these surveys for cross-country comparisons of (inequalities in) GALI disability.