Methods and representativeness of a European survey in children and adolescents: the KIDSCREEN study

Health-related quality of life (HRQoL) measurement can provide useful data for health services research. The measurement of HRQoL is currently more advanced in adult than in child populations, though a substantial number of generic and disease-specific questionnaires now exist to measure HRQOL in younger respondents[1].

Reference values can facilitate the interpretation of scores on an instrument by providing a point of comparison for individual or group responses obtained in clinical or epidemiological studies [2]. This is particularly helpful if reference values are available by age group, sex, and other relevant characteristics. Population norms can also be used to develop norm-based, standardized scores. In this case, scores are standardized so that a score of 50, for example, represents the mean score for the general population in a given country or region, and 10 points is equivalent to 1 standard deviation[3]. Reference values and norm-based, standardized scores can be used to determine the extent to which an individual or group differ from the standard for their age, sex, etc.

General population norms are available for some generic HRQoL questionnaires for adults [4], but they are not usually developed simultaneously in different countries. Likewise, reference values are not available for most of the instruments designed for use in children and adolescents. When obtaining reference values, the sampling techniques used should ensure representativeness in terms of age, socioeconomic status, and gender and any other variables considered important [5]. Methods which systematically exclude certain segments of the population should be avoided and the external validity of the results should be evaluated[6] by comparison with an external source of information such as those provided by census data. Previous studies have indicated that the social and individual characteristics of eligible subjects, sampling, and fieldwork procedures can influence the decision to cooperate or not in a survey[7, 8] and it has been found that non-response tends to be higher among individuals with lower educational levels and in low-income groups [9, 10].

The KIDSCREEN is a generic questionnaire designed to measure HRQoL in children and adolescents aged 8 to 18. The instrument was developed and pilot tested in seven European countries [11], and is intended primarily for use in multinational clinical trials and health interview surveys [12, 13]. Three different versions of the instrument have been developed: the core version contains 52 items spread over 10 dimensions [11], the second has 27 items in 5 dimensions, and a 10 item version exists which provides a single index score [12]. Versions for both children and parents or guardians are available for all variants of the questionnaire. The psychometric properties of the 52-item version were tested after administration of the instrument to respondents in 13 European countries [13]. A further aim of the KIDSCREEN project was to obtain reference values or norms which would facilitate interpretation of scores with the instrument.

The objective of the present analysis was to compare the results obtained using different sampling and questionnaire administration methods in the international KIDSCREEN project in terms of participation, response rates, and completion rates. A further objective was to assess the representativeness of the samples obtained in the thirteen participating countries by examining their external validity.

Table 1 shows the socio-demographic characteristics of the samples included in the participating countries. Data from a total of 22,827 respondents were eligible for analysis. Mean age (SD) in the child and adolescent samples was 9.7 (1.1) and 14.4 (1.7) years, respectively. 51.3% of the child sample was female, and 53.8% of adolescents.

It is agreed that it's important to ensure that reference values for HRQoL instruments are based on representative samples. However, there has been relatively little analysis of aspects such as sampling and administration methods, cooperation and response rates, and the external validity. The fact that the present study was carried out in several different European countries using different sampling and administration methods made it even more important in this case. This sort of analysis also provides information about sampling and questionnaire administration strategies which may be useful for future studies to obtain reference values.

Cooperation and response rates varied considerably by country and by the sampling and administration methods used. As was expected, school-based surveys produced higher cooperation rates, which might be explained by a greater degree of confidence among parents due to the school's involvement from the outset (unsolicited telephone contact vs. an individualized letter from the school)[7]. A further advantage of school-based sampling is that once parental consent to participate was obtained, completion and response rates were almost 100%. On the other hand, one of the main advantages of the centralized CATI method is that it is a relatively efficient way of obtaining broad geographical coverage, which might be more difficult to achieve using school sampling. This needs to be balanced against lower participation and response rates, as well as the fact that phone coverage is limited in some European countries. Whereas the percentage of households with phones was over the 90% in most of the countries included in the present survey [16], it was under 50% in CZ, HU and PL [17], leading to an obvious source of sample bias. For this reason, among others, the survey was carried out using school or household sampling in EL, HU, IE, PL, SE, and CZ.

Using the telephone sampling method, we also observed considerable variation in the cooperation, completion, and response rates across countries, with reasonably good rates in NL and German-speaking countries, but lower rates in ES, FR, and the UK. Similar patterns in response rates have been found in earlier Europe-wide studies in adults, although earlier studies have also noted wide variations in response rates within regions and countries [18].

Differences in participation and response rates between countries using the same sampling and administration procedures could be due to several factors. In particular, it was observed that countries with better-educated populations tended to produce higher response rates, though cultural factors could also play a role. This will have implications regarding expected non-response rates for sample size estimates in different European countries. Other reasons for the variations observed between countries might include differences in the reminder procedures used, or differences in study materials even though considerable effort was made to standardize study materials and follow-up procedures. Similar strategies were also used across countries to increase response rates, such as contacting participants before sending the questionnaire, use of personalized questionnaires and letters, inclusion of stamped addressed envelopes for return of completed questionnaires, and a second telephone contact to non-respondents [19].

Despite differences in sampling and administration methods between countries, the final samples obtained were generally very similar to the reference populations. As is frequently observed in surveys of this type, and as indicated by the population fraction rates, there was a slight tendency for a higher response rate from females compared to males and from children compared to adolescents. This was true across all sampling methods and countries. Likewise, the mothers' educational level in the KIDSCREEN sample was similar to that of the reference sample in most countries, though those in the lowest educational category tended to be underrepresented. Although differences between the KIDSCREEN sample and the Eurostat reference populations were small, they could affect scores on the KIDSCREEN instrument, and therefore the validity of the KIDSCREEN reference values, as HRQoL scores have been shown to differ by age, sex, and level of education [20]. To test the possible effect of these differences on KIDSCREEN global and dimension scores additional analyses using re-sampling methods such as bootstrapping were performed. These indicated no appreciable effect on KIDSCREEN scores (data not shown).

Finally, there were some differences in socio – demographic characteristics between participants and refusals, with non-responders in general having lower levels of education and reporting slightly poorer health. Future studies may need to take this into account by over-sampling these groups to ensure adequate representation in the final sample.

The study had some limitations. In particular, the fact that the results cannot be generalized beyond the countries studied and the fact that there may have been some confounding between the countries and the different sampling approaches. Whereas the former limits the study's external validity, the latter might affect the internal validity of the conclusions. Future studies could focus on applying and comparing different strategies within the same country. The results could also be due to psychological and social factors that were not included in the present analysis [21], although the literature indicates that the main factors associated with non-response were included, i.e. age, sex and socioeconomic or educational level[8, 9]. Finally, although we made considerable efforts to ensure that standardized procedures were used for follow-up, there may have been some differences between countries that we were unable to control. Any such differences could affect the study conclusions, particularly as aspects such as reminder letters and follow-up phone calls have been shown to have a considerable impact on response rates [18, 22].

We found that school-based surveys provided the highest cooperation and response rates, although they did not necessarily provide the most representative samples, at least in terms of age and sex. This approach also requires sophisticated sampling strategies in order to avoid biases, and gaining access to schools may not always be easy. Centralized phone sampling and mail administration led to slightly lower cooperation rates, although this was variable between countries. This approach also led to a lower rate of participation amongst those in lower educational categories, although the distribution by age and sex was more representative than that achieved with school sampling. Finally, multistage random sampling of communities and households achieved a well-balanced age-sex, and urban-rural survey, but at a higher cost than other methods. The most appropriate method to use will likely depend on researchers' available resources, telephone coverage in target countries, and the need for representativeness on given variables. This study has also indicated some areas in which oversampling and additional follow-up strategies may be required. Finally, the results suggest that the KIDSCREEN survey has achieved a sufficient degree of representativeness to provide reference population values, which will improve the interpretability of the KIDSCREEN questionnaire.