The Saving and Empowering Young Lives in Europe (SEYLE) Randomized Controlled Trial (RCT): methodological issues and participant characteristics

SEYLE had one study site in each of the eleven European countries described above. At each site at least one study catchment area, reasonably consistent with an administratively established geographic area was selected. The selected catchment areas in each country are described in Table 1. To meaningfully interpret the potential representativeness, key parameters, such as mean age, number of immigrants, population density, net income and gender proportion for each site were compared to the corresponding national data. Data at the national and local levels were extracted from Eurostat [22] and collected for each participant site. Effect sizes of mean age and number of immigrants at the country and study site levels were calculated for each country according to Cohen’s d, measured as small (d=0.3), medium (d=0.5) and large (d=0.8). Differences in gender distribution among 15-year olds at the country and study site levels were evaluated with a test of proportions. Population density and net income were compared between each country’s national data and the respective study site.

At each site, eligible schools were randomly selected to participate in SEYLE. A list containing all available schools was generated at each site, and the schools were categorized as large or small and randomized into one of the four study arms for possible inclusion according to a randomized order. Simple randomization was used as a method of randomization of schools through a random number generator. Schools were categorized as small if they had less than or equal to the median number of students in all schools in the study area/region; and large if they had greater than the median number of students in all schools in the study area/region. Schools were considered eligible if they were public, contained at least forty 15-year-old pupils, had more than two teachers for pupils 15 years of age and no more than 60% of the pupils were of the same gender. These inclusion criteria were selected to allow for the recruitment of a comparable sample of schools and pupils across study sites, in spite of differences in sociocultural factors and in the organization of the educational system. However, a few exceptions were made in the case of sociocultural particularities of a specific country’s education system and applying the exact same criteria would increase selection bias instead of reducing it across sites. In particular in Ireland, single gender schools were allowed to participate in pairs with a single gender school of the opposite sex and of similar size. In Germany, due to the unique design of the school system, a sample of schools in the three categories of German high schools were selected and randomized separately. National and/or regional school authorities were contacted and informed about the project in general terms in order to get approval, which was obtained in all participating countries. The representatives of SEYLE at each study site then met with the school principals in the respective areas to describe the intervention of the Arm to which their school had been randomized and to explain the general objectives and procedures of that Arm. Each school was selected to participate in one Arm only and no information was disclosed about the interventions to be performed in other Arms of the RCT. On the basis of general information about SEYLE objectives and specific information about the specific intervention Arm into which the school was randomized, the school could accept or refuse to join. When a school refused to participate, the next school randomized in the same category was approached to replace it. It is important to note that schools were replaced only with other schools that were already in the randomization list. This procedure was designed to generate a balanced number of large and small schools in each intervention Arm, to minimize bias and increase the validity of the results. Within each school, all classes with a majority of 15 year olds were approached for participant recruitment, with a minimum of two schools per Arm. This procedure was repeated until a minimum of 250 students were recruited in a each Arm. Prior to requesting consent from the parents and assent from the pupils, general information about the SEYLE study and details about the specific Arm they were invited to participate in, was provided. Not all pupils for whom parental consent and adolescent assent were obtained actually participated, as some students were absent from school on the day the questionnaire was administered. Consent rates were calculated as the percentage of approached pupils for whom parental consent and pupil’s assent were both given. Participation rates were calculated as the percentage of assented pupils with parental consent who actually took part in the baseline questionnaire. In order to evaluate the impact of the consent rates of schools and pupils on the external validity of the collected data; school size, in terms of number of attending pupils, was compared between participating and non-participating schools. Moreover, gender proportion of pupils with and without consent were compared. Drop-out rates were calculated as the number of pupils assessed at baseline who did not participate at the first (3-months) and/or second (12-months) follow-up. Sociodemographic variables obtained at baseline and average scores on the scales employed, were used to evaluate differences between Arms.

A full description of assessment instruments and interventions was previously published [15].

Each SEYLE site used the same methodology in an effort to obtain comparable study results. Homogenous methodology was achieved through two different means. First, a detailed procedures manual (328 pages) was developed, containing information regarding every aspect of the study implementation, including school selection, recruitment, randomization, clinical backup, ethical issues, translation procedures and methods of cultural adaptation, detailed descriptions of each intervention and intervention time-lines, as well as the baseline and follow-up questionnaires. Second, uniform training procedures were conducted. All site leaders were initially trained centrally, in Stockholm. Site leaders then conducted local training for their own teams that included a minimum of 27 hours of group work, with at least 4 hours devoted to each intervention. To ensure study fidelity to the methodology, a series of monitoring site visits were carried out by representatives of the coordinating centre (NASP), together with the two consultants from Columbia University visiting each site. The site visits took place to overlap with training by site leaders and consisted of two-day consultations with local staff involved in the SEYLE project. Present at the site visit were the Intervention Arm coordinators, as well as the site leader (Table 2). During the site visit, local staff were required to present their understanding of the study and its procedures, as described in the manual, as well as the requirements of conducting each of the interventions. Site visits also provided an opportunity to correct any misunderstandings and to provide additional training, if necessary, to assure adherence to the protocol.

Cross-site collaboration was an important study objective and was facilitated prior to data collection, throughout implementation and up until study completion. For example, most sites assumed primary responsibility for one major study requirement, called a Work Package (e.g., translations, cultural adaptation, quality control, ethical requirements, data management, etc.) and collaborated with each of the other sites concerning this specific topic. Some of the major collaborative efforts are described below.

As part of the SEYLE project, a method for quality control was developed and implemented. A series of questionnaires were sent to intervention coordinators in each country in order to ensure that all preparatory procedures were correctly conducted and that the interventions implemented at each site were faithful to the initial intervention models of SEYLE.

Analysis of these data allowed for assessment of the degree of discrepancy between different sites and between implementation in each site compared with the SEYLE model, as well as the effect variations had on the projects overall results and conclusions. Three quality control assessment tools were used - I. A questionnaire administered during site visits to assess the preparedness for intervention implementation; II. A pre-intervention questionnaire focused on questionnaire coding and on specific requirements to be carried out prior to and immediately after the intervention, and; III. A post-intervention questionnaire focused on the implementation of each intervention. Analyses of these questionnaires showed very small differences between the sites in the implementation process and did not identify any major modification in the implementation of the interventions in any site.

The Hungarian site coordinated the translation processes in collaboration with the site-specific translation coordinators and the coordinating centre in Sweden (NASP). Translation coordinators oversaw site-specific translations, back translations and pilot interviews of all SEYLE materials. All materials were forward and back translated in each participating language. German was used in both Germany and Austria but was translated by the German site. All SEYLE materials (instruments & intervention Arm packages) were developed originally in English and then translated into the following languages: Gaelic (Irish), German, Estonian, French, Hebrew, Hungarian, Italian, Romanian, Slovenian and Spanish. In order to confirm the quality of the translations, staff from each site reviewed all back translations, evaluated reports on the respective pilot interviews and provided feedback to the Hungarian site. The site translation coordinators also implemented cultural adaptation: primarily concerning local linguistic phenomena and expressions. Focus groups were then conducted at each site to provide feedback on the cultural adaptation resulting from the pilot testing. In the case of ambiguity, consultation with a cultural linguistic advisor was sought. Based on these procedures, culturally adjusted language replaced the original in the final versions. A report concerning language issues, including possible ambiguity was sent to the coordinating centre for resolution, when necessary.

The scales used in the SEYLE questionnaires were included in the officially translated and validated version, when available, in the respective language, e.g., the SDQ [21], the WHO-5 etc. [15]. If the scale was not available in the required language it was translated (and back-translated) for SEYLE, using the same procedure as for the other study materials. Internal reliability for all scales used in SEYLE was assessed through Cronbach’s alpha [23].

Data were collected on paper questionnaires, with the exception of the Austrian site, where data were collected electronically for direct data entry. In Germany, data collection forms were scanned for automatic data entry while in the other nine countries manual independent double data entry procedures were followed. In those countries information was entered twice using the Statistical Software Package SPSS 17.0. The Estonian site, which was responsible for the data management procedures, provided continuous oversight to other centers and promptly responded to any queries arising during the data entry process. A two-stage data cleaning and quality control procedure was performed to guarantee clean and reliable data. The first stage of quality control was performed locally, based on the two data files generated through double data entry. These files were compared and inconsistencies were resolved by checking the paper material. Based on this corrective action an accurate data file was generated. The second stage of quality control was performed centrally at the Estonian site, by double-checking each local dataset, attempting to detect other errors, such as incompleteness (missing values), inconsistencies (incorrectly followed skip-outs), irregularities (numbers inserted in text variables), and out of range data (i.e. very large number of siblings or sexual partners). The results of these control procedures generated a list of queries that was sent for resolution to the specific site. After finalizing quality control procedures, the Estonian site pooled the data into one database for all respondents for each wave (i.e., baseline, 3-month and 12-month follow-up). Pooled databases for each wave were then merged into one longitudinal pooled database.

Ethical issues were discussed with an independent ethical advisor from Basel University in Switzerland. Each site obtained permission from the local ethics committee to implement the SEYLE study in their respective country. According to guidelines from the local ethics committees, after thorough examination of the SEYLE study objectives and procedures, decisions were made locally to obtain consent through an opt-in method (parents had to sign a consent form if they allowed their child to participate) or an opt-out method (parents had to sign a refusal form if they did not want their child to participate). Study subjects were then recruited into the study accordingly, after obtaining the required informed consents from parents and assent from pupils [24].

A specific procedure to identify and immediately assist emergency cases with acute suicidality was implemented at each site in all four Arms. A minimum set of requirements regarding the identification of emergency cases was followed by each center. However each centre had the opportunity to reinforce the ethical requirements, according to the indications of the local Ethics Committee. Emergency cases were identified through responses to two specific questions in the questionnaire: those who reported moderate or severe suicidal ideation in the previous two weeks, or those who reported attempting suicide in the previous two weeks. Subjects identified as emergency cases were followed-up by local SEYLE personnel until successful referral to the local healthcare system. However, since it was not possible to follow up individuals while in treatment due to confidentiality, it is not known if the clinical intervention had impact on the collected data. However, all emergency cases were allowed to participate in the active interventions and in the control arm. Therefore, these pupils are included in the total data set.

Subsequent to conducting SEYLE, an interdisciplinary workshop^b was held in the Psychiatric Clinics of the University Basel, to analyse ethical issues, especially confidentiality towards minors involved in SEYLE Study. Additionally, relevant codes and guidelines for guidance in mental health research with minors were analysed. While unresolved questions may remain, such as whether and when confidentiality might or should be overridden in cases of emergency [25], within the SEYLE study, all problems of confidentiality were determined to have been handled according to the indications of the local ethical committees and the local laws and regulations of the participating countries.

Response rates for SEYLE are reported in terms of consent and participation rates for schools and pupils.

The internal reliability of each scale was assessed separately for each country. The results are reported in Table 6. The internal reliability for the Z-SAS [18], the BDI-II [19], the WHO-5 [20] and the SDQ [21] was high or very high in most countries.

The major strength of the SEYLE RCT is its application of a robust and homogenous methodology applied across eleven study sites in eleven different countries, selected to provide a broad geographical representation of Europe. Due to extensive collaboration across sites through Work Packages, that required cross-site cooperation of all participating sites throughout the study, uniform adherence to the study methodology was assured. Moreover, the standardized translation methodology and cultural adaptation allowed for the fine-tuning of interventions to be responsive to local cultural contexts, thus ensuring that the project was meaningful and useful data were collected at each site. Another major strength of the project is the inclusion of a control group and the selection of outcome measures, which are related to mental health and wellness, as well as risk behaviours, thus allowing for the study outcomes to be associated with three distinct interventions. Finally, the SEYLE interventions are able to be tested on a combined, large sample of European adolescents, generating the first such findings from a large-scale RCT of adolescent well-being in Europe, providing an important cohort that can be followed over time.

In any large-scale multi-site study using a complex methodology, securing sufficient funding is always an important challenge. In the case of SEYLE, there were two major limitations due to funding: namely, the funding duration precluded a long follow-up after the intervention ended. It would have been of greater value to identify the long-term effects of the SEYLE interventions by having a longer follow-up, as many preventive effects may only be observed after a longer time post-intervention. In fact, a five-year instead of a three-year timetable for SEYLE would probably have allowed for more knowledge to be gained regarding the study’s outcomes. In SEYLE, one site per country was chosen for study participation. Sufficient funds to allow the inclusion of more than one site per country would significantly have improved representation of the urban and rural areas and therefore understanding of different populations. Moreover, the analysis of representativeness of the recruited sample in relation to the respective country was limited by the availability of sociodemographic indicators in Eurostat at the local level (NUTS2). It was not possible to directly compare the SEYLE data and the same indicators at the country level because these were not available for the adolescent population or were collected with different methodologies, ultimately being incompatible.

Consent rates of schools and pupils varied across countries. The consent rates of pupils were very good in eight countries and lower in the three countries where extended consent procedures were imposed by the local ethics committees. However, it has been reported that response rates between 30% and 70% are, at most, only weakly associated with bias [37]. Available indicators such as school size did not differ significantly between participating and non-participating schools with the exception of Slovenia, where more small schools participated in the study. The study was necessarily performed during school hours and consequently there was limited opportunity to collect other than questionnaire data regarding pupil’s behaviour. This school-based approach necessarily required a very limited number of outcome measures. Another limitation is that all data were collected through self-report questionnaires.