Test implementation of a school-oriented drug prevention program “Study without Drugs”: pre- and post-testing for effectiveness

Process evaluation can focus on various levels (content/organization) and target groups (intermediary and end target groups), and can be conducted by applying several research methods. Process evaluation is also used to determine whether the intervention on each target group was implemented uniformly. Process evaluation provides information about the reasons behind a program’s effectiveness or ineffectiveness. Whether or not the intervention has an influence is also measured, and the reasons behind its success or failure. Many interventions consist of several activities. Process evaluations can then be used to determine how each activity influences the outcome results separately. It is important to know what activities the most or the least contributed to this success. This type of evaluation will help explaining the relationship between specific intervention activities. Process evaluation can help to prove why the intervention failed. Examples of failure of interventions are: inappropriate intervention design, incomplete implementation and insufficient accessibility of the target audience, an intervention can be adjusted to make it yet successful, through a process analysis.

Thus, research is conducted on the conditions necessary for the effect. Furthermore, insight is given into possible improvements or revisions, for example, to improve the intervention design and methods for the future [12‐14]. The following components are also discussed with regard to process evaluation: context, reach, dose delivered, dose received, fidelity, implementation, and recruitment [15]. Not all components need to be included in a process evaluation. Priority should be given to questions on the size, dose delivered, dose received, and consistency.

With effect evaluation, the effects on behavior are measured. Effect evaluation can focus, for instance, on changes in behavior determinants. It is not always possible to measure hard, precise, and objective outcomes with health-promotion interventions. For this reason, secondary effect measures are established and measured. The secondary measures are behavior determinants that may have been influenced during and because of the intervention. The idea is that change in determinants will lead to a change in behavior [16]. Thus, two questions can be asked: 1) has the program attained its short-term expectations and 2) has the program achieved the desired long-term results?

The Drug Demand Reduction Program (DDR) conducted a nationwide Rapid Situation Assessment in 2005 [17]. The assessment studied 480 participants aged 13 years and older. The results showed that the majority of smokers started smoking before they were 20 years old.

More than 50% of the youngsters (13-16 year) indicated that their fathers smoked. With relation to age, 60% of the male respondents had their first alcoholic drink before turning 20, as did 38% of the female respondents.

School surveys conducted in 2004 [18] and 2006 [19] by the DDR (by order of CICAD [Inter-American Drug Abuse Control Commission]) found alcohol, cigarettes, and marijuana to be the most commonly used drugs. Cigarettes and marijuana were predominantly used by male students aged between 13 and 15 years. Female students most commonly used alcohol for the first time when aged between 13 and 14 years. Most students who smoked were in the second year of high school and those who used alcohol were in their first year. On average, 30% of the respondents had access to illicit drugs. For detailed data on drug use among Surinamese adolescents, please see Tables 1 and 2.

The DDR (2005) [20] also executed a small-scaled study among 125 adolescents aged between 12 and 16. The results of this research are as follows: 20% of respondents stated that they knew someone (or more than one person) that used drugs. The drugs users were, for instance, direct family members, friends, schoolmates, or strangers (e.g., people from the neighborhood and junkies). The most commonly used drugs were tobacco (29.6%), and marijuana and alcohol (25%).

Based on the available data, alcohol, tobacco, and marijuana are popular among youth in Suriname. Although not many young people use cocaine, they could easily point out people in their social environment who did. This leads to the assumption that cocaine is readily available among the Surinamese population.

Thus, while the level of drug use among youth in Suriname is not alarmingly high (yet), prevention measures must be implemented to ensure it does not escalate.

To interpret what has been recorded concerning evidence-based school-oriented drug prevention programs, and keeping in mind the assumption that the presented data from the executed school drug surveys and other (small-scale) studies represent just the tip of drug use, a school-oriented drug prevention program “Study without Drugs” was developed using intervention mapping (IM) protocol. “Study without Drugs” is a drugs demand-reduction program that has been developed for Surinamese adolescents at junior high level. The development of the drug prevention program was carried out using IM protocol, which consists of six steps. The added value of this protocol is that it lends itself to prepare and execute health-promotion activities. IM is based on theory, empirical evidence, and additional qualitative and quantitative research, and the involvement of stakeholders and actors is central. The six phases of the protocol enable health-promotion activities to be prepared and executed, taking into account situational and personal circumstances. In phase 6, an evaluation plan is made that focuses on effect and process evaluations of the program, and an evaluation model is developed [13]. The school-oriented drug prevention program developed in March–April 2010 consists of 10 activities, namely: seven lessons (e.g., Dutch, biology, physical education, drawing, religion, and social studies), one informative activity, and two evaluation procedures. The intervention was carried out from June 2010 until –July 2010 (see Table 3). In designing “Study without Drugs” attempts have been made to integrate many criteria which have been proven to be effective in school programs, such as: use of interactive methods; “Teaching of Skills” is central; training of persons who are going to offer the program is necessary; the research design includes a pre-and post-test; an integrated approach and position of the prevention program within the curriculum is desirable; peers as implementers ensure increased effectiveness; activities that fall outside the curriculum should be included; and there is an approach where the family (parents), the society or community or parts thereof are involved or have an input in the program [21‐24].

The two groups of respondents (all pupils from one junior high school) consisted of 60 pupils: 24 boys and 36 girls aged between 12 and 15 years, with an average age of 14 years. Class 2A had 31 pupils (12 boys and 19 girls) and 2B 29 pupils (12 boys and 17 girls). It was a select sample survey based on voluntary participation. School management and teachers committed themselves to carrying out the test implementation at their school. The group of teachers consisted of three women and two men (four teachers had been in service for less than 10 years). Two teachers were qualified to teach at senior high level, two at junior high level, and one had a different teaching qualification. Carrying out the surveys and interviews for both the process evaluation and the pre-and post-test were done by trained interviewers who were not known to the students nor the teaching staff.

The process evaluation by the pupils was done via a questionnaire (developed by the researcher) after each lesson. The objective of the process evaluation by the pupils was to record their opinion on the content of the lessons taught and the teaching method. The pupils also got the opportunity to express whether or not they enjoyed the lessons by qualifying the lesson with a number between 1 and 10. The pupils had a maximum of 30 minutes to fill in the questionnaire.

A structured interview was also conducted with the teachers after each lesson. The interview lasted 45–60 minutes and consisted of questions on the following aspects: reach, dose delivered, dose received, fidelity, subject connection, general remarks, lesson strengths, and areas for improvement. These interviews were recorded with a voice recorder and later transcribed into a computer.

The transcribed conversations were presented to the respondents for possible modification (comments). Then the data were coded and the transcripts analyzed.

For the effect evaluation, a pre-test was conducted before the execution of the program, after two-and-a-half month followed by a post-test after the program. Each of the tests lasted 35 minutes. Measurements were taken on the basis of a one-group pre- and post-test design. These tests aimed to measure the pupils’ knowledge, skills, and the impact of behavior determinants before and after attending the prevention program.

The questionnaire for the effect measurement by pupils comprised of 17 propositions aimed to determine the respondents’ opinions on the content (nine propositions) and teaching method (eight propositions).

Measurement was done using a 5-point Likert scale that ranged from completely disagree (1) to completely agree (5). For the content, aspects such as attractiveness, level of difficulty, and use of material were measured. The measurement of the teaching method included the way in which the teacher dealt with the teaching methods, how (s)he used the material, and linking to the initial situation.

Cronbach’s alphas for the process evaluation questionnaires were calculated. The coefficients of the different lessons ranged from 0.789 to 0.859. The internal consistency of coefficients were also checked per subject per dimension (content and teaching method). The coefficients of the dimension content ranged from 0.659 to 0.808, and those for the teaching method from 0.693 to 0.813.

Table 4 shows the design of the pre- and post-tests. The pre- and post-tests have identical content. The Cronbach’s alphas of both tests were calculated: pre-test, 0.602–0.729; post-test, 0.633–0.851.

Ethical clearance was obtained from the Ministry of Education (in particular, the Department of Inspection for Continuing Education at Junior level). A letter from the Ministry was sent to the participating school in which had been indicated that consent for the test deployment was granted and cooperation of the management and staff were asked for the (test-) implementation.

With regard to obtaining permission/cooperation of the staff, each participating staff member was personally approached. Parents of participating students were asked by the school for permission to take part. None of the parents has objected against their children participating in the (test-) implementation.

Data were analyzed using Statistical Package for Social Sciences (SPSS) software version 19. For the process evaluation, the means, standard deviation, ANOVA, and Pearson correlation were calculated. For the effect evaluation, the pre- and post-test were administered. With questions 1, 2, 3, 4, 5, and 7 (see Table 4), two scores were possible, namely1 (correct/good) and 0 (incorrect/wrong). The total score is the number of correct answers.

The influence of the behavior determinants was measured in question 6. This question is based on the Theory of Planned Behavior (TPB). In question 6 pupils chose their answer by indicating a score (totally disagree–totally disagree) on a 5-point Likert scale. The General Repeated Measure (repeated measurement) was applied. (By using the General Linear Model (GLM) ‘repeated measures’ a variance analysis was carried out on the inter-dependent measurements. This occurs when in the same or similar (with the same level of measurement) measurements were carried out with the same subjects. In case N^nd measurements are performed on different groups, there is - apart from a ‘within subjects design’ also- a ‘between subjects design’).

The advantage of a within- subject design versus a between- subject design is that a within- subject design is more efficient in the sense that the sample need not be less large, while still a significant result is attained. This is because the scores of a person are associated with each other. This provides additional information, making the conclusions more reliable. In a between- subject design the averages of the condition will differ from each other because in one condition other subjects are present rather than in the other condition. In a within- subject design, this may cause no difference between the averages of the condition, because the same people are used for all conditions. The differences between the averages of condition parts of a within- subject design are therefore less subject to change. Due to the additional information of the link, the scores can be compared within a person. This makes it possible to assess whether the effect occurs consistently in all subjects (there is a small MS interaction); that consistency is additional information making the conclusions more reliable.

The pupils scored the lessons from 0 to 10. Except for the first Dutch lesson, there were no significant differences between classes (2A and 2B) and sexes. The average score for each lesson was 8.6 or higher, ranging from 8.6 to 9.4. A comparison of classes shows that class 2A had a range from 8.5–9.7 and class 2B from 8.0–9.3. There is no appreciable difference based on class and sex. With the boys, the social studies lesson scored lowest with 8.0 (range for boys: 4–10), while the girls’ lowest scoring lessons were social studies and biology with 8.7 (range for girls: 6–10). A comparison of the two classes shows that the first Dutch lesson significantly differed from the other lessons: F(2.120) = 0.5.41, p < 0.05.

A comparison of the average scores and p-values with the assessment of lesson content and teaching method did not show any significant differences between sexes and classes. The average grade given by pupils showing their (positive) assessment was between 38 and 41. Based on the SD one can also conclude that male pupils (both in the full sample and at a class level) showed a greater spread in relation to the average score per lesson; however, this difference is not significant. There was significant difference between classes 2B and 2A with regard to the content of the social studies lesson.

There was no difference in class and sex with the teaching method of the lessons. The average score given by the pupils for (positive) appreciation was between 25 and 32 for the full sample, and in class 2B there was a greater dispersion in relation to the average score of the lesson. Again, this difference was not significant.

The results of the Pearson’s correlation show that (p ≥ 0.05) there is a strong positive relation between lesson content and teaching method (Table 5).

The text below describes the responses of teachers to questions concerning key components: reach, dose delivered, dose received, and fidelity. Comments on areas for improvement and lesson strengths are also included.

Time is a significant main effect for attitude F(1, 56) = 6.24, p = 0.000. The post-test showed that pupils now have a more negative attitude towards drugs. Furthermore, sex is also a main effect for attitude F(1, 56) = 4.11, p = 0.000. The attitude of both girls and boys showed a more positive shift in the post-test.

Girls have a much more negative attitude towards drugs in the post-program test. With regard to the subjective norm, time F(1, 56) = 20.63, p = 0.000 and class F(1, 56) = 5.30, p = 0.000 are found to be mains effects. The averages of both classes (2A and 2B) were higher in the post-test. This indicates that pupils became more sensitive towards their own appraisal of the social pressure to use drugs or not.

Looking now at the behavior determinant, observed behavior control, the variables time F(1, 56) = 39.5, p = 0.000 and sex F(1, 56) = 0.487, p = 0.000 are shown to be main effects. The averages were higher in the post-test. The variables sex and class have a significant interaction effect F(1, 56) = 5.05, p = 0.000. Although the boys showed a positive shift in the post-test, the girls showed a greater improvement. Thus, the girls were better able to control their behavior regarding drugs (use). With regard to the effect of time on class, 2A had a better initial score, but 2B showed greater improvement.

With intention there was a main effect for time F(1, 56) = 66.31, p = 0.000 and class F(1, 56) = 8.45, p = 0.000. There was an increase in the averages, which indicates that the pupils experienced a positive shift: there was an increase in the intention to show/execute behavior against drugs. Class 2B had a higher average than class 2A in both the pre- and post-tests (see Table 7).

Twelve propositions were presented representing a skill that indicates defensibleness against drugs. The variable class has a main effect. Based on class F(1, 56) = 118.08, p < 0.05), a difference (increase) can be observed between the averages of the measurements before and after. Class 2A showed a greater positive shift in the post-test.

Two skills showed a significant decrease in the post-test, one remained stable and nine skills showed a positive shift (increase) (see Table 7).

The interviews with the teachers showed that they considered the process evaluation of the drug prevention program to accurately represent the success of the program. The teachers’ positive reactions regarding the implementation and/or adoption of the interventions are based on the following characteristics: appropriate level of complexity, can be communicated, can be included with existing curricula, observability of activities to be executed and results, possibility to adapt content through reversibility and modification, reformulating instructions and getting feedback [25‐27].

Personal characteristics like attitude, subjective norm, observed behavior control, and the intention of the recipient or person delivering the intervention contribute to the experiences/opinions [28]. The factors that impact on the willingness of teachers to pass on the content of an intervention are as follows: the subjective norm, own effectiveness, interpretation of one’s job, the extent in which one feels responsible for passing on the content, and instrumentality of materials [29, 30]. That is, the extent in which the material has clear instructions for its preparation, presentation, and evaluation and the extent in which the material leads pupils to be active and enthusiastic [31‐33].

In the process evaluation, it was mentioned that in some instances, time constraints led to improvisation. With other health-promotion programs, times limitations have been described as bottlenecks [34‐37].

With regard to dose delivered, it can be concluded that the teachers worked fairly well with the (sub) goals of the lessons and that those goals were achieved. With regard to dose received, the teachers reported being satisfied with the intervention. The material was received positively; it was largely suitable for the age and level of the pupils. With regard to its reach, the majority of pupils participated in the program.

The assessment/qualification of lessons by pupils depends on the way in which they (participants in an innovation/intervention) experience the session. Factors like a pupil’s initial level, attractiveness of the lesson content and materials, complexity of the lesson, instructional skills and expertise of the teacher, and application/use of multimedia appear to affect the performance of pupils [26, 31, 33, 38, 39]. Group dynamics processes can also affect a class’s assessment/qualification, as well as assessment by sex. Group dynamics can affect an individual’s performance, making it either better or worse [40, 41].

The differences that became apparent through the chi-square test can be explained with the taxonomy of Bloom [42] and Anderson et al. [43]. Educational psychology indicates that pupils can score considerably better with less complex cognitive activities/goals. These cognitive activities can include listing and/or recognizing drugs (reproduction of knowledge).

A comparison of the results of the pre- and post-tests showed an increase in the percentage of correct responses with the listing of drugs, recognition of drugs, and aid organizations. There is, however, no great increase in knowledge when coupling effects with the correct drugs, propositions on drugs, and mastery of skills [44, 45]. It is difficult to give a precise reason for the occasional decreases in the post-test. It is possible that an overestimation by the pupils occurred in the pre-test, resulting in a more flattering picture of the situation. The post-test is possibly a more realistic representation because the pupils were confronted with accurate information at the intervention. With replication research, the execution of tasks can also be measured by means of observation.

With the measurement of the effect of time on sex and class, class influence is a feature of the group dynamics processes [40, 46]. Items that are of influence in the class are, among others, the number of pupils, composition of the group, group atmosphere and the resulting social interaction, and the group objectives [33].

With regard to the impact of the pupils’ sex, such characteristics can also affect the overall school performance. There are clear differences between the performances of female and male pupils.

Females profit more form individual learning than from collaborative learning. Males profit from both to the same extent. Divergent patterns in the collaboration between females and males show a cognitive chasm between female and male pupils. There are also likely to be differences in the way they elaborate, mode of communication, and their leadership styles [47‐49].

According to Veendrick, Tavecchio, and Doornenbal [50] the learning performance of boys can also be affected by the phenomenon whereby education is becoming feminized. Thus, boys go without male role models, with all its consequences. The difference in learning and performance motivation between individuals in groups/classes may play an important part. Thorpe [51] stated that boys are expected not to like school, as reinforced by comments from their parents and teachers. This has negative consequences for learning motivation and performance. Girls experience just the opposite and they do like school, thus performing better. A third explanation in the literature is a difference in learning styles. A learning style is the manner in which somebody best acquires knowledge. One person learns better, for instance, when the material is presented visually, while another learns best when the content is presented aurally [52]. Boys and girls presumably differ with regard to learning styles, and presumably those used in schools are not appropriate for boys. Van Langen and Driessen [53] have stated that boys perform well in mathematics and physics. They prefer learning rules and abstract facts by heart, and respond to episodic, factual, and detailed comments. Girls prefer open-ended tasks that are coupled with real situations. They respond more extensively to questions, putting lessons learned in a broad context.