Effectiveness of an individual, online e-learning program about sexually transmitted infections: a prospective cohort study

In the Netherlands, GPs perform 270,000 STI-consultations per year, accounting for 70% of the STI-consultations [17, 18]. CME for Dutch GPs is offered by the Dutch College of GPs (NHG), which develops and manages guidelines and individual CME-programs, and organizes CME-meetings ten times a year [19]. The CME-programs of the NHG are accessible by all GP’s and GP-trainees in the Netherlands. New CME-programs are promoted by e-mail, by mail, and via the website of the NHG.

The individual e-learning program “The STI-consultation”, which uses the CtC-method, was created by the NHG in co-operation with the Dutch expertise-center “STI-aids Netherlands” and the department of General Practice of the Academic Medical Centre-University of Amsterdam. The content e-learning program was based on the content of the NHG-guideline “The STI-consultation” [20]. The e-learning program “The STI-consultation” was constructed using the “Competence profile and attainment targets for GPs” [21‐23], which is based on the CanMEDS competency framework [23, 24]. The specific learning objectives for the program per competence area are presented in a separate file [see Additional file 1: Learning objectives for the e-learning program “The STI-consultation”, presented per competence-area.]. The e-learning program is available from the website of the NHG since September 2013. It is freely available to all 12,500 Dutch GPs and GP-trainees, participation in the e-learning program is voluntary. Completing the e-learning program “The STI-consultation” takes about two hours and participants can decide themselves whether they choose to complete the e-learning program in one time or in different attempts. CME-credits will be awarded after the full completion of the e-learning program [25].

Changing the behavior of health-care professionals in daily practice requires three stages of change: confrontation with activities that trigger the consideration of change (priming), experiencing strategies to motivate or facilitate the change (focusing), and maintaining the behavioral change by means of a supportive activity (follow-up) [26‐28]. The e-learning program “The STI-consultation” uses various didactic methods to meet the learning objectives of the program and to achieve a behavioral change [21]. The didactic methods of interest in this study were the quickscan (priming), the short and interactive videotaped case-studies with examples of good and less appropriate behavior (priming and focusing) and the CtC-method (follow-up). The questions in the quickscan concerned the knowledge and attitude of participants towards the STI-consultation. The questions used in the quickscan are presented in an additional file [see Additional file 2: English-language version of the questionnaire used in the study]. They were based on the NHG-guideline [29], and on difficulties that GPs experienced during a STI-consultation, obtained by means of a focus group among GPs. The quickscan was both used for triggering the consideration of change (priming) and to evaluate the effect of the e-learning program on the knowledge and attitude of participants about STI. The videotaped case-studies were used to introduce a role model for the GPs. Role models are persons who a learner can identify with, who are in a position that the learner wants to be in, and master skills that the learner wants to master. Role models are able to influence the attitudes and behavior of learners [30‐32]. The CtC-method asked participants which change they intended to make as a result of what they had learned in the CME-course. After completing the CME-course, participants were reminded of their intended change two times by e-mail and they were asked if they had implemented the intended change in their daily practice [16]. In order to evaluate the effect of the e-learning program on the knowledge, attitude, and behaviour of participants over time, we compared the results of participants that completed the e-learning program more than one year ago to the results of the participants that completed the e-learning program one year ago or less.

The e-learning program “The STI-consultation” was freely accessible for all Dutch GPs to allow the full population of GPs to participate in the study. The data for this project were collected among GPs that had completed the program. Data were collected since the e-learning program became available in September 2013 and the data-collection was completed in September 2015. In total 2387 GPs completed the e-learning program, 2203 participants gave permission to collect their data (Fig. 1, Flowchart of the selection of participants). Eleven participants had to be excluded because of missing data, so the data of 2192 participants were used for analysis. Table 1 shows the characteristics of the participants. The mean age of the participants was 38.9 (+/− 9.7) years, 14.8% was GP-trainee. The mean time since the completion of the e-learning module was 18.2 (+/−5.8) months.

Participants were asked about their knowledge and attitude towards STI by means of the 11-question quickscan. The answers were given on a 4-point Likert-scale, ranging from “does not apply” (1) to “very applicable” (4). After completing the program, participants were asked to indicate their three most important learning points and their intended change for daily practice both in reaction to what they had learned in the e-learning program.

Two years after the introduction of the e-learning program, in October 2015, participants received an evaluation questionnaire. In total 287 participants completed this questionnaire, 36 participants had to be excluded because of missing data and two participants had to be excluded because of double participation. The responses of 249 (11.4% of original study population) participants were used for analysis. The questionnaire again contained the eleven questions from the quickscan. Participants who formulated intended changes were asked to what extent they had succeeded to implement their intended changes in daily practice. Participants were also asked if they had implemented any “unintended” changes in daily practice. These “unintended” changes were defined as changes that participants made in daily practice in reaction to what they had learned during the e-learning program, without the intention of making the particular change directly after completing the e-learning program. The intended changes and the “unintended” changes were compared to evaluate whether the content of the changes differed.

To evaluate if completing the e-learning program contributed to a change in the knowledge and attitude of participants towards STI, before —and after-scores on the quickscan were compared. To evaluate if scores changed over time since completing the program we compared groups based on time since completing the program (more than one year ago, or one year ago and less).

The learning points, intended changes and implemented changes in daily practice were analyzed by matching them to the specific learning objectives of the e-learning program [see Additional file 1: Learning objectives for the e-learning program “The STI-consultation”, presented per competence-area.]. If a learning point or change was matched to more than one learning objective, multiple learning points or changes were counted. The matching was done by one researcher (LBB) and checked by another (NvD).

For normally distributed, continuous variables the mean (+/− standard deviation) was calculated. Differences in means between different groups were analyzed by means of ANOVA with the post-hoc Bonferroni-method. Differences in scores over time, were analyzed using the paired sample T-test. Nominal values are displayed in cross tables and analyzed using the chi-square test. A p-value of <0.05 was considered statistically significant.

Because of the high amount of non-responders in the study we only analyzed the data available to us.

All statistical analyses were performed using SPSS version 22 [33].

Table 2 shows change on the questions of the quickscan before and after the completing of the e-learning program. Participants showed a statistically significant increase on 5 of the 11 questions. These questions concerned the sexual history taking when patients presented with questions concerning STI, the feeling of competence regarding the STI-consultation, the STI-testing policy in patients under the age of 25 years, the consideration of the diagnosis of HIV in patients with flu-like symptoms, and the persuasion of an active investigation policy in the GP-practice towards HIV-infections.

There were no significant differences present between participants who completed the e-learning program less than one year ago and participants who completed the e-learning program more than one year ago (data not shown).

Of the 2203 participants who completed the e-learning program, 193 participants (8.8%) together formulated a total of 601 learning points. Most learning points concerned sexual history taking (28.1%), additional investigation (27.5%), and the treatment of patients with STI (17.5%) (Table 3).

Didactic methods most frequently used to formulate learning points were the videotaped case-studies and the quickscan. There were no statistically significant differences in knowledge and attitude between participants who did formulate learning points and participants who did not formulate learning points after completing the e-learning program (data not shown).

In total 179 participants (8.2%) intended to make a total of 261 changes in daily practice (Table 4). Most intended changes concerned the performance of additional investigation (42.5%), the sexual history taking (37.4%), and the treatment of STI (3.8%).

Participants that formulated intended changes, reported that 34.3% of the intended changes was fully implemented, 62.9% was partially implemented, and 2.8% was not implemented. Reasons for partially or not implementing the intended changes concerned forgetting the intended changes (50.0%), small amount of consultation for STI in daily practice (21.4%), time pressure (14.3%), shame or resistance to talk about STI (7.1%), and the costs of STI-tests (7.1%).

In total 114 participants made a total of 180 “unintended” changes in daily practice. These changes mostly concerned the sexual history taking (41.7%), the additional investigation (31.7%), and a change in the working conditions of the STI-consultation (9.4%). Participants described the change in the working conditions concerning the STI-consultation, as:

“I do not make a phone-call anymore when a patient has a question about STI, but I want to see the patient in my office, so I can more easily ask the right questions.” (participant)

“We extended the duration of a STI-consultation from 20 minutes to 30 minutes. Quality-improvement!” (participant)

Our findings suggest that, next to the traditional forms of CME, individual and online forms are effective in improving STI-knowledge and attitudes of health-care professionals. This supports the findings of previous research on the effectivity of internet-based CME and CME for STI [7, 9]. Since health-care professionals, also in our study, still feel resistance to talk about STI or to take a sexual history [34‐36], offering an individual e-learning program that participants can complete in their own environment and at their own pace [37], may take away a barrier for completing a CME course on STI, thereby offering the opportunity to improve the knowledge, attitude, and behaviour of health-care professionals concerning the STI-consultation.

The videotaped case-studies are an important feature of the e-learning program. Previous research shows that both the use of the movies or videotaped case-studies and role models are able to contribute to the process of acquiring knowledge and changing attitude [31, 38, 39]. This might especially hold for programs concerning STI, since a videotaped case-study with a role model can show that, when treating a patient in an open, respectful, and unbiased manner, questions concerning the sexual behaviour of the patients are accepted and answered by the patient. The participants also considered the videotaped case-studies to be important for the e-learning program: “The videotaped case-studies are a nice way to support the sexual history taking of doctors.” (participant).

In this study we have also seen that participants formulated intended changes for daily practice, of which 97.2% was partly or fully implemented in daily practice. This does not only support the findings of previous research performed on the use of the CtC-method in traditional CME-courses [13‐15], but it also indicates that the CtC-method might be effective in individual online CME-courses. Interestingly, participants also made changes without intending to make that particular change in advance. This phenomenon may be explained by individual differences in the priming-stage of the process of making an change in daily practice. In the priming-stage, a certain degree of dissatisfaction with the events in daily practice has to emerge in order to be able to make a change in daily practice. Not every person recognizes or accepts the signals of dissatisfaction at the same time, leading to individual differences in the process of change in daily practice [28]. This is confirmed by the content of these “unintended” changes, which are more frequently related to changes in the working conditions concerning the STI-consultation and the attitude of the participants towards STI and sexual behaviour. The latter may be explained by the theory that a change in professional behaviour or attitude can only take place when one recognizes the impact of the specific behaviour or attitude on others [40], leading to a change whenever a person again is confronted with the particular situation.

Our study is performed in a large study population of over 2000 GPs. Additionally, we were able to analyze the effect of the e-learning program up to two years after the program was completed, showing that changes induced by individual e-learning can have a long-term effect.

Our study is limited by the use of self-reported data of the participants, possibly leading to socially desirable answers. Especially the self-reported data on behavioural changes in daily practice may be subject to self-report bias, leading to the under-report of behaviour that may be considered as non-desirable by the participants and over-report of behaviour that may be considered as desirable by participants [41]. Additionally, the response-rate for the evaluation questionnaire was low, leading to a considerable amount of missing data and a possible overestimation of the study-results. This low response-rate may be the effect of a few of the disadvantages of e-learning programs, like the lack of social interaction between participants and confusion or frustration as a result of the use of a new and unknown technique [37]. However, analysis of the basic characteristics shows that there are no statistically significant differences between the participants who did respond to the evaluation questionnaire and participants who did not respond to the evaluation questionnaire. It seems that the participants who did respond to the evaluation questionnaire are a good representation of the original study population. Another limitation in our study is the small effect-size in the results of the quickscan. This may be caused by the fact that only a 4-point Likert-scale was used, and by the fact that some of the scores on the questionnaire were already very high before the start of the program. Because we did not use a control group, it is not possible to determine whether the effects we found can be fully attributed to the e-learning program. We are not informed on additional CME-courses that participants followed and the effect of experiences in daily practice on knowledge, attitude and behaviour are difficult to determine.

Our results provide insight in the effectiveness of individual, online CME-courses. Since online CME-courses are gaining popularity [8, 9], it is important to gain more knowledge on their efficacy. Future research could benefit from an intensive comparison of online, individual CME-courses and traditional CME-courses, especially on the STI-consultation, to further determine the role that could be awarded to individual, online CME-courses. Additionally, we could benefit from the use of a control group, non-self-reported data, more detailed response-options for the participants, and a more intensive follow-up procedure to gain more insight in the effects of (components of) individual, online CME-courses on the knowledge, competences, and the behaviour in daily practice of health-care professionals.