Using an Analysis of Behavior Change to Inform Effective Digital Intervention Design: How Did the PRIMIT Website Change Hand Hygiene Behavior Across 8993 Users?

There is accumulating evidence that well-designed digital behavior change interventions (DBCIs), with appropriate content, can deliver effective self-management of health and can change health behaviors across a wide population [1]. Importantly, successful behavior change necessarily requires users to engage with the intervention. Although improving, DBCI non-usage and dropout rates remain high [2], and recent research has investigated how to increase engagement in order to improve intervention outcomes [3, 4]. Studies with large populations are well suited to this line of enquiry, as they can allow analysis of how usage and engagement differ between population subgroups.

Current work examining usage has typically focused on easily available metric data. Examples of such metrics are the total time spent using the intervention, the total number of times a user has accessed the intervention, number of interactions with usable web content or the sum total of intervention pages accessed [5, 6]. However, there are limitations to the interpretation of each of these—for example, a large amount of log-ins from an individual user could indicate high intervention engagement but could also indicate poor intervention usability [7]. Some researchers have reported composite measures of the above [8] and more detailed measures such as number of interactions with usable web content [9], but it can be difficult to establish a causal relationship between intervention usage and overall outcome. For example, when an individual ceases to use a digital intervention, it may be hard to determine whether the dropout is caused by premature disengagement from the intervention or due to ‘success of the goal intervention’—i.e. the user’s behavior has changed to an extent that digital engagement with the intervention is no longer needed. To progress beyond the assumption that greater usage is always optimal, usage analysis instead needs to determine the point at which users have reached ‘effective engagement’ with the intervention (i.e. have used the intervention sufficiently to effect desired and positive outcomes; [10]).

What constitutes effective engagement is often context dependent and needs to be established empirically [10]. One solution is to combine observational usage measures with behavioral outcomes, in order to identify particularly effective intervention components, or a ‘minimum threshold for change’, i.e. an amount and/or pattern of usage commonly required to instigate a positive outcome. Such findings can then be used to optimise intervention efficacy—for example, by determining whether particular intervention components are effective (and then delivering them early to users)—or by finding out whether a particular number of sessions or level of engagement can effectively change behavior. Previous work has explored content and session usage in mental health interventions for depression [11, 12], and such techniques have not yet been applied routinely to examine behavior change interventions. It may also be useful to supplement usage analyses with theory-based measures that can help elucidate change in precursors of behavior (such as attitude or intention).

As well as increasing effectiveness and efficacy, analysis of usage with behavioral measures can also be used to understand and improve intervention reach, in line with the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework for translating research into meaningful changes in healthcare practice [13]. Any minimum threshold for change may well differ across different user subgroups, according to differences such as preintervention attitudes, beliefs and behaviors. For example, it is conceivable that digital interventions aimed at modifying complex behaviors (such as hand washing) may be used differently and have different impacts across users with high vs. low education or in a population subgroup with particular health conditions that make behaviors more or less important to change.

An example of an intervention targeting a complex behavior is the PRimary care trial of a website based Infection control intervention to Modify Influenza-like illness and respiratory tract infection Transmission (PRIMIT) trial. This was a handwashing intervention aimed at reducing infection transmission in the home [14]. Participants who had access to the intervention reported fewer respiratory tract infections (RTIs) and reduced RTI transmission within households (vs. controls). Given the established effectiveness of the website, we aimed to understand which elements of the intervention were effective in changing behavior, to provide insights for future development in line with calls for more personalised interventions to increase hand hygiene behavior [15]. In addition to specific insights relevant to behavioral interventions similar to PRIMIT, our analysis is also intended to illustrate an approach to analysing engagement that could be applied to other complex interventions targeting different behaviors (likely generating different minimum thresholds for change).

The design of PRIMIT drew primarily on the theory of planned behavior, which was selected as the principal theoretical framework informing intervention development and evaluation because it can be applied in a wide variety of contexts and combined with other models and predictors, and there is evidence that the constructs are key predictors of health-related behavior [16‐20]. The PRIMIT intervention also targeted perceived risk, in line with predictions from protection motivation theory [16], and evidence from our pilot work [21] that increased perceived risk of infection might promote handwashing behavior. We examined the degree to which changes in these cognitions were associated with changes in the hand hygiene.

Our analyses of the reach of the intervention were informed by studies identifying gender differences in hand washing behavior (males less likely to wash hands or use soap; [22]) and some evidence that lower education is associated with lower hand hygiene with soap [23]. Taken together with evidence that digital interventions can potentially vary in effectiveness and usage across users subgroups (e.g. age, gender and education; [24‐26]), our study aimed to explore whether the PRIMIT intervention behavior change differed across education, age and gender.

In this study, we conducted a well-powered quantitative analysis of usage metrics from a successful randomised controlled trial (RCT) of the PRIMIT intervention. We used data generated by the LifeGuide software platform [27] to examine objective measures of website usage and their associations with changes in preself-report/post-self-report measures of cognitions and behavior. We asked the following questions:

This study used objective, quantitative analysis of usage and self-report measures of cognitions and behavior to examine how the PRIMIT intervention changed hand hygiene in a large population sample. The PRIMIT intervention improved self-reported handwashing behavior, and the analysis presented here confirmed that improved self-reported hand hygiene was related to decreased likelihood of reporting infection. This finding is consistent with evidence that good hygiene habits are associated with reduced infection risk [31].

All constructs of the theory of planned behavior changed in line with intervention aims, and cognitions were strongly positively associated with self-reported behavior in line with our predictions. In terms of determining effective engagement—the usage threshold for behavior change—increases in hand hygiene, behavior was largest in users who visited all four sessions, but by far the largest increase occurred after visiting the first session. Hand hygiene increased in all participants who visited a minimum of the motivation pages and the if-then planning pages. The intervention was equally effective for men and women and for older and younger people. Furthermore, the intervention was particularly effective in users with lower education, although also effective for those with more education.

Our findings have important implications for directing implementation and future iterations of this intervention beyond the context of the trial RCT. It is encouraging that the intervention was equally effective for all sectors of the population that took part in the trial, including men, who are known to engage in hand hygiene less frequently than women [32] and so are in greater need of an intervention. Digital interventions are often more engaging and therefore effective for women with higher levels of education [33] and can therefore risk increasing social inequalities in health [34]. Our ‘person-based approach’ [34, 35] to development involved in-depth iterative evaluation of user reactions to every element of the intervention [36], helping us to identify and address any content that was not accessible and engaging for all users. However, uptake in the trial was low, and so efforts to improve the reach of this intervention need to consider how to motivate uptake. Since perceived risk of infection was a key predictor of attitudes and intentions towards hand hygiene in this trial, raising awareness of personal risk of infection could improve both uptake and adherence.

The trials of intervention principles (TIPs) approach to trialling digital interventions [37] suggest that key characteristics of an intervention can be designated as essential ‘intervention principles’ that must be preserved between iterations, allowing other features of the intervention (such as delivery format) to vary. Our analysis provides one example of how intervention principles can be identified empirically; in this case, these key ingredients appeared to be the behavior change techniques in the first session of the intervention, since the largest change in behavior took place after the first session of the intervention. Hence, the next iteration of the intervention could be designed as one stand-alone session, with content from sessions two–four, accessible immediately after completion of session one. This would mean that users could benefit from immediate access to core content without requiring long-term engagement. Redesigning the intervention in this way could potentially increase uptake, reach and cost-effectiveness, since more users would not need to register and engage extensively (which can be a barrier to uptake and engagement). Although experimental comparison is necessary for confidence that modification to a single-session structure would maintain an equivalent impact on hand hygiene, our analysis provides an efficient means of generating evidence relevant to this question.

A strength of our study lies in its large sample size, which was facilitated by the automatic data collection permitted by a digital intervention. This allowed us to test for moderator effects, which are often examined only in an exploratory capacity due to lack of power. It also allowed us to have confidence in the validity of effect sizes that were relatively small but would nonetheless be useful at a population level. A limitation was that our findings are based on observational data rather than a factorial design (e.g. multiphase optimisation strategy [38]). The content of the core PRIMIT session was ‘tunnelled’, meaning that participants could only access later content (e.g. if-then planning, tailored content) after having accessed prior content (i.e. motivational pages). This meant it was not possible to infer causality—whether hand hygiene behavior change was greater in users who accessed if-then planning because they were more motivated to engage for longer (and whether this was due to differential responses to the prior motivational content) or whether the later content itself effected changes in hand hygiene. However, although factorial designs can offer estimates of between group differences that go some way to answering such questions, they are unable to allow for individual differences in how users may choose to engage or not engage with the different elements of an intervention. For example, in our analysis, we combined ‘tailored content’ that differed across participants (depending on their questionnaire responses) into one content type, with an assumption that each user would be accessing content specific to their needs. Addressing the usage of this tailored content within a controlled, factorial design would involve group means that included users who would be obliged to access content that did not match their needs, therefore limiting ecological validity. Thus, well-powered observational research such as ours remains an effective way to explore intervention usage and can complement factorial research exploring questions such as the optimal number of core sessions required for effective behavior change.

The degree to which our findings are specific to the PRIMIT intervention or could inform behavior change interventions more broadly is an interesting question for further research. Future studies applying similar analyses to other interventions may be able to determine more general ‘cross-intervention’ engagement thresholds, although it is likely that these will vary for different behaviors, interventions and populations. For example, further research could use similar usage analyses to explore ‘effective engagement thresholds’ in interventions targeting different behaviors, such as weight management [39] or smoking cessation [40], and also examine whether interventions modified according to our analysis findings (e.g. core content in stand-alone first session) demonstrate equivalent hand hygiene behavior changes. While beyond the scope of our study, our analysis technique could detect whether ‘targeted cognitions’ were modified by particular pages, providing a valuable tool for intervention optimisation. Such an approach would be particularly useful when developing interventions for the improvement of common factors that exist across multiple chronic diseases (e.g. increased risk perception). Research of this kind could also support metaregression techniques that seek to identify effective intervention components across interventions [41] and could facilitate exploration of how intervention components work synergistically within a single intervention.

In summary, the combination of objective usage analysis and assessment of cognitions and behavior proved an informative, powerful process for examining the behavioral effects of the PRIMIT digital hand hygiene intervention. In particular, we were able to determine a ‘threshold of effective engagement’, comprising the core components of the first session of the PRIMIT intervention. Our findings and methodology may prove useful to inform future intervention development and implementation, helping to maximise the opportunities afforded by digital interventions to provide population level support for effective self-management of health.