Background
Effectiveness data for digital health or electronic health (eHealth) interventions to promote smoking cessation continues to grow [
1‐
4]. Digital health interventions can provide several functions, including motivational messaging, facilitating connections with experts, and peer communities. Motivational messaging is a frequently used function in digital health interventions and has been adopted in several real-world smoking cessation programs [
1‐
5]. Tailored motivational messaging, also called computer-tailored health communication (CTHC), is the process of selecting optimal motivational messages for an individual participant to improve the relevance of the messages [
6]. CTHC builds on the concepts of personal relevance, relatedness, and cultural similarity [
7‐
9]. It has been widely used to motivate behavior change, including smoking cessation [
10‐
20]. CTHC systems can be implemented in many ways. Most CTHC systems are rule-based, where patients’ baseline characteristics (e.g., age, race, sex) are matched to if-then rules to select messages [
6,
21]. One limitation of these systems is that they use pre-designed rules, and when being used, do not seek user feedback to improve message selection.
A new alternative approach to developing CTHC systems is using a recommender system [
22‐
25]. The recommender system studied here is a system recently developed and applied to the domain of smoking cessation [
22,
23]. The system actively seeks feedback from users (i.e., asking users to rate the motivational messages the system sent) and uses the feedback and other information about the users to inform the selection of the next message. The recommender system used a hybrid machine learning algorithm, which combined collaborative filtering and content-based ranking, to select messages that are most suitable for individual smokers [
22,
23]. It used multiple information sources as its input to generate the recommendations (i.e., selecting the messages for users): metadata description of the messages, smokers’ demographic characteristics, and feedback data (e.g., smokers’ ratings on the messages). The recommender system had shown promising results in motivating smokers to quit in a pilot study [
26]. This pilot study tested the short-term (30 days) effects of the recommender system on smokers (
N = 120). During the 30-day follow-up, the intervention group (using the recommender system;
n = 74) rated the message as influential (i.e., agreed or strongly agreed that the messages influenced them to quit smoking) more frequently than the comparison group (using the rule-based CTHC system;
n = 46) (74% vs. 45%,
P < 0.01). Among those who completed the follow-up, 36% (20/55) of intervention participants and 32% (11/34) of the comparison participants stopped smoking for 1 day or longer (
P = 0.70).
The aim of the current study was to observe and analyze the behavior of the recommender system for promoting smoking cessation in a real-world setting to gain insights about how the system worked. We studied this from the perspective of user-system engagement and its impact on the cessation outcome. Our study was motivated by three reasons. First, it is usually assumed that digital health interventions need a certain level of user engagement to produce effects [
27‐
29]. Second, measuring engagement is important for understanding the effects of digital health interventions on promoting behavior change [
29,
30]. Third, traditional CTHC systems are one-way communication systems, which does not allow a closer examination of the user-system engagement. The recommender system, by design, seeks user feedback on the message it sent and uses this feedback to improve the selection of the next message. This system property provides us a unique opportunity to examine the granular user-system engagement patterns in terms of both behavior (e.g., user’s response rate) and subjective experience (e.g., user-perceived influence of the messages sent by the system) [
28,
31].
In this report, we present our exploration of the following questions. How did the users engage with the recommender system? Did the recommender system perform and respond to users’ feedback as expected? What was the impact of user-system engagement on retention? What was the association between user-system engagement and the smoking cessation outcome?
Discussion
Tailored motivational messaging or CTHC systems have proven effective in promoting smoking cessation [
15‐
20], but the knowledge about smokers’ engagement with such systems is still limited. In this study, leveraging data around the use of a recommender system for CTHC, we examined the granular user-system engagement patterns during the intervention process and their impact on the cessation outcome. We found that user response rates for the recommender system were heterogenous, with 20% users responding to (i.e., rated) over 60% messages they received and 38% users not responding to any messages. Most users who responded to the system thought that the messages were influential. User’s response rate and perceived influence of the messages were correlated, but not in a simple linear pattern. Users with high response rates or giving the messages high influence rating scores were more likely to quit smoking at 6-month follow-up. Below we further discuss our findings within the context and their implications for future research.
Prior tailored motivational messaging systems typically used rule-based algorithms to select messages [
15‐
20,
40‐
44], where user’s baseline characteristics are matched to if-then rules [
6,
21]. In a prior study, we had implemented a rule-based system to select messages for a smoker based on their readiness to quit [
17]. The Text2Stop system is another example of this rule-based approach. The system used user’s demographic and other information (such as smoker’s concerns about weight gain after quitting) collected at baseline to select motivational messages for each user [
19]. The recommender system we studied is different from these previous systems. It proactively sought a user’s feedback (i.e., rating) on a message it sent and used the feedback to improve the selection of the next message for this user using a machine learning algorithm [
22,
23]. The actions of seeking feedback, improving message selection, and sending the next message would continue if the user kept engaging with the system. In this study, leveraging this unique system property, we closely tracked and examined user-system engagement patterns and studied their impact on the cessation outcome.
Measuring engagement is important for understanding the effects of digital health interventions to promote behavior change [
29,
30]. Previous work conceptualized engagement with digital behavior change interventions in terms of both behavior (e.g., extent of usage and reactions to the intervention) and subjective user experience (e.g., attention, interest, and affect) [
28,
31]. A variety of methods, such as semi-structured interviews, observations, self-report questionnaires, ecological momentary assessment, psychophysiological measures, and the analysis of system usage data, have been applied to measure engagement [
28,
30,
31]. Our method is closely related to ecological momentary assessment (EMA), which assesses users’ current experiences and behaviors in real time and in their natural environment [
45]. The use of EMA in eHealth research has been focused on assessing health behavior and determinants rather than engagement [
30]. Our study contributes to the literature by providing a small, successful use case of EMA-like methods in measuring engagement for eHealth interventions. For example, by examining the continuously tracked engagement data, we found that the low-engaged users tended to stop responding to the system much earlier than the highly-engaged users (Fig.
1).
To the best of our knowledge, our work was the first to measure user-system engagement around using a recommender system for smoking cessation. As there were no standard measures available for this problem, our measure design was guided by general principles recommended in the engagement literature and also driven by our data. For example, we defined two types of measures for user-system engagement, as represented respectively by user’s response rate and user’s rating score for the influence of the messages sent by the system. This was informed by the literature on measuring engagement for digital health interventions, which emphasized the importance of measuring both behavior and user experience [
28,
31]. Our two measures complement each other in several ways. The user response rate is an objective measure that focuses on user behavior; the influence rating score is a subjective measure that focuses on user experience and perceptions. The former quantifies the engagement for all users receiving the motivational messages; while the latter focuses on users who have responded to the motivational messages. Intuitively, we expect users to be more willing to respond to the system if they think the messages sent by the system are influential. In our study, we found that these two measures were indeed correlated, but their relationship was not a simple, linear one (Fig.
3). This suggests that the two measures are different and both are valuable for measuring engagement.
When assessing user engagement by the response rate, we used both the mean response rate and four levels of response rate. The second measure provided additional information about user engagement when the user response rates were not normally distributed (as in our case). We used a data-driven approach to define this measure (see Data Analyses) because there were no standard criteria for defining the level of engagement. Therefore, the numeric values we used to define the levels of engagement may not be generalizable to other studies. However, our data-driven approach, including separating zero-response users from other users and using the quantiles to inform the definition of response levels, is generic and can be applied to other study settings.
Light-touch digital health interventions like motivational messaging are more accessible to study participants than in-person or telephone counseling, and therefore are likely to reach a wider population of smokers. However, low levels of engagement and high rates of drop-out are common in such interventions [
46‐
48], which may reduce their impact. In this study, we found a retention rate (i.e., 1 – dropout rate) of 0.56 for using the recommender system. This level of retention rate is low, and is comparable to several prior studies using digital interventions, including motivational messaging, to promote smoking cessation [
17,
18,
20,
40,
49,
50]. In addition, we found a high-degree of heterogeneity in the user-system engagement measured by user’s response rate. Active users (20%) responded to over 60% messages they received; while quite some users (38%) did not respond to the system at all. The retention rate for highly-engaged users (> 0.6 response rate) was more than 3 times of the retention rate of zero-response users (0.953 vs. 0.294; Fig.
2a). A higher-level of engagement was also associated with a higher likelihood of quitting smoking (Table
4). These results, taken together, suggest that developing strategies to engage low-engaged users is the key to improve retention and the impact of the recommender system on promoting smoking cessation.
There are several strategies that may be useful for improving user’s engagement with the system. Our trend analysis of engagement showed that low-response users decreased their engagement with the system early (after 1 month of participation) (Fig.
1). We also found that low response rate was associated with low retention rate (Fig.
2a). In particular, the zero-response users had a very high dropout rate (0.706). These results suggest that having a strategy to track engagement and enhance the dose of intervention (e.g., having a tobacco cessation specialist to intervene) for low-response, and especially zero-response, participants at the early phase (e.g., the first couple of weeks) may improve engagement and prevent dropouts. In addition, our analysis results highlight improvements that can be made to the system. The protocol of the current recommender system is to send a user the same message next time if the user did not rate the message. This design can be suboptimal, especially when the user did not rate the message mainly because they did not like the message or think it helpful. In the future, we will explore the following strategies: (1) improving the next message selection by treating non-response lasting for more than 1 week as negative feedback; (2) randomly selecting another message to send; and (3) developing new methods to seek implicit feedback from users’ response patterns. Furthermore, strategies targeting young adults may be also useful. Prior studies found that young adults were less likely to use tobacco cessation treatment [
51,
52], including web-based cessation interventions [
53,
54]. Similarly, we found that younger adults were less likely to engage with the recommender system than older adults.
Because the rating score directly measures a user’s perceived influence of the motivational messages, it provides more information about the quality of the engagement than the response rate does. In this study, we found that, among users who have responded to the recommender system, 57% of them strongly agreed (i.e., average rating score > 4.0) and 37% always agreed that the messages they received were influential (Table
3). Among the top-10 high-impact messages ranked by the rating score, 6 messages focus on behavior treatment methods, indicating the important role of behavior therapy in smoking cessation. In addition, for 58% of the users, the system was able to improve its selection of the next message for over 80% cases where it received a low rating score (Table
3). These results indicate that the recommender system worked quite well for this subset of users. Compared with other users, African-American users assigned higher rating scores to the messages sent by the recommender system. This finding is compatible with that from our pilot study [
55], suggesting that the recommender system may be helpful in engaging and motivating this harder-to-reach and harder-to-engage group. This hypothesis needs to be further tested in studies specifically targeting this group. Additionally, we found that the rating scores assigned by users were usually high and were very close to each other. Therefore, the amount of information that the system can learn from these scores may be limited. Using a wider scale (e.g., 1–10) for the rating score may be helpful in addressing this limitation.
Our study has several limitations. First, similar to other observational studies, we can’t ascertain causality. Although we have adjusted for baseline participant characteristics for the association analyses, there may be unobserved factors to impact the analysis results. Second, our messaging system is also limited in that there is a potential that participants may not have received the motivational messages we sent them. We were unable to confirm whether the zero-response cases were partially due to the failure in message delivery. Third, due to an issue with system setup at the beginning of this study, participants (less than 5%) who registered for the study before November 10, 2017 received daily messages before that date. To reduce the impact of this issue on data analysis, we used the response rate (rather than the total number of messages rated by a participant) to measure participant’s engagement with the system and adjusted the association analyses by the total number of messages received by each participant. Fourth, we found that user engagement leveled out after the rating score reached 4.5. However, it is hard to interpret this result due to lack of qualitative data from the users. Future studies that use both quantitative and qualitative methods to study the engagement are warranted to better understand this phenomenon. Finally, we had a low retention rate of 0.56 which was associated with low engagement, but we do not know specifically why these participants dropped out.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.