Effect of mobile text message reminders on routine childhood vaccination: a systematic review and meta-analysis

This review was conducted according to an a priori published protocol on PROSPERO International Prospective Register of systematic reviews for publication registration number CRD42017074230 (available at https://​www.​crd.​york.​ac.​uk/​PROSPERO/​#myprospero).

PICOS approach was used to set inclusion and exclusion criteria.

Initially, databases were searched for the same systematic review done before to avoid duplications of efforts. Search terms were pre-defined for a comprehensive search strategy that included text fields within records, and Medical Subject Headings (MeSH terms) were used to expand the searching. We used Boolean operators for the search strategies. A systematic literature search of databases (PubMed/MEDLINE, EMBASE, Cochrane/Wiley library, and Science direct) was conducted. Gray literatures were searched using Google and Google Scholar. In addition, reference lists of relevant studies were identified and reviewed for inclusion (Table 1).

Database search results were combined, and duplicate articles were removed using Endnote (version7) and manually. Eligibility assessment was performed independently by two reviewers (Zeleke Abebaw and Kasahun Dessie). Discrepancies between the two reviewers in this process were discussed with the other review team member (Binyam Tilahun) until consensus was reached. Articles identified through the electronic literature searches and from other sources were comprehensively reviewed based on the eligibility criteria for inclusion. The methods for data collection and analysis were based on the Cochrane Handbook of Systematic Reviews for Interventions [19].

Two authors (Zeleke Abebaw and Kashaun Dessie) independently assessed the risk of bias for each included study by using the Cochrane Collaboration Risk of Bias Tool [19]. Articles were scored as having low, high, or unclear risk of bias for the seven attributes: sequence allocation, allocation concealment, blinding of participant and personnel, blinding of outcome assessor, incomplete outcome data, selective reporting, and other sources of bias. The two authors resolved disagreements in the assessment of risk of bias by discussion and consensus, consulting a third author (Binyam Tilahun) for any persistent disagreements. The kappa statistic was used to assess the level of agreement during risk of bias assessment by the two authors.

Data were extracted by two authors (Zeleke Abebaw and Kasahun Dessie) using a standardized data extraction format. The two authors’ resolved disagreements by discussion consulting a third author (Binyam Tilahun) for any persistent disagreements. The final data were entered into the Cochrane Collaboration Review Manager Version 5.3 statistical software for risk of bias analysis. Finally, Stata version 14 software was used to conduct the meta-analysis.

Heterogeneity in meta-analysis refers to the variation in study outcomes between studies where I² statistic describes the percentage of variation across studies that is due to heterogeneity rather than chance [20]. We quantified the level of statistical heterogeneity with I² of 50% or more indicating the presence of substantial heterogeneity which is an indication to use random effects model and to perform subgroup analysis or meta-regression [20]. We reviewed clinical heterogeneity in the setting, participants, intervention, and outcomes of included studies in order to make a qualitative assessment of the extent to which the included studies were similar to each other. Forest plot was used visually to assess the level of heterogeneity. In addition, Cochrane’s Q statistic was performed with a P value for the chi-squared test of less than 0.1 indicating the presence of statistical heterogeneity.

Publication bias occurs when the results of published studies are systematically different from the results of unpublished studies. In this review, evidence of publication bias was investigated by examining the symmetry of the funnel plot and by performing a Harbord statistical test [20].

Systematic reviews and meta-analyses are essential tools for summarizing evidence for which PRISMA statement flow diagrams ensure transparent and complete reporting [21]. Forest plots and funnel plots were used to visualize the level of heterogeneity and publication bias, respectively. Random effects model was used during analysis, and risk ratios with their 95% CI were used to present the pooled effect sizes [20].

We assessed the certainty of evidence using GRADE (Grading of Recommendations, Assessment, Development and Evaluation). This method results in an assessment of the quality of the body of evidence as high, moderate, low, or very low. Body of the evidence was assessed against the following criteria: risk of bias, heterogeneity, imprecision, indirectness, and publication bias [19].

A total of 1771 articles were searched. Out of those 1771 articles, 558 duplicated articles were removed using Endnote7 software. Finally, the 1213 articles were screened based on the eligibility criteria. After screening of titles and abstracts, 28 articles were included for full-text screening based on the eligibility criteria (Fig. 1). Finally, 10 studies were eligible and included for the review [22‐31], while the remaining 18 full-text articles were excluded with reasons (Table 3 in the Appendix).

A total of ten articles met the inclusion criteria. These studies were conducted in the USA (5), Kenya (2), Nigeria (1), Guatemala (1), and Zimbabwe (1) (Table 2). Author, year of publication, country, study design, study setting, population, intervention used, duration of intervention, timing of message, vaccine type, and outcome data were extracted to describe the characteristics of the studies. The total sample size of the studies was 10,625 (5273 in the intervention group and 5325 in the control groups). Majority (60%) of the studies enrolled parents/women from health facilities [22, 23, 26, 29‐31], three enrolled children who presented for a first dose [24, 25, 28], and one study enrolled caregivers at the village level [27].

Of the included studies, five (50%) were published after 2015. The studies were mostly conducted in the USA, 5 (50%), followed by Kenya, 2 (20%). The follow-up period between studies varied from 14 weeks to 2 years with very high lost to follow-up in two studies [22, 30]. The frequency of messages and timing of the text message reminders was different in the included studies where two studies [23, 25] sent the reminders three times for a single schedule. Four studies [24, 25, 27, 30] measured full vaccination coverage, while three studies measured the outcome for three doses of coverage [22, 23, 26]. The remaining studies measured the outcome at two doses [28] or one dose [29, 31] (Table 2).

The risk of random sequence generation and allocation concealment was low for 80% of studies while unclear for the remaining two studies [24, 28]. Risk of bias in relation to blinding of participant and personnel was high for two studies [27, 29], while risk of bias in blinding of outcome assessments was low for majority of the studies. The risk of attrition bias was high for three studies [22, 29, 30] and low risk for the remaining studies. The risk of selective reporting was high for one study [29] and low for the remaining studies. The kappa statistic showed that level of agreement during risk of bias assessment by the two authors (Zeleke Abebaw and Kasahun Dessie) was 69.2% (P value < 0.0001) (Fig. 3 in the Appendix).

The magnitude of the heterogeneity (64.3%) also showed that there is a need to conduct subgroup analysis to investigate the sources of heterogeneity.

In studies from high-income countries, the effect of a text message on childhood vaccination was positive and statistically insignificant (RR = 1.06; 95% CI 0.96–1.18) with moderate heterogeneity (I² = 56.6%). The predictive interval in this review overlaps the null (0.78, 1.45), indicating considerable uncertainty about the size and direction of an effect in a new study.

On the other hand, there was a statistically significant positive effect of text message reminders among LMICs (RR = 1.13; 95% CI 1.06; 1.21) with moderate heterogeneity (I² = 70.4%). The predictive interval in this review overlaps the null (0.91, 1.41), indicating considerable uncertainty about the size and direction of an effect in a new study. The test for subgroup differences also indicated that there is no statistically significant difference across subgroups (P = 0.31) (Figure 4 in the Appendix).

The forest plot showed that those studies in which the last reminder messages were sent on the due date had a statistically significant positive effect (RR = 1.17; 95% CI 1.11–1.22) on childhood vaccination with no heterogeneity (I² = 0%). Similarly, sending reminder text messages 1 day before the due date also had a significant positive effect (RR = 1.14; 95% CI 1.02–1.28) on childhood immunization with substantial heterogeneity (I² = 81.9%). However, sending the text messages 2 or more days before the due date has no significant effect on childhood immunization coverage (RR = 1.05, 95% CI 0.96–1.13) with low heterogeneity (I² = 42%). The test for subgroup differences also indicated that there is no statistically significant difference across subgroups (P = 0.07) (Figure 5 in the Appendix).

The result showed that the number of messages sent for a given vaccination appointment schedule had different effects on childhood coverage. Accordingly, sending two text messages for one schedule visit had shown a statistically significant effect on childhood vaccination (RR = 1.09; 95% CI 1.01–1.18) with moderate heterogeneity (I² = 74.4%). The test for subgroup differences also indicated that there is no statistically significant difference across subgroups (P = 0.76) (Figure 6 in the Appendix).

Meta-regression is a tool used in meta-analysis to examine the impact of variables on the pooled effect size using regression-based techniques [20]. In this review, the effect of each individual study’s follow-up time on the pooled estimate was assessed using meta-regression. Though not statistically significant, the findings revealed that an increase in a month of follow-up at which outcome was measured corresponds to a decrease in the logRR of intervention effect by 0.003 (P value = 0.568) (Figure 7 in the Appendix).

A funnel plot provides evidence on the presence of small study effects based on subjective visual inspection. If the individual plots are located symmetrically in the funnel plot, it indicates the absence of publication bias [20]. In this review, the funnel plot observation suggests the absence of publication bias (Figure 8 in the Appendix). To objectively measure the presence of publication bias, a Harbord test was also conducted. The results from the Harbord test suggested that there is no evidence for publication bias (P value, 0.340) (Figure 9 in the Appendix).

The GRADE approach was used to assess the quality of evidence for systematic review and meta-analysis based on defined parameters as high, moderate, low, or very low [19]. In this review, the risk of bias was serious and there was moderate heterogeneity among studies making the quality of the body of evidence low level (Figure 10 in the Appendix).

The main strength of the current review lies in our adherence to international standardized guidelines on the conduct and reporting of systematic reviews. We included studies only from peer-reviewed English-language journals, which may have restricted our findings.

The evidence presented in this review shows promise for the use of text message reminders to strengthen immunization programs. Thus, text message reminders deserve more attention as a potential innovation to improve healthcare operations more particularly in developing countries, where the mobile phone penetration is growing exponentially.

Current evidence around mHealth interventions to improve vaccination is of low quality and with few studies. Therefore, more research is needed in this area.