Can opportunities be enhanced for vaccinating children in home visiting programs? A population-based cohort study

The Families First home program was initiated in Manitoba, Canada, in 1999. It was modeled on Hawaii’s Healthy Start program and uses the Growing Great Kids Inc. curriculum [17, 18]. Approximately 80 % of all births in the province of Manitoba are screened for eligibility for the program, initially with a brief newborn screen of biological, social, and demographic risk factors and then, for those screened at-risk, followed up with a comprehensive risk factor assessment using a parent survey. This parent survey is based on the Kempe Family Stress Inventory that covers a variety of domains, including psychiatric history, criminal and substance abuse history, childhood history of care, emotional functioning, attitudes towards and perception of child, discipline of child, and level of stress in the parent’s life [19]. Parents’ responses on this survey are used to calculate a risk score; herein referred to as the ‘parent survey score.’ Families with a parent survey score ≥ 25 are eligible for the program, which involves working with a trained paraprofessional home visitor overseen by a public health nurse. The average length of time in the program is 18 months and the average number of visits per month is just over two, although this varies depending on the family’s needs.

This study was undertaken in Manitoba, a province of approximately 1.2 million people geographically located near the centre of Canada. Residents of Manitoba receive health care services under a universal program of insurance coverage funded by the provincial government, including a program of publicly funded vaccinations for all Manitobans. Data for this study were derived from the PATHS (Pathways to Health and Social Equity) Data Resource, a unique database developed for the PATHS program of research containing anonymized individual-level data from the Population Health Research Data Repository, which is housed at the Manitoba Centre for Health Policy (MCHP) [20]. The PATHS Data Resource includes data provided by Manitoba Health, Healthy Living and Seniors; Manitoba Jobs and the Economy; Manitoba Family Services; Healthy Child Manitoba; the Winnipeg Regional Health Authority; Statistics Canada; and Manitoba Housing and Community Development. The data were not ‘openly available’, but rather permission for its use was requested and received from each of the individual data providers, as well as the Health Information Privacy Commission (HIPC) of the Government of Manitoba. However, other researchers interested in using these data are encouraged to apply for access; more information on applying for access can be found, elsewhere [20].

The PATHS Data Resource comprises population-based individual-level data on the health and social services use for over 99 % of all Manitoba children. The validity of the data included in the PATHS resource has been well documented [21‐23]. Individual-level data from several databases across multiple domains were linked using encrypted identifying numbers. Databases used in the analyses were:

This study included children born between January 1, 2003 and December 31, 2009 who had at least three risk factors on the newborn screen, program data, and an actual or imputed parent survey score ≥ 25. Families First program data were collected by public health nurses and the Healthy Child Manitoba Office and included the newborn screen, parent survey, and program implementation data.

Figure 1 outlines the process of selecting study participants. The original Families First database had information on 16,153 families. There was no parent survey score for 5,369 families. These families’ parent survey scores were estimated using multiple imputation (explained below) because excluding them could bias the results of our study. In practice, the parent survey may not have been done due to public health nurse challenges (e.g., large workloads, lack of experience with at-risk families) or family characteristics such as lack of trust of service providers, no address, no telephone, or addictions issues (personal communication-Marion Ross). Imputing the missing parent survey scores added 2,736 families to the comparison group, 104 to the program group, and 2,253 were excluded because the imputed scores were lower than 25. An additional 904 families were excluded because the program assignment variable and other key variables confirming program entry were missing. The final sample included 4,562 children from families in the program (the intervention group) and 5,184 children from families who were eligible for the program, but did not receive it (the comparison group). Based on statistics kept by Healthy Child Manitoba, we estimate that of the 5,186 eligible families (imputed or documented parent survey score ≥25) that did not participate in the program, 37 % of families refused the program, 11 % did not enter because the program was full and another 52 % were never offered the program because the parent survey questionnaire was unable to be completed for reasons discussed above.

A considerable amount of the Families First program data were missing (38.8 %) due to parents declining to provide certain information, staff not able to obtain information, coding error and lost records. We used multiple imputation to account for missing data to ensure our cohort was representative of at-risk children in the population.

Multiple imputation can provide a valid statistical technique for handling missing data, which accounts for the inherent uncertainty caused by missing values [25, 26]. Multiple imputation uses known data from included covariates to estimate values for covariates with missing data. We used data from the covariates listed in Table 2 in the multiple imputation. Markov Chain Monte Carlo methods were used to fill in missing values 10 times, generating 10 complete data sets using the SAS procedure MI. We analyzed each of the 10 complete datasets using the analytic methods described below. We combined the results from the 10 complete datasets using the SAS procedure MIANALYZE. Analyzing 10 multiple imputed data sets and then combining the results using MIANALYZE (a) reflects the fact that uncertainty remains with respect to the unknown values and (b) provides statistically valid inferential results [25, 26].

There were 4 steps to the analyses.

Logistic regression was used to estimate propensity scores for all children eligible for the program (n = 9,745). The propensity score is the child’s predicted probability of receiving the Families First program given his/her observed characteristics.

Three treatment effects were of interest: the average treatment effect (ATE), the average treatment effect on the treated (ATT), and the average treatment effect on the untreated (ATU). The ATE is the effect of the Families First program on vaccination among the entire eligible population. The ATT is the average effect of the program on vaccination among those who actually received the Families First intervention. The ATU is the anticipated effect of the program on vaccination if the comparison group had instead received the Families First program. These concepts are explained in significant detail, elsewhere [28, 29].

The propensity scores were used to create three sets of inverse-probability-of-treatment weights (IPTWs), each set corresponding to the three treatment effects of interest as described above. IPTWs were applied to the models to balance differences between the intervention and comparison groups’ potential confounders; this is one method for adjusting for measured confounding [28].

Standardized differences were calculated prior to and after the weighting procedure for both the Families First group and those who were eligible but not enrolled in the Families First program to assess the effect of the weighting procedure between groups. Standardized differences measure effect sizes between two groups and are independent of the sample size, which makes them valuable in comparing baseline covariates for studies that use propensity scoring [30]. A standardized difference of less than 10 % is generally accepted as being satisfactory with respect to homogeneity between intervention and control groups for a given covariate [31].

Generalized linear modeling was used to estimate the predicted probability for each vaccination outcome of interest, by exposure-status. Separate models were estimated for each outcome and were used to estimate risk ratios and risk differences. Multiplicative models were used to estimate risk ratios and additive models were used to estimate risk differences [32]. Measures of precision (e.g., 95 % confidence intervals and standard errors) came from the models and an a priori significance level of p < 0.05 was used. All analyses were performed using SAS® version 9.2 [33].

Finally, we determined whether the effectiveness of the program differed by income quintile by introducing an interaction term (urban income quintile x Families First program) to the models. We examined the interaction term using income quintile as a continuous variable and only considered families living in urban areas for this analysis. The urban income quintile variable was made up of five values, 1 for the lowest income quintile and 5 for the highest. Rural areas were not included because they tend to have more mixing of income groups compared to urban groups, which have more homogenous income groups within a given geographical boundary [34].

With propensity score methods, it is assumed that estimates are not sensitive to unmeasured confounding. We conducted a sensitivity analysis to test this assumption. A gamma value was generated, indicating the strength of unmeasured confounding required to invalidate statistically significant results. In other words, Gamma is the hidden confounding which would make the relationship between Families First and vaccination appear significant when, in fact, it is not significant [27, 28]. A gamma value was calculated for each statistically significant result.