These sub-studies tested the willingness and ability of church members who had already returned their questionnaires, to use their enthusiasm to persuade others who had not completed their questionnaire to do so. Two different monetary incentives were tested for the enrollers, who were already members of AHS-2.
Statistical methods and selection of controls
Churches received individual goals for the number of returned questionnaires. Membership rosters were often inaccurate so these goals were determined as 50% of the number of bulletins printed for the weekly church services, as this number approximately matched the number who regularly attended at that time.
The measure of effect used for all sub-studies is the proportional increment, defined as (increment in returns during the sub-study intervention period)/Goal. The analyses for the two “Enroll Another” sub-studies are described separately below. The other sub-studies used church as the unit of observation and analysis. Details of control selection for the telephone follow-up and art poster sub-studies have been published [
5,
9]. However, these whole studies were published in more details in the previous publications than they are explained here. We describe all study endeavors in this one paper so that readers can understand the context of each sub-study and make informed comparisons. Thus the design and results of these two studies are also summarized here. More details are found in the original reports.
With the large number of churches involved, we were able to find suitable control churches to allow a useful comparison, where necessary. Intervention and control periods were always defined as the 6 months after the intervention or defined control status began, and questionnaires that were returned in that period from churches or members, were counted. It was also required that these churches had not participated in other special promotions in the 6 month period before the study of interest began, to avoid contamination from other interventions.
Controls were of two main types: a) churches acted as their own controls during an adjacent 6 month time period when there was no intervention; or b) Other churches that were matched by size, geographical region and time period, acted as controls. In the first situation we were aware that the underlying rate of return of questionnaires in a control period was gradually diminishing as time passed after the main enrollment promotion in a particular region. Thus a control period that followed the intervention period could give a non-conservative bias. Thus we either: a) took control periods for some churches before and other churches after the intervention period. Where the earlier and later periods involved different numbers of churches, the likelihood for the later control periods was weighted by the ratio a/b where a is the number of churches in the early and b in the later periods, as a means of restoring balance; b) took the average for each church of two control periods, that immediately before and immediately after the intervention period, where there were no other interventions in these control periods.
The model used for statistical analysis, for all except the two “Enroll Another” sub-studies, assumed an over-dispersed Poisson process governing the returns from churches. The returns were evaluated during intervention and control periods as a proportion of the predetermined goals given to each of the J churches. The Poisson parameter, λ (the increment in returns), was modeled as
where G is the goal for church j, I is intervention status (0 = control; 1 = intervention), the C
j are 0/1 variables indicating, in different sub-studies, either church identity or group identity (of a matched set of churches), and e is a Poisson error. Thus λ
j/G
j is the proportional increment (PI) predicted by the model, and β
1
is the coefficient of interest that indicates whether the intervention made a detectable difference. A p value tests the null hypothesis that β
1
= 0, which is equivalent to testing that E (PI
I) = E (PI
C), where subscripts I and C refer to intervention and control. The β
2j
coefficients are used, where necessary, to extract a component of variance due to individual churches or matched groups of churches, in a repeated measures approach. Coefficients were estimated by maximizing the likelihood, programming in the R language. An over-dispersion parameter was calculated [
10] and used to adjust the z score used to test hypotheses. This accounts for the fact that churches are not identical units (aside from random error), even conditional on covariates (where these were included).
The two “Enroll Another” sub-studies considered the 500 selected members as the units of observation, with the understanding that any returns from the additional questionnaires that they were each supplied was an increment that could only result from the intervention. Thus, there are no control groups. The statistic of interest is (number of returns during the sub-study)/750 and its 95% confidence interval, treated as a proportional increment. At the time of these sub-studies we were at about 2/3 of the study goal for the whole cohort. So if the interventions had been applied to all current respondents in the cohort, the denominator of the effect statistic (i.e. the goal) would have been 50% larger than the number of current participants. We proportionately apply the same reasoning to the 500 subjects actually in these sub-studies, so providing a value for G and a metric that can be compared to PI values from other sub-studies that were based on church as the unit of observation. Equivalently one could think of these 500 subjects as a synthetic church that was at 2/3 of its goal of 750 returns. The observations are treated as 500 independent Bernoulli outcomes, and the analyses use logistic models having only a constant term. The resulting predicted proportions of returns were multiplied by 2/3 to change the denominator to the sub-study goal. The alpha coefficient, its standard error, and the confidence interval for PII were all also adjusted appropriately.
Table
2 summarizes the control selection and analytic strategies for each study.
Table 2
Details of control selection for each sub-study
A. Associated with baseline promotion |
| Black Art Alone compared to $10 | Other churches* | 24/19 | Concurrent |
| Black Art + $10 compared to $10 | | 24/62 | Concurrent |
B. Early post-baseline |
| Telephone follow-up | Other churches (randomized) | 13/27 | Concurrent |
C. Mid post-baseline |
| Operation 30,000 | Self (152 churches) | N/A | 71 churches before and 81 after intervention period§
|
| Student recruiters | Other matched churches | 9/36 | Concurrent |
| Local Recruiters | Other matched churches | 12/10 | Concurrent |
| Tournament of Healing | Other matched churches | 22/38 | Concurrent |
D. Late post-baseline |
| Lake Union $20 | Self (80 churches) | N/A | All controls are an average of periods before and after intervention |
| Enroll Another $2 bill†
| None | 0/500†
| N/A |
| Enroll Another $10†
| None | 0/500†
| N/A |