Background
Missing data due to loss to follow-up has a widely detrimental effect on research studies. It can compromise internal and external validity of results for several reasons, including diminished statistical power due to a smaller effective sample size, a negative impact on generalisability, and the potential to bias results [
1]. The implementation of strategies that effectively retain participants is therefore very important. The use of clinical assessments for follow-up can be very costly, so perinatal trials often use bespoke research questionnaires to collect data, including the primary outcome. It is therefore essential to mitigate loss to follow-up by maximising the return rate of questionnaires.
Systematic reviews suggest that incentives are effective at improving response rates for research questionnaires in clinical trials [
2,
3]. Brueton [
2] and Khadjesari [
4] both reported that offering a monetary incentive improved questionnaire return rates compared with no incentive, in postal and online settings respectively. Edwards [
3] reported that unconditional incentives (i.e. a reward given in advance as a goodwill gesture) led to superior response rates compared with conditional incentives (i.e. the promise of a reward on receipt of a questionnaire). However, there was significant heterogeneity among these studies. Dillman [
5] and Singer and Ye [
6] also reported results that favoured unconditional over conditional incentives.
However, the use of financial incentives is costly, and studies may not have the funds or resources to provide unconditional incentives. A conditional incentive, promised on receipt of a completed questionnaire, could be a more cost-effective means of enhancing retention by rewarding only those participants who respond, minimising wastage from monetary incentives sent out to non-responding participants.
There is also little evidence of which incentive methods are most effective in the context of follow-up of participants (via parents) in perinatal randomised controlled trials (RCTs). There is heightened sensitivity around the population (parents of vulnerable infants) and often a lengthy period between recruitment and outcomes captured at follow-up. Hardy et al. [
7] successfully carried out a study of incentives at 1-year follow-up and reported it to be the only known study of incentives in this population. Kenyon et al. [
8] investigated the effects of incentives in a perinatal trial, but this was for follow-up when the children were 7 years old. More recently, Bradshaw et al. investigated the use of incentives for 2-year follow-up in a perinatal trial, but this trial population excluded extremely preterm infants [
9].
It is therefore prudent to use opportunities to narrow this evidence gap by investigating which method of incentive provides the greatest return, particularly in perinatal RCTs.
Methods
Aim, design and setting of the SWAT
The main objective of this Study Within A Trial (SWAT; for the definition of a SWAT visit
https://www.nihr.ac.uk/documents/studies-within-a-trial-swat/21512) was to establish whether, in parents of preterm babies, offering an unconditional incentive in advance (with the first mailing of a questionnaire) or promising a conditional incentive (in the first mailing) on completion of a questionnaire, improves response rate. This randomised controlled SWAT was nested within the Speed of Increasing milk Feeds Trial (SIFT), a multicentre RCT in neonatal units in the UK and Ireland caring for very preterm or very low birthweight (VLBW) infants. SIFT recruited 2804 infants, randomised 1:1 to either a slower (18 ml/kg) or faster (30 ml/kg) daily increase in milk volume until reaching full feeds, between 8 June 2013 and 30 June 2015. The primary outcome was survival without moderate or severe disability at 2 years of age (corrected for prematurity), assessed by a questionnaire sent directly to parents (principally mothers). Questionnaires were sent (to all parents where infant(s) vital status and address could be confirmed) by both post and as a link to an online submission form via email and text message where these contact details were available. The SWAT investigated whether an unconditional incentive (a monetary voucher given before completion of a questionnaire) was more effective than a conditional incentive (the promise of a monetary voucher on receipt of a completed questionnaire) and has been reported elsewhere [
10]. For ‘Before’ and ‘After’ letters and reminders see Additional file
1.
Participant characteristics
Eligible participants were parents of infants recruited to SIFT who were due to be sent a questionnaire at 2 years of age (corrected for prematurity) at the SWAT start date (1 Feb 2017). SIFT recruited in 55 centres in the UK and Republic of Ireland. Eligibility criteria for inclusion in SIFT have been described elsewhere [
11]. Participants were traced for survival and to confirm current residence. Where these details remained unknown, parents were not contacted. In addition, parents who had withdrawn consent to the 2-year follow-up were excluded.
Interventions and comparisons
Participants were allocated randomly to one of two groups:
-
A (After): the first letter posted to parents included a promise of an incentive (£15 gift voucher redeemable at high street shops) after receipt of a completed form.
-
B (Before): the first letter posted to parents enclosed the incentive (£15 gift voucher redeemable at high street shops) before the receipt of a completed form.
Participants were randomised with an allocation ratio 1:1 by permuted block randomisation (using variable block sizes of 2, 4, 6 and 8 allocated in proportion to elements of Pascal’s triangle i.e., a ratio of 1:3:3:1) and stratified by original SIFT allocation (faster/slower) and by singleton/multiple birth by the Trial Statistician at the NPEU CTU. The interventions were assigned to the participants immediately because they had all been enrolled to SIFT at that point, so there was no gap during which the sequence needed to be concealed. Infants from multiple births were allocated to the same incentive group. Vouchers were allocated per questionnaire, so parents of multiple births received a voucher for each infant. SIFT office staff at the NPEU Clinical Trials Unit were aware of allocation due to the nature of the interventions and the practicalities involved in sending out the letters and the vouchers.
The incentive was a £15 high street shop voucher (€15 for participants recruited in the Republic of Ireland), sent via post. Reminder letters in both groups mentioned the incentives. Letters to those in the group allocated ‘After’ reiterated the promise of an incentive; letters to those in the group allocated ‘Before’ tactfully mentioned the incentive sent with the first letter. Parents were also reminded to complete the questionnaire via text and/or email during the follow-up window. All parents were offered the option of completing the questionnaire online or, as a last resort, via telephone.
The incentives SWAT was implemented midway through the SIFT follow-up. Parents of SIFT participants who had returned their follow-up questionnaire prior to the incentives study being implemented were sent a £15 voucher in a spirit of fairness.
Outcomes
The primary outcome was the occurrence of questionnaire return, defined as receipt of a completed or partially completed questionnaire at the SIFT office. A questionnaire was considered completed or partially completed if the first three out of the five subsections of the questionnaire were completed (as these sections were required for the derivation of the primary outcome for the main SIFT trial).
Secondary outcomes included:
Total cost included postage, receipt of material via prepaid Freepost envelope, cost of envelopes, supplementary materials (for example, sticker sets sent with questionnaires for infants to play with), and value of gift vouchers. It did not include Freepost licence fee, printing, telephone calls, or trial staff time. All costs for participants were calculated in GBP. The cost of the €15 vouchers sent to participants in the Republic of Ireland was converted to GBP using the exchange rate (via
xe.com) on 10 May 2017, the date of the invoice for these vouchers. Costs for these participants also included Air Mail postage fees.
Sample size
SIFT completed recruitment on 30 June 2015, with 2804 infants randomised. Given an expected 5% mortality rate in the patient population, it was estimated that 1250 of these infants would survive until the projected start date of the SWAT (originally 1 December 2016). It was also estimated that about 10% of the parents of these infants would be lost to the follow-up phase, due to withdrawal of consent or to lack of information on contact details or infant survival status (always checked prior to engaging with parents). This would result in approximately 1100 infants eligible for the SWAT, giving 550 per group.
It was estimated that the response rate with no incentives would be approximately 66%, based on experience from the BOOST-II UK trial with a similar patient population and 2-year follow-up methods [
12].
Based on the most applicable literature on studies investigating incentives, it was anticipated that the addition of an incentive would result in an absolute increase of 10% in the response rate [
4,
8]. Five hundred and fifty infants per group would allow detection of an absolute difference in response rate of around 7% at 90% power and a two-sided 5% level of significance.
Statistical analysis
Infants (i.e. parent responses) were analysed in the groups to which they were randomly assigned, comparing the outcome of all infants allocated to ‘Before’ with all those allocated to the ‘After’ group, regardless of deviation from the protocol.
Baseline demographic information was summarised by randomised group using frequency counts and percentages for categorical data, means and standard deviations for normally distributed continuous data, or medians with interquartile ranges for other continuous data.
Comparative analysis entailed calculating the absolute difference in the proportion responding with corresponding 95% confidence interval (CI), and the difference in mean cost (plus 95% CI). In addition, the cost in £s per 1% increase in response rate was calculated, factoring in administration costs such as the number of reminder letters, as well as the monetary value of the incentive. For other outcomes relating to the method of completion and reminder letters, a similar strategy was used based on the distributions/type of data collected.
The principal comparison was the incentivised ‘Before’ group versus the incentivised ‘After’.
Pre-specified subgroup analysis examined the consistency of the effect of the timing of the incentive for the original allocation in SIFT (slower versus faster) and singleton versus multiple births, using the statistical test of interaction. An exploratory subgroup analysis exploring regional (England, Scotland, Wales, Northern Ireland, and Republic of Ireland) variation was also performed.
Pre-specified exploratory analysis examined the response rate in the period prior to the incentives study starting and during, overall (i.e. irrespective of incentive group allocation) with a 95% confidence interval.
No adjustment was planned for multiple testing since this SWAT involves a very small number of focused hypothesis tests.
Discussion
In this SWAT, allocation to an unconditional incentive (‘Before’ group) produced a significantly more favourable response rate than allocation to a conditional incentive (‘After’ group), with an absolute increase of 6.8% (83.0% versus 76.1% respectively). This concurs with results from previous studies [
3,
6]. The overall response rate was significantly higher after the SWAT was implemented, at 79.5% (95% CI 76.8% to 82.0) compared to 72.1% before (95% CI 69.9% to 74.1%).
The majority of questionnaires were returned by post and this method demonstrated the significant increase between groups. There was little difference between groups for the online or phone-supplied questionnaires, but there were relatively low numbers of responses overall via these two methods.
Both incentive methods are cost-effective in terms of increasing response rate. However, providing an unconditional incentive comes with an additional cost, in this case almost £3 per infant. The group allocated ‘Before’ led to an absolute increase in response rate of 11%, at an average additional cost of £14.80 per infant. The group allocated ‘After’ gave an absolute increase in response rate of 4%, at an average additional cost of £11.84 per infant. So, while the unconditional incentive is more expensive, it represents good value for money with a cost of £1.35 per infant for every 1% increase in response rate. This contrasts with the conditional incentives, which costs an additional £2.95 per infant for every 1% increase in response rate.
There was no evidence of a differential response rate between trial allocation or between single and multiple births. However, there was a greater response from parents of multiples on the group allocated ‘Before’. There could be a cumulative factor involved here, as parents of multiples received greater in advance incentives (£30 for twins, £45 for triplets). However, this promising single result has to be interpreted in the context of three similar studies. The totality of evidence regarding the effect on retention of the addition of a monetary incentive (unconditional) versus the addition of a monetary reward (conditional) remains highly uncertain though, especially in the presence of significant heterogeneity [
13].
Strengths and limitations
This SWAT nested within a perinatal RCT contributes to the evidence base for the effectiveness of incentives in this context. The study was run efficiently requiring minimal resources as it was integrated into the existing SIFT follow-up schedule. Ethics approval delays and logistics meant the SWAT study did not begin until February 2017, resulting in fewer participants still due to receive their questionnaire. Despite this, results showed a significant increase in the response rate in the group allocated an unconditional incentive (‘Before’ group).
This study is limited to a particular study population: parents of infants born extremely prematurely. They may be more likely to be interested in their infants’ progress. It is also limited to the follow-up juncture of 2 years. Given the scarcity of studies in the perinatal field, it is important to bear this in mind. Longer follow-up periods (for example at school age), and different modes of data collection and assessment (for example clinical assessment) and other incentive amounts may produce different results.
Acknowledgements
We are grateful to the parents of participating infants and the administrative and support colleagues at the NPEU CTU. We would also like to thank Madeline Walsh for her help with editing, reformatting, and general comments and Andrew King for proofreading. We also thank Jenny Shilton-Osborne at NPEU for registering the SWAT.
SIFT collaborative group:
Vasha Bari1, Janet Berrington2, Ursula Bowler1, Elaine Boyle3, Jon Dorling4, Nicholas Embleton2, Oliver Hewer1, Madeleine Hurd1, Samantha Johnson3, Edmund Juszczak1,7, Andrew King1, Alison Leaf1, Louise Linsell1, Kenny McCormick5, William McGuire6, David Murray1, Christopher Partlett1,7, Tracy Roberts8, Ben Stenson9.
1 National Perinatal Epidemiology Unit, Nuffield Department of Population Health, University of Oxford, Oxford, UK
2 Newcastle Neonatal Service, Royal Victoria Infirmary, Newcastle upon Tyne, UK
3 Department of Health Sciences, University of Leicester, Leicester, UK
4 Division of Neonatal-Perinatal Medicine, Department of Pediatrics, Faculty of Medicine, Dalhousie University, Halifax, NS, Canada
5 John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
6 Centre for Reviews and Dissemination, University of York, York, UK
7 Nottingham Clinical Trials Unit, University of Nottingham, Nottingham, UK
8 Health Economics Unit, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
9 The Simpson Centre for Reproductive Health, Royal Infirmary of Edinburgh, University of Edinburgh, Edinburgh, UK
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.