Keywords
Retention, pen, social incentive, cover letter, randomised controlled trial, embedded trial, SWAT, postal questionnaire, response rate
-
Views
-
Downloads
How to cite this article
James S, Parker A, Cockayne S et al. Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.12688/f1000research.23767.2)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Export Citation
Sciwheel
EndNote
Ref. Manager
Bibtex
ProCite
Sente
▬
✚
Research Article
Revised
Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial
[version 2; peer review: 2 approved]
PUBLISHED 08 Nov 2021
OPEN PEER REVIEW
REVIEWER STATUS
This article is included in the Studies Within A Trial (SWAT) collection.
Abstract
Corresponding author:
Sophie James
Competing interests:
No competing interests were disclosed.
Grant information:
The OTIS study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number: 14/49/149). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This SWAT was funded by PROMETHEUS and the results from this project will contribute to the evidence towards trial methodology for improving retention of participants. This will study will be linked with a national research programme PROMETHEUS led by York Trials Unit (https://www.york.ac.uk/healthsciences/research/trials/research/swats/prometheus/) and findings will be combined with other studies in meta-analyses to detect small but cost effective differences. This will help future trials to be designed with effective interventions in place to maximise retention and avoid introduction of bias and reduced study power.
The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Copyright:
© 2021 James S et al.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
How to cite: James S, Parker A, Cockayne S et al. Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.12688/f1000research.23767.2)
First published: 17 Jun 2020, 9:623 (https://doi.org/10.12688/f1000research.23767.1)
Latest published: 08 Nov 2021, 9:623 (https://doi.org/10.12688/f1000research.23767.2)
The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Revised Amendments from Version 1
We have added additional detail and clarification to the revised version of the article in response to comments raised by the reviewers. We have made clearer the definition and concept of social incentive which underpinned the intervention cover letter and the difference in this terminology from social pressure or social reward.
We have also made minor amendments to Table 2 to remove BMI and EQ-5D-5L scores which the reviewers felt were not relevant. We added details about the PROMETHEUS programme and edited the description of the meta-analysis to address reviewer comments on the reason for the meta-analysis and the other studies that were included in this. We have also updated the discussion to also reflect the point that is mentioned in the methods that all participants were given a £5 monetary reward.
Finally, we have updated two figures in the meta-analysis, due to a recently published new version (version 2) of one of the studies included in the meta-analysis ('Mitchell A, Cook L, Dean A, et al.: Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial') being published on F1000. This version made amendments to their results in light of a duplicate randomisation that was found in the host trial, these changes were negligible and did not affect the interpretation of the results. However this means that for accuracy the meta analysis figure of this paper has been updated to include these new figures.
See the authors' detailed response to the review by Susanne Grylka-Baeschlin
See the authors' detailed response to the review by Taylor Coffey, Louisa Lawrie and Eilidh Duncan
Introduction
Randomised controlled trials (RCTs) are the gold standard to assess effectiveness of treatment options and to inform care decisions1, yet only a few hundred studies exist to assess the effectiveness of different methods to improve retention or recruitment into RCTs2.
Trial methodologists and funders have highlighted the need to evaluate participant recruitment and retention strategies in order to provide evidence on which to base decisions around the design and conduct of RCTs3.
Postal questionnaires are frequently used in randomised controlled trials to collect outcome data on participants; however, poor response can introduce bias, affect generalisability and validity, and reduce statistical power. Several systematic reviews report on the topic of retention strategies, including improving response rates to questionnaires4–7. However, there remains a lack of definitive evidence regarding some commonly adopted practices such as sending a pen or using a cover letter with a questionnaire to encourage the participant to return it8–10. The results of a study within a trial (SWAT) evaluating these two strategies are reported here.
Methods
Design
A two-by-two factorial RCT was embedded within the OTIS trial of occupational therapist-led home assessment and modification for the prevention of falls (ISRCTN22202133)11. OTIS recruited participants over the age of 65 years who were at risk of falling. Participants were randomised to receive an occupational therapist delivered visit or usual care. They were followed up for 12 months for falls data and were sent postal questionnaires at four, eight and 12 months. This SWAT was embedded at the 12-month time point. Ethical approval for this SWAT was received from the NHS West of Scotland Research Ethics Committee 3 (16/WS/0154) and Health Research Authority and Research Ethics approval in July 2018. Approvals were obtained from the University of York, Department of Health Sciences Research Governance Committee. Participants provided informed consent to be enrolled into the OTIS trial and to be sent study related information by post. Consent for the SWAT was therefore waived by the above-named ethics committee.
Participants
A total of 779 participants due to receive their 12-month questionnaire between 16th October 2018 and 2nd August 2019 were randomised into the SWAT in a single tranche in September 2018. Participants who had withdrawn from the OTIS study prior to this were excluded from randomisation.
The allocation sequence was generated by the OTIS statistician, who was not involved with the sending of the questionnaires, using STATA v1512. The identification numbers of OTIS participants to be involved in the SWAT were randomised 1:1:1:1 in a single block. Because there were no descriptive details of the participants attached to the identification numbers this meant the randomisation was concealed.
Interventions
Table 1 details the combination of interventions sent in the post with the 12-month questionnaire. We included an unconditional £5 note with the questionnaire for all participants.
The non-standard cover letter offered a mild level of social incentive, in the form of a personalised table that indicated whether or not a questionnaire had been received from the participant at the earlier (4 and 8-month) time points. The concept of social incentive that underpinned the intervention for this study was that a social incentive is something that persuades people to behave in a certain way by the promise that their actions will be noticed or made public10. Therefore, the cover letter was intended to highlight to the participant that their questionnaire responses are noted and valued10.
Table 1. Intervention groups.
| Pen York Trials Unit branded pen, standard cover letter (Supplementary File 1)* | Control Group No pen, standard cover letter (Supplementary File 4). |
| Pen and Social Incentive Cover Letter York Trials Unit branded pen, social incentive cover letter (Supplementary File 3). | Social Incentive cover letter Social incentive cover letter (Supplementary File 2), no pen. |
*Supplementary Files are available as Extended data13.
Blinding and quality assurance
Participants were blind to their participation. Research administrators and research team members posting the questionnaire packs were not blind to the intervention; however, administrators who recorded the outcome data were blind to allocation.
Primary objective
To assess whether a pen and/or social incentive text cover letter sent with the 12-month questionnaire increased postal questionnaire response rates for participants in the OTIS trial.
Primary outcome
The primary outcome was response rate, defined as the proportion of participants in each group who returned the 12-month questionnaire.
Statistical analysis
The data were analysed in SPSS v2514 using two-sided tests at the 5% significance level on an intention-to-treat basis. Participants who withdrew or died before the 12-month questionnaire was sent were excluded from the analysis. The primary outcome was compared using a logistic regression model adjusting for age (retention is generally higher in participants ˂75 years and older adults may respond differently to incentives15), gender (to control for potential differences in anticipation of social incentives between males and females16) and host trial treatment allocation. The presence of an interaction between the two interventions was tested by introducing the interaction term into the logisit model. Time to questionnaire return (calculated as days from questionnaire sent to return) was analysed using Cox Proportional Hazards regression, adjusting for the same covariates as in the primary analysis. The proportional hazards assumption was assessed using Schoenfeld residuals17. Completeness of response (defined as number of items completed) was analysed by linear regression model and adjusted as for the primary analysis.
Cost effectiveness was calculated for each group using the total cost of the pen/letter/postage/stationary and staff time.
Due to SWATs typically being under-powered to show small effects, it is essential that the results are seen within the context of the wider literature. A fixed effect meta-analysis using the Mantel-Haenszel method was conducted using review manager v5.318 to pool the results of this study for enclosing a pen with the 12-month questionnaire with other RCT evidence. These were located utilising the Cochrane systematic review7 search strategy (Supplementary file 14) in MEDLINE and EMBASE, along with hand searching of previous systematic reviews references, published retention research reference lists, conference papers and co-author personal knowledge of studies. The results of this study were pooled with four previous SWATs8,9,19,20 investigating the same intervention, with the same dichotomous outcome of response to the questionnaire or not. Pooled odds ratios and corresponding 95% CIs were calculated. Heterogeneity between trials was assessed using the Chi-squared and I2 statistics. The meta-analysis was facilitated by the PROMoting THE USE of SWATs (PROMETHEUS) programme, which supports host trial teams to conducted SWATs and for data obtained to be collated and meta-analysed.
A meta-analysis of the results of the social incentive intervention was not undertaken as the only previous study using this was conducted within a cohort study rather than an RCT10.
Results
Figure 1 depicts the recruitment and retention of participants in the embedded trial. Table 2 presents summary statistics for the baseline characteristics of the SWAT participants.
Figure 1. Flow diagram depicting the recruitment and retention of participants in this embedded trial.
Table 2. Baseline characteristics of the SWAT participants.
Primary outcome
Between randomisation into the SWAT and being sent their 12-month questionnaire, 24 randomised participants either died or withdrew from the host trial and so were not sent the questionnaire. A total of 721/755 (95.5%) returned the 12-month questionnaire. The response rate was identical in the pen only group (184/192, 95.8%), social incentive cover letter only group (181/189, 95.8%) and control group (182/190, 95.8%). However, it was marginally lower in the pen and social incentive cover letter group (174/184, 94.6%).
No evidence of a difference in response rates was found between participants with or without pens (pen: 358/376 [95.2%]; no pen: 363/379 [95.8%]; adjusted OR 0.90, 95% CI 0.45 to 1.80, p=0.77) nor with or without the social incentive cover letter (cover letter: 355/373 [95.2%]; no cover letter: 366/382 [95.8%]; adjusted OR 0.84, 95% CI 0.42 to 1.69, p=0.63) (Table 3).
Table 3. Primary outcome results.
The interaction between the interventions was found to be non-significant (interaction effect size estimate OR 0.79 95% CI 0.20, 3.15 p = 0.74).
Secondary outcomes
Time to return. Median time to return the questionnaire was nine days, with a mean of 12.2 days. No statistically significant difference between the groups was found (Table 4).
Table 4. Secondary outcome results.
Reminders sent. In total, 83/755 (11.0%) participants required a reminder letter. The pen and social incentive cover letter group required the least reminders (19/184 10.3%) and the control group required the most reminders (24/190 12.6%). No statistically significant evidence was found of a difference of participants requiring a reminder between the groups (Table 4).
Completeness of response. Overall average completeness of the questionnaires was 27.8/31 questions (89.6% complete) with no evidence of a difference in completeness of the questionnaire between pen received or not (Table 4).
Cost effectiveness. Due to the non-statistically significant effect of the interventions on response rates calculating overall associated costs provides evidence of potential cost savings not to send the social incentive cover letter and/or pen (Extended data: Supplementary File 913).
Meta-analysis
A fixed effect meta-analysis of enclosing a pen with a follow-up postal questionnaire on response rate was conducted (Figure 2). This included five studies8,9,21,22 (n=13012 participants) and gave a statistically significant pooled OR favouring the intervention (1.21, 95% CI 1.09 to 1.35 p = 0.0004). Negligible heterogeneity was observed (chi-squared = 2.88 I2= 0%). The risk of bias was low, as indicated by the Cochrane’s risk of bias tool assessment undertaken23 (Extended data: Supplementary File 1013).
Figure 2. Meta-analysis of enclosing a pen with a questionnaire to increase response rate to a postal questionnaire.
Discussion
This SWAT found no evidence that sending a pen and/or a social incentive cover letter with a postal, trial follow-up questionnaire improved response rate, time to return, requirement for a reminder, or questionnaire completeness.
A limitation was the average age of the participants (79.9 years) giving a narrow age demographic thus restricting generalisability of results. Further investigation of the pen and social incentive cover letter in RCTs are required across more diverse populations.
The OTIS trial hosted three other methodological SWATs; therefore, there was a potential for contamination or interaction. It is preferable to plan all SWATs that will be undertaken in the early design stages19, to ensure they are planned accordingly to reduce the potential of this.
The overall response rate of the 12-month postal questionnaire for all SWAT participants was 95.7%, which may have been helped by the inclusion of £5 to all participants as standard. This high response rate is therefore difficult to improve upon, furthermore the incentives may not have been as effective with participants who are very committed to the behaviour10. The incentive required for committed participants may be different10,20. A learning point being that future SWATS testing these interventions should avoid doing so in trials with already high response rates.
Conclusion
Whilst neither the pen nor the social incentive cover letter showed an effect on response rate, the meta-analysis evidence remains that including a pen increases response rates. This reinforces that for interventions where small effects are likely, it is important to undertake a number of trials and combine these to be confident of an intervention’s effectiveness. Further investigation of the social incentive cover letter in RCTs is required to determine effectiveness.
Data availability
Underlying data
Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT, https://doi.org/10.17605/OSF.IO/7TDRB13.
Extended data
Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT, https://doi.org/10.17605/OSF.IO/7TDRB13.
This project contains the following extended data:
Full study protocol
Supplementary File 1- Cover letter for the Pen only group.
Supplementary File 2 - Cover letter for the Social incentive cover letter only group.
Supplementary File 3 - Cover letter for the Pen and social incentive cover letter group.
Supplementary File 4 - Cover letter for the control group.
Supplementary File 5 - Results table by intervention group
Supplementary File 6 - Graph Survival curve of pen vs no pen and time taken to return 12-month questionnaire.
Supplementary File 7 - Graph Survival curve of Social incentive cover letter vs no social incentive cover letter and time taken to return 12-month questionnaire.
Supplementary File 8- Survival curve of host trial allocation and time taken to return 12-month questionnaire.
Supplementary File 9 – Costings table
Supplementary File 10 – Cochrane Risk of bias tool assessments for Bell et al., (2016)8, Sharp et al., (2006)9, Cunningham-Burley et al., (2020)21, Mitchell et al., (2020)22 and James et al., (2020).
Supplementary File 11 – Summary of all SWATs undertaken in the OTIS study
Supplementary File 12 – Copy of the OTIS reminder letter
Supplementary file 13 – Summary of studies included in the meta-analysis
Supplementary File 14 – Copy of the search strategies of Brueton et al., (2014)
Reporting guidelines
Open Science Framework: CONSORT checklist for ‘Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial’, https://doi.org/10.17605/OSF.IO/TYJDP13.
Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).
Download
Export To
metrics
| Views | Downloads | |
|---|---|---|
| F1000Research | - | - |
|
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Citations
CITE
how to cite this article
James S, Parker A, Cockayne S et al. Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved] F1000Research 2021, 9:623 (https://doi.org/10.12688/f1000research.23767.2)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.
Open Peer Review
Current Reviewer Status:
Key to Reviewer Statuses
VIEW
HIDE
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations
A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 2
VERSION 2
PUBLISHED 08 Nov 2021
Revised
Views
6
How to cite this report:
Duncan E, Lawrie L and Coffey T. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.77829.r99236)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v2#referee-response-99236
https://f1000research.com/articles/9-623/v2#referee-response-99236
NOTE: it is important to ensure the information in square brackets after the title is included in this citation.
Reviewer Report 19 Jan 2022
Louisa Lawrie, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Taylor Coffey, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Approved
VIEWS 6
Comments have
... Continue reading
CITE
HOW TO CITE THIS REPORT
Duncan E, Lawrie L and Coffey T. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.77829.r99236)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v2#referee-response-99236
https://f1000research.com/articles/9-623/v2#referee-response-99236
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
5
How to cite this report:
Grylka-Baeschlin S. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.77829.r99235)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v2#referee-response-99235
https://f1000research.com/articles/9-623/v2#referee-response-99235
NOTE: it is important to ensure the information in square brackets after the title is included in this citation.
Reviewer Report 06 Dec 2021
Approved
VIEWS 5
Thank you for thoroughly revising this manuscript. The limitations of the study are now addressed in the discussion, nevertheless diminish the importance of the results. However, the significance of the study was increased by the meta-analysis. I only have one
... Continue reading
CITE
HOW TO CITE THIS REPORT
Grylka-Baeschlin S. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.77829.r99235)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v2#referee-response-99235
https://f1000research.com/articles/9-623/v2#referee-response-99235
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Version 1
VERSION 1
PUBLISHED 17 Jun 2020
Views
18
How to cite this report:
Grylka-Baeschlin S. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.26227.r91091)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v1#referee-response-91091
https://f1000research.com/articles/9-623/v1#referee-response-91091
NOTE: it is important to ensure the information in square brackets after the title is included in this citation.
Reviewer Report 16 Aug 2021
Approved with Reservations
VIEWS 18
Thank you very much for giving me the opportunity to review this article. Studies Within A Trial (SWAT) are important and a good opportunity to make optimal use of resources. Despite their disadvantage that the original trial has not been
... Continue reading
CITE
HOW TO CITE THIS REPORT
Grylka-Baeschlin S. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.26227.r91091)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v1#referee-response-91091
https://f1000research.com/articles/9-623/v1#referee-response-91091
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
-
Author Response 08 Nov 2021Sophie James, York Trials Unit, University of York, UK, York, Y010 5DD, UK08 Nov 2021Author ResponseWe thank the reviewer for the helpful comments and points raised. We hope that our responses are satisfactory and the article has been updated in response to the points raised ... Continue reading
COMMENTS ON THIS REPORT
-
Author Response 08 Nov 2021Sophie James, York Trials Unit, University of York, UK, York, Y010 5DD, UK08 Nov 2021Author ResponseWe thank the reviewer for the helpful comments and points raised. We hope that our responses are satisfactory and the article has been updated in response to the points raised ... Continue reading
Views
68
How to cite this report:
Coffey T, Lawrie L and Duncan E. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.26227.r66639)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v1#referee-response-66639
https://f1000research.com/articles/9-623/v1#referee-response-66639
NOTE: it is important to ensure the information in square brackets after the title is included in this citation.
Reviewer Report 21 Jul 2020
Taylor Coffey, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Louisa Lawrie, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Approved with Reservations
VIEWS 68
The present article by James et al. details a study within a trial (SWAT) of including a pen and/or cover letter with a postal questionnaire. A two-by-two factorial randomized control trial was conducted that assessed the effect of including a
... Continue reading
CITE
HOW TO CITE THIS REPORT
Coffey T, Lawrie L and Duncan E. Reviewer Report For: Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial [version 2; peer review: 2 approved]. F1000Research 2021, 9:623 (https://doi.org/10.5256/f1000research.26227.r66639)
The direct URL for this report is:
https://f1000research.com/articles/9-623/v1#referee-response-66639
https://f1000research.com/articles/9-623/v1#referee-response-66639
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
-
Author Response 08 Nov 2021Sophie James, York Trials Unit, University of York, UK, York, Y010 5DD, UK08 Nov 2021Author ResponseMajor points to address: The peer review team would like to highlight the following areas for clarification.
Potentially confounding incentives- The ‘social incentive’ intervention cover letter is described
COMMENTS ON THIS REPORT
-
Author Response 08 Nov 2021Sophie James, York Trials Unit, University of York, UK, York, Y010 5DD, UK08 Nov 2021Author ResponseMajor points to address: The peer review team would like to highlight the following areas for clarification.
Potentially confounding incentives- The ‘social incentive’ intervention cover letter is described
Comments on this article Comments (0)