The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT)

Catherine Arundel; Elizabeth Coleman; Caroline Fairhurst; Emily Peckham; Della Bailey; Simon Gilbody

doi:10.12688/f1000research.21059.2

Home Browse The effectiveness of a contingent financial incentive to improve trial...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT)

[version 2; peer review: 2 approved]

Previously titled: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up

Catherine Arundel ¹, Elizabeth Coleman¹, Caroline Fairhurst¹, Emily Peckham¹, Della Bailey¹, Simon Gilbody¹

Catherine Arundel ¹, Elizabeth Coleman¹, [...] Caroline Fairhurst¹, Emily Peckham¹, Della Bailey¹, Simon Gilbody¹

PUBLISHED 17 Jul 2020

Author details Author details

¹ Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

Catherine Arundel
Roles: Conceptualization, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Elizabeth Coleman
Roles: Formal Analysis, Methodology, Writing – Review & Editing

Caroline Fairhurst
Roles: Formal Analysis, Methodology, Writing – Review & Editing

Emily Peckham
Roles: Conceptualization, Methodology, Project Administration, Writing – Review & Editing

Della Bailey
Roles: Project Administration, Writing – Review & Editing

Simon Gilbody
Roles: Conceptualization, Methodology, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Studies Within A Trial (SWAT) collection.

Abstract

Objectives
To evaluate the effectiveness of a contingent financial incentive (£10 note in addition to a routinely provided £10 voucher) versus no contingent financial incentive, on improving the retention rate in a randomised controlled trial (RCT).
Methods
A two arm ‘Study within a Trial’ (SWAT) embedded within a host RCT (SCIMITAR+). Participants were randomised to the SWAT using a 2:1 (intervention:control) allocation ratio. The primary outcome measure was the proportion of participants completing a CO breath measurement at the first SCIMITAR+ follow up time point (6 months). Secondary outcomes were withdrawing from follow-up after contact and time from assessment due date to completion. Analyses were conducted using logistic or Cox Proportional Hazards regression as appropriate.
Results
A total of 434 participants were randomised into this SWAT. Completion of the CO breath measurement at 6 months was 88.5% (n=247) in the intervention arm of the SWAT and 85.4% (n=123) in the control arm (Difference 3.1%, OR 1.29, 95% CI 0.71-2.33, p=0.41). There was also no evidence of a difference in the proportion of participants withdrawing from follow-up after contact (intervention n=7 (2.5%), control n=5 (3.5%); OR 0.76, 95% CI 0.23-2.44, p=0.64), nor in terms of proximity of 6-month visit completion to due date (HR 1.07, 95% CI 0.86-1.33, p=0.55).
Conclusion
It is unclear if contingent financial incentives increased rates of face-to-face follow-up completion within the SCIMITAR+ trial population. However, the sample size of this SWAT was constrained by the size of the host trial and power was limited. This SWAT adds to the body of evidence for initiatives to increase response rates in trials.

Keywords

SWAT, retention, randomized controlled trial

Corresponding author: Catherine Arundel

Competing interests: No competing interests were disclosed.

Grant information: This work was conducted as part of the SCIMITAR+ Trial which was funded by the NIHR Health Technology Assessment Programme (11/136/52). The views expressed are those of the author(s) and not necessarily those of the NIHR of the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2020 Arundel C et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Arundel C, Coleman E, Fairhurst C et al. The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT) [version 2; peer review: 2 approved]. F1000Research 2020, 8:1937 (https://doi.org/10.12688/f1000research.21059.2) First published: 20 Nov 2019, 8:1937 (https://doi.org/10.12688/f1000research.21059.1) Latest published: 17 Jul 2020, 8:1937 (https://doi.org/10.12688/f1000research.21059.2)

Revised Amendments from Version 1

Manuscript title amended to make clear this manuscript relates to the SWAT and not the SCIMITAR+ Trial in the main which may have been implied by the original wording.
Two P values were inadvertently included in the Abstract Results section. The incorrect P value has been removed.
Given the limited power of SWATs, reference to statistical significance may be misleading. All references to ‘statistical significance’ have been amended to reflect the lack of clarity around the estimate due to wide confidence intervals.
Addition of further justification of SWAT in Introduction to make clear the evidence base on which this SWAT builds.
Revision of ‘reward’ to ‘incentive’ throughout the manuscript for consistency.
Addition of clarification on randomisation process (Methods – Randomisation; Results), to justify exclusion of some participants post randomisation
Addition of detail in relation to proportional hazards assumptions assessment in relation to secondary outcome (proximity to due date).

See the authors' detailed response to the review by Frances Shiely
See the authors' detailed response to the review by Sarah A. Rhodes

Introduction

Attrition is a major problem for randomised controlled trials (RCTs) with 25% experiencing more than 10% attrition¹.

Bower et al. (2014)² identified financial incentives as an effective retention strategy (RR 1.18; 95% CI 1.09 to 1.28), and effectiveness was increased if this incentive was provided on receipt of a completed questionnaire (RR 1.25; 95% CI 1.14 to 1.38). Bailey et al. (2013)³ identified that varying the incentive level (£20 compared to £10) increased response to postal questionnaires by up to 10%.

Contingent incentives have shown to be effective in increasing questionnaire response rate, but have not been tested in relation to face-to face-visits. This SWAT evaluated the effectiveness of a contingent financial incentive - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial incentive, on improving the retention rate in the SCIMITAR+ trial.

Methods

Design

This SWAT was embedded within the SCIMITAR+ RCT which evaluated the effectiveness of a bespoke, individually-tailored, smoking cessation programme, compared to usual care, for adult smokers with severe mental ill health conditions⁴. The SCIMITAR+ Trial was registered prospectively: ISRCTN72955454

This paper refers to the methods and results of the SWAT only.

Participants

The SWAT⁵ was conducted in 21 NHS Trusts and 16 primary care settings and was implemented after the start of SCIMITAR+ follow-up. Participants were eligible for this SWAT if they reached the SCIMITAR+ 6-month follow-up on or after 31^st September 2016.

Intervention

When participants in the SWAT intervention group were contacted by the research team to arrange their follow-up appointment, they were advised of the potential of receiving £10 cash contingent on providing a carbon monoxide (CO) breath measure as part of their 6-month face-to-face study appointment, in addition to the £10 gift voucher routinely provided to all participants. Participants in both groups received all other pre-planned retention strategies within SCIMITAR+.

Outcomes

The primary outcome for the SWAT was the proportion of participants completing a CO breath measurement at the SCIMITAR+ 6 month follow-up time-point. Secondary outcome measures were: i) the proximity of visit completion to visit due date; ii) the proportion of participants withdrawing from follow-up in the two months after initial contact was made to arrange the 6-month visit.

Sample size

The sample size was determined by the number of participants followed-up at 6 months in SCIMITAR+ from the point at which this SWAT was embedded.

Randomisation

Simple randomisation using random numbers was carried out by an independent statistician at the York Trials Unit using Stata v13⁶. All potentially participants eligible for inclusion, at the time of randomisation, were allocated with a 2:1 allocation ratio (intervention:control) due to the anticipated effectiveness of financial incentives increasing questionnaire response rates.

Blinding

It was not possible to blind research staff to the participant’s allocation. Participants were not informed about the SWAT so were blind to the study hypothesis.

Approvals

The SWAT was approved by the Research Ethics Committee Yorkshire and Humber – Leeds East (15/YH/0051). As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive the additional incentive, informed consent was not obtained for participation in this SWAT.

Statistical analysis

Analyses were conducted using Stata v15⁷ on an intention to treat basis using two-sided statistical tests at the 5% significance level, adjusting for host trial allocation.

The proportion of participants who provided a 6-month CO breath measure was analysed using logistic regression. The odds ratio (OR), 95% confidence interval (CI) and p-value are presented.

The 6-month appointment due date was 183 days after randomisation. Participants who withdrew a month either side of the 6-month appointment due date were classed as withdrawn. The proportion of participants withdrawing from SCIMITAR+ in the two months after contact were analysed in the same way as the primary outcome.

A Cox Proportional Hazard model compared the proximity of the visit completion to visit due date (time in days). Participants who completed their visit before or on the due date had their time-to-visit set to 0.1.

Results

In total, 434 participants were randomised into this SWAT (n=286, 65.9% intervention group; n=148, 34.1% control group). Due to randomisation occurring at a single time point, eleven participants were excluded from analysis as they withdrew from SCIMITAR+ following randomisation but prior to being contacted for their 6-month visit. There were 423 eligible participants (intervention group n=279, 66.0%; control group n=144, 34.0%) (Figure 1).

Figure 1. Study flow diagram.

Overall, 87.5% (n=370) of participants completed the CO breath measurement at 6 months and it was unclear if there was a difference between intervention (88.5%, n=247) and control groups (85.4%, n=123) (3.1% difference, OR 1.29, 95% CI 0.71-2.33, p=0.41). It was unclear if there was any difference in withdrawals between trials arms (intervention n=7, 2.8%; control n=5, 3.5%; OR 0.76, 95% CI 0.23-2.44, p=0.64) or proximity of 6-month visit completion to due date (hazard ratio 1.07, 95% CI 0.86-1.33, p=0.55). Assessment of assumptions noted that the Kaplan Meier curves crossed, however Grambsh and Therneau testing did not indicate any statistically significant non proportionality (SWAT allocation p=.047; Main trial allocation p=0.52; global test p=0.63).

Discussion

It is unclear whether an additional £10 in cash increases the likelihood of participants completing a face-to-face follow-up, the proportion of the participants withdrawing, or have an effect on the proximity of the visit to the due date.

Strengths and limitations

A small positive difference was observed; however, despite the large sample size, the study was underpowered to confidently rule out a small ‘true’ effect. Due to the small effect size (3.1% increase in response) the cost per additional person attending would be in excess of £300.

Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.

Data was not collected on how study staff followed the guidance on discussing the contingent £10 note to intervention group participants when arranging follow up visits. This may have diluted the effect of the intervention.

Conclusion

It is unclear if contingent financial incentives increase rates of face-to-face follow-up completion in this trial. However, there were sample size and power limitations. Future SWATs are needed to add to the evidence base.

Data availability

Underlying data

Figshare: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set, https://doi.org/10.6084/m9.figshare.10060202.v2⁸.

Reporting guidelines

Figshare: CONSORT checklist for SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up, https://doi.org/10.6084/m9.figshare.10060202.v2⁸.

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Acknowledgements

The SCIMITAR+ team acknowledge the support of the PROMETHEUS programme (MRC MR/R013748/1) in developing this manuscript.

Faculty Opinions recommended

References

1. Hewitt CE, Kumaravel B, Dumville JC, et al.: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010; 63(11): 1264–70. PubMed Abstract | Publisher Full Text
2. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15: 399. PubMed Abstract | Publisher Full Text | Free Full Text
3. Bailey JV, Pavlou M, Copas A, et al.: The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res. 2013; 15(12): e278. PubMed Abstract | Publisher Full Text | Free Full Text
4. Peckham E, Arundel C, Bailey D, et al.: Smoking Cessation Intervention for Severe Mental Ill Health Trial (SCIMITAR+): study protocol for a randomised controlled trial. Trials. 2017; 18(1): 44. PubMed Abstract | Publisher Full Text | Free Full Text
5. SWAT 48 – Effects of a £ 10 note on retention. Reference Source
6. StataCorp: Stata Statistical Software: Release 13. College Station, TX: StataCorp LP. 2013.
7. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC. 2017.
8. Arundel C, Coleman E, Fairhurst C, et al.: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set. figshare. Dataset. 2019. http://www.doi.org/10.6084/m9.figshare.10060202.v2

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 20 Nov 2019

Author details Author details

¹ Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

Catherine Arundel
Roles: Conceptualization, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Elizabeth Coleman
Roles: Formal Analysis, Methodology, Writing – Review & Editing

Caroline Fairhurst
Roles: Formal Analysis, Methodology, Writing – Review & Editing

Emily Peckham
Roles: Conceptualization, Methodology, Project Administration, Writing – Review & Editing

Della Bailey
Roles: Project Administration, Writing – Review & Editing

Simon Gilbody
Roles: Conceptualization, Methodology, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This work was conducted as part of the SCIMITAR+ Trial which was funded by the NIHR Health Technology Assessment Programme (11/136/52). The views expressed are those of the author(s) and not necessarily those of the NIHR of the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 17 Jul 2020, 8:1937

https://doi.org/10.12688/f1000research.21059.2

version 1

Published: 20 Nov 2019, 8:1937

https://doi.org/10.12688/f1000research.21059.1

© 2020 Arundel C et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Arundel C, Coleman E, Fairhurst C et al. The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT) [version 2; peer review: 2 approved] F1000Research 2020, 8:1937 (https://doi.org/10.12688/f1000research.21059.2)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 2

VERSION 2

PUBLISHED 17 Jul 2020

Revised

Views

Reviewer Report 20 Jul 2020

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland; HRB Clinical Research Facility, University College Cork, Cork, Ireland

Approved

https://doi.org/10.5256/f1000research.27418.r67390

Thank you to the authors for making these revisions and providing a detailed response to my original ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 20 Nov 2019

Views

Reviewer Report 24 Apr 2020

Sarah A. Rhodes, Centre for Biostatistics, School of Health Sciences, University of Manchester, Manchester, UK

Approved

https://doi.org/10.5256/f1000research.23176.r62472

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Statistics, randomised trials, embedded trials, meta-analysis.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

17 Jul 2020

Author Response

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different ... Continue reading There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point.
As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.
The results section has been updated to detail assessment of hazards assumptions.
There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point.
As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.
The results section has been updated to detail assessment of hazards assumptions.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

17 Jul 2020

Author Response

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different ... Continue reading There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point.
As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.
The results section has been updated to detail assessment of hazards assumptions.
There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point.
As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.
The results section has been updated to detail assessment of hazards assumptions.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 03 Dec 2019

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland; HRB Clinical Research Facility, University College Cork, Cork, Ireland

Not Approved

https://doi.org/10.5256/f1000research.23176.r56823

Thank you for the opportunity to review this SWAT. I am in favour of publishing negative results as well as positive results, so my reason for rejecting this SWAT is not because the findings are negative. I wish to be clear on that.

This was a SWAT within the SCIMITAR+ trial. The purpose of the SWAT was to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only. SCIMITAR+ was a smoking cessation trial. The authors conclude that contingent financial incentives did not statistically significantly increase rates of face-to-face follow-up completion within the SCIMITAR+ trial.

In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome. The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive. It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

No
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No source data required
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Bailey JV, Pavlou M, Copas A, McCarthy O, et al.: The Sexunzipped trial: optimizing the design of online randomized controlled trials.J Med Internet Res. 2013; 15 (12): e278 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Epidemiology; trial methodological reserach

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Author Response 17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

17 Jul 2020

Author Response

In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a ... Continue reading In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.
We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.
Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.
Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.
We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹
There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.
We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.
In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.
We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.
Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.
Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.
We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹
There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.
We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

17 Jul 2020

Author Response

In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a ... Continue reading In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.
We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.
Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.
Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.
We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹
There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.
We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.
In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.
We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.
Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.
Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.
We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹
There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.
We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 20 Nov 2019

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 17 Jul 20	read
Version 1 20 Nov 19	read	read

Frances Shiely, University College Cork, Cork, Ireland; University College Cork, Cork, Ireland
Sarah A. Rhodes, University of Manchester, Manchester, UK

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

7 Views

20 Jul 2020 | for Version 2

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland; HRB Clinical Research Facility, University College Cork, Cork, Ireland

7 Views Cite this report Responses(0)

Approved

Thank you to the authors for making these revisions and providing a detailed response to my original queries. I think the revisions have greatly improved the article and am happy to recommend it be accepted.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology; trial methodological reserach

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

12 Views

24 Apr 2020 | for Version 1

Sarah A. Rhodes, Centre for Biostatistics, School of Health Sciences, University of Manchester, Manchester, UK

12 Views Cite this report Responses(1)

Approved

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Statistics, randomised trials, embedded trials, meta-analysis.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.
We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.
We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.
We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.
We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.
We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.
We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point.
As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.
For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.
The results section has been updated to detail assessment of hazards assumptions.

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

44 Views

03 Dec 2019 | for Version 1

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland; HRB Clinical Research Facility, University College Cork, Cork, Ireland

44 Views Cite this report Responses(1)

Not Approved

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

No
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No source data required
Are the conclusions drawn adequately supported by the results?

Partly

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology; trial methodological reserach

Respond to this report

Responses (1)

Author Response

17 Jul 2020

Catherine Arundel, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

In the background section of the abstract, you describe the SWAT as “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.
We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.
Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.
Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.
We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire.¹
There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.
We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Hewitt CE, Kumaravel B, Dumville JC, et al.: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010; 63(11): 1264–70. PubMed Abstract | Publisher Full Text

[2] 2. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15: 399. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Bailey JV, Pavlou M, Copas A, et al.: The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res. 2013; 15(12): e278. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Peckham E, Arundel C, Bailey D, et al.: Smoking Cessation Intervention for Severe Mental Ill Health Trial (SCIMITAR+): study protocol for a randomised controlled trial. Trials. 2017; 18(1): 44. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. SWAT 48 – Effects of a £ 10 note on retention. Reference Source

[6] 6. StataCorp: Stata Statistical Software: Release 13. College Station, TX: StataCorp LP. 2013.

[7] 7. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC. 2017.

[8] 8. Arundel C, Coleman E, Fairhurst C, et al.: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set. figshare. Dataset. 2019. http://www.doi.org/10.6084/m9.figshare.10060202.v2

The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT)

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Design

Participants

Intervention

Outcomes

Sample size

Randomisation

Blinding

Approvals

Statistical analysis

Results

Figure 1. Study flow diagram.

Discussion

Strengths and limitations

Conclusion

Data availability

Underlying data

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated