Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial

Alex S. Mitchell; Liz Cook; Alexandra Dean; Caroline Fairhurst; Matthew Northgraves; David J. Torgerson; Mike Reed

doi:10.12688/f1000research.23018.2

Home Browse Using pens as an incentive for questionnaire return in an orthopaedic...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial

[version 2; peer review: 1 approved, 1 approved with reservations]

Alex S. Mitchell ¹, Liz Cook¹, Alexandra Dean¹, [...] Caroline Fairhurst¹, Matthew Northgraves², David J. Torgerson¹, Mike Reed³

Alex S. Mitchell ¹, Liz Cook¹, [...] Alexandra Dean¹, Caroline Fairhurst¹, Matthew Northgraves², David J. Torgerson¹, Mike Reed³

PUBLISHED 07 Sep 2021

Author details Author details

¹ York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK
² Hull Health Trials Unit, University of Hull, Hull, HU6 7RU, UK
³ Northumbria Healthcare NHS Foundation Trust and University of York, North Tyneside General Hospital, Rake Lane, North Shields, Tyne and Wear, NE29 8NH, UK

Alex S. Mitchell
Roles: Data Curation, Formal Analysis, Software, Writing – Original Draft Preparation, Writing – Review & Editing

Liz Cook
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Alexandra Dean
Roles: Investigation, Project Administration, Resources, Writing – Review & Editing

Caroline Fairhurst
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – Review & Editing

Matthew Northgraves
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

David J. Torgerson
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Supervision, Writing – Review & Editing

Mike Reed
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Studies Within A Trial (SWAT) collection.

Abstract

Background: We did a ‘study within a trial’ (SWAT), evaluating the effectiveness of the inclusion of a pen with a postal questionnaire, compared to no pen being included, on the retention rate in a large orthopaedic trial.
Methods: The SWAT was embedded in the KReBS trial. The primary outcome was the proportion of 12-month questionnaires returned. Secondary outcomes were the proportion of questionnaires completed and time to questionnaire return. Binary data were analysed using logistic regression and time to return using Cox proportional hazards regression. Odds ratios (OR) and hazard ratios (HR) are presented, with associated 95% confidence intervals and p-values.
Results: In total, 2305 participants were randomised into the SWAT. In the pen group, 1020/1145 (89.1%) of participants returned a questionnaire, compared to 982/1147 (85.6%) in the no pen group. The absolute difference in questionnaire return rate was 3.5% (95% CI: 0.8% to 6.2%; p=0.01). There were statistically significant differences in questionnaire return rate (OR 1.36; 95% CI: 1.06 to 1.74; p=0.02), questionnaire completion rate (OR 1.40; 95% CI: 1.11 to 1.78; p<0.01) and time to questionnaire return (HR 1.17; 95% CI: 1.07 to 1.27; p<0.01) favouring the pen group.
Conclusion: This SWAT adds to the growing evidence base for whether pens are effective as an incentive for retention, and indicates their potential effectiveness.

Registration: KReBS trial registered on 20 February 2019, ID ISRCTN87127065; SWAT registered on 1 April 2019, ID SWAT92.

Keywords

SWAT, Study Within A Trial, attrition, follow-up

Corresponding author: Alex S. Mitchell

Competing interests: No competing interests were disclosed.

Grant information: The KReBS trial was funded by 3M, while the SWAT was supported by the PROMETHEUS programme (MRC MR/R013748/1).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2021 Mitchell AS et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Mitchell AS, Cook L, Dean A et al. Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial [version 2; peer review: 1 approved, 1 approved with reservations]. F1000Research 2021, 9:321 (https://doi.org/10.12688/f1000research.23018.2) First published: 04 May 2020, 9:321 (https://doi.org/10.12688/f1000research.23018.1) Latest published: 07 Sep 2021, 9:321 (https://doi.org/10.12688/f1000research.23018.2)

Revised Amendments from Version 1

We have provided more information in the introduction justifying the need for this study. We have provided detail on the pre-planned retention strategies used, and also given more detail on how participants were randomised into the study. We have also made minor amendments to the results in light of a duplicate randomisation that was found in the host trial, meaning that 2334 rather than 2335 participants were randomised to the host trial. This duplicate randomisation was provided with an allocation for the SWAT, and therefore the results of the SWAT have changed, however, these changes are negligible and do not affect the interpretation of the results. We have added a new version of the flowchart to take these changes into account.

See the authors' detailed response to the review by Frances Shiely

Introduction

Recruitment and retention of participants have been identified as serious issues for randomised controlled trials (RCTs)^1,2.

Incentives, both monetary and non-monetary, are used by UK clinical trials units as part of recruitment and retention strategies³. In particular, the use of pens has recently been evaluated as a non-monetary incentive for recruitment⁴, while in 2016, Bell et al. assessed whether pens were effective in improving retention⁵. The study by Bell and colleagues indicated that the inclusion of a pen with postal follow-up questionnaires may increase return rates; however, the authors stated that there was a need to test the intervention in more diverse patient populations, and in addition the results were of borderline statistical significance

In response to this uncertainty, we did a ‘study within a trial’ (SWAT) evaluating the effectiveness of the inclusion of a pen with a postal questionnaire, compared to no pen being included, on the retention rate in a large orthopaedic trial.

Methods

Design

This paper details the methods and results of a SWAT embedded within the prospectively registered KReBS RCT (ISRCTN87127065). KReBS evaluated the effectiveness of a two-layer compression bandage compared with a standard wool and crepe bandage applied post-operatively on patient-reported outcomes in total knee replacement patients⁶.

Participants

The SWAT was conducted in 26 NHS hospital trust sites and was implemented after the start of KReBS follow-up. All KReBS participants were eligible for this SWAT provided they were not deceased or withdrawn from follow-up before being due to be sent their 12-month follow-up postal questionnaire.

Intervention

Participants in the SWAT intervention group received a pen (branded with the York Trials Unit and University of York logos) with their 12-month questionnaire.

All SWAT participants received pre-planned retention strategies within KReBS. This consisted of a reminder letter and additional copy of the questionnaire if the participant had not returned a completed copy 4 weeks after sending out the original copy. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient-reported outcomes.

Outcomes

The primary outcome was the proportion of participants who returned a 12-month questionnaire. Secondary outcomes were proportion of participants who completed the questionnaire and time to questionnaire return. A questionnaire was considered complete if the participant had answered 11 or more questions of the 12-item host trial primary outcome, the Oxford Knee Score⁷.

Sample size

Since this was an embedded trial, the sample size was determined by the number of participants in the main KReBS trial⁶, which aimed to recruit 2600 participants.

Randomisation

Participants were randomised into the SWAT in two batches, using a 1:1 allocation ratio, in a single large block the size of the batch. The SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early there was a risk of participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on. The allocation schedule for each batch was generated by a statistician at York Trials Unit using Stata v15⁸.

Blinding

Participants were not informed of their explicit participation in the SWAT, but due to the nature of the intervention could not be blinded to receipt (or not) of a pen with their questionnaire. Similarly, it was not possible to blind research staff to SWAT allocation.

Approvals

The SWAT was incorporated into the host trial protocol and approved as part of Substantial Amendment 2 by the Research Ethics Committee North East – Newcastle and North Tyneside on 13/04/2018. As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive the additional incentive, informed consent was not obtained for participation in this SWAT.

Statistical analysis

Analyses were carried out using Stata v16.0⁹. A diagram detailing the flow of participants through the SWAT is provided, and baseline characteristics are presented by SWAT allocation. Outcomes are summarised descriptively. Statistical tests were two-sided, used a 5% significance level, and were done on an intention to treat basis. All analyses (except the calculation of the absolute difference in return rate) used mixed effects, adjusting for SWAT allocation and host trial allocation as fixed effects and trial site as a random effect. The absolute difference in return rates was estimated using the two-sample test of proportions. Relevant parameter estimates are presented with associated 95% confidence intervals (CI) and p-values.

The proportion of participants who returned a 12-month questionnaire was analysed using logistic regression. Questionnaire completion was analysed in the same manner.

Time to questionnaire return was analysed using a Cox proportional hazards shared frailty model. Participants who did not return a questionnaire were censored at 90 days.

Results

In total, 2334 participants were recruited into the KReBS trial and 2305 were randomised into the SWAT (Figure 1). The average age was 69.0 years and 55.2% were female (Table 1). A further 13 participants died or withdrew following randomisation and as a result, 1145 participants in the pen group, and 1147 in the no pen group, were sent a 12-month questionnaire and were included in the analysis.

Figure 1. Study flow diagram.

Table 1. Baseline characteristics of the SWAT participants.

Variable	Pen (n=1153)	No pen (n=1152)	Total (n=2305)
Gender, n (%) Male Female Missing	510 (44.2) 642 (55.7) 1 (0.1)	519 (45.1) 630 (54.7) 3 (0.3)	1029 (44.6) 1272 (55.2) 4 (0.2)
Age n (%) Mean (SD) Median (IQR)	1152 (99.9) 68.8 (8.8) 69.1 (62.8, 74.9)	1149 (99.7) 69.2 (9.0) 69.6 (62.7, 75.5)	2301 (99.8) 69.0 (8.9) 69.3 (62.7, 75.3)
Oxford Knee Score n (%) Mean (SD) Median (IQR)	917 (79.5) 20.2 (7.8) 20 (15, 25)	953 (82.7) 20.2 (7.9) 20 (14, 26)	1870 (81.1) 20.2 (7.8) 20 (14, 25)

In the pen group, 1020 (89.1%) of participants returned a questionnaire, compared to 982 (85.6%) in the no pen group (Table 2). The absolute difference in return rate was 3.5% (95% CI: 0.8% to 6.2%; p=0.01). There was a statistically significant difference between the groups in the likelihood of returning a questionnaire (OR 1.36; 95% CI: 1.06 to 1.74; p=0.02), and also in the likelihood of returning a complete questionnaire (OR 1.40; 95% CI: 1.11 to 1.78; p<0.01). In addition, there was evidence of a reduction in time to return in favour of the pen group (HR 1.17; 95% CI: 1.07 to 1.27; p<0.01). See Underlying data for full, individual-level data¹⁰.

Table 2. Descriptive summaries of primary and secondary outcomes.

Variable	Pen (n=1145)	No pen (n=1147)	Total (n=2292)
Returned questionnaire, n (%) Yes No	1020 (89.1) 125 (10.9)	982 (85.6) 165 (14.4)	2002 (87.3) 290 (12.7)
Completed questionnaire, n (%) Yes No	1005 (87.8) 140 (12.2)	958 (83.5) 189 (16.5)	1963 (85.6) 330 (14.4)
Time to return, days n (%) Mean (SD) Median (IQR)	1020 (100) 16.0 (18.0) 10 (7, 15)	982 (100) 17.6 (19.7) 11 (8, 19)	2002 (100) 16.8 (18.8) 10 (8, 17)

Discussion

There is strong evidence that the use of a pen as a non-monetary incentive in the KReBS trial increased the proportion of questionnaires returned and completed, and also decreased the time to return. The decrease in time to return provides support to the findings of Bell et al.⁵. However, completion rate was calculated as a proportion of all SWAT participants rather than all SWAT participants who returned a questionnaire, and as a result questionnaire completion was highly correlated with questionnaire return. On the other hand, the large sample size of this SWAT means the results can be generalised to other orthopaedic trials.

Conclusion

This SWAT adds to the growing evidence base for whether pens are effective as an incentive for retention and indicates their potential effectiveness.

Data availability

Underlying data

Open Science Framework: Underlying data and CONSORT checklist for Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial. https://doi.org/10.17605/OSF.IO/BEPN3¹⁰.

This project contains the underlying data in CSV, SAS and V8XPT formats, alongside a data key.

Reporting guidelines

Open Science Framework: CONSORT checklist for ‘Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial’. https://doi.org/10.17605/OSF.IO/BEPN3¹⁰.

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Faculty Opinions recommended

References

1. Hewitt CE, Kumaravel B, Dumville JC, et al.: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010; 63(11): 1264–1270. PubMed Abstract | Publisher Full Text
2. Tudur Smith C, Hickey H, Clarke M, et al.: The trials methodological research agenda: results from a priority setting exercise. Trials. 2014; 15: 32. PubMed Abstract | Publisher Full Text | Free Full Text
3. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15: 399. PubMed Abstract | Publisher Full Text | Free Full Text
4. Whiteside K, Flett L, Mitchell A, et al.: Using pens as an incentive for trial recruitment of older adults: An embedded randomised controlled trial [version 1; peer review: 2 approved]. F1000Res. 2019; 8: 315. PubMed Abstract | Publisher Full Text | Free Full Text
5. Bell K, Clark L, Fairhurst C, et al.: Enclosing a pen reduced time to response to questionnaire mailings. J Clin Epidemiol. 2016; 74: 144–150. PubMed Abstract | Publisher Full Text
6. Cook L, Northgraves MJ, Fairhurst C, et al.: Knee Replacement Bandaging Study (KReBS) evaluating the effect of a two-layer compression bandage system on knee function following total knee arthroplasty: study protocol for a randomised controlled trial. Trials. 2019; 20(1): 261. PubMed Abstract | Publisher Full Text | Free Full Text
7. Dawson J, Fitzpatrick R, Carr A, et al.: Questionnaire on the perceptions of patients about total hip replacement. J Bone Joint Surg Br. 1996; 78(2): 185–190. PubMed Abstract | Publisher Full Text
8. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC, 2015.
9. StataCorp: Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC, 2019. Reference Source
10. Mitchell A, Cook L, Dean A, et al.: Underlying data and CONSORT checklist for Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial. 2020. http://www.doi.org/10.17605/OSF.IO/BEPN3

Comments on this article Comments (1)

Version 2

VERSION 2 PUBLISHED 07 Sep 2021

Revised

Comment

Version 1

VERSION 1 PUBLISHED 04 May 2020

Discussion is closed on this version, please comment on the latest version above.

Reviewer Response 27 Aug 2020

Sarah Cotterill, Centre for Biostatistics, School of Health Sciences, University of Manchester, Manchester, UK

27 Aug 2020

Reviewer Response

This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding ... Continue reading This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding that the addition of a pen is an effective way of increasing response to questionnaire sent to participants in a clinical trial.

Is it appropriate to describe a pen that is sent with a questionnaire as an ‘incentive’? According to the Behaviour Change Technique Taxonomy, an incentive is a where someone is promised in advance that ‘money/vouchers or other valued objects will be delivered if and only if there has been effort and/or progress in performing the behaviour’¹. In this experiment, the pen was provided to all participants, regardless of whether or not they performed the behaviour (return of the questionnaire), and perhaps that would be better described, not as an incentive, but rather as something else, such as ‘adding objects to the environment’, which is defined as ‘Add objects to the environment in order to facilitate performance of the behaviour.’ (Examples from the BCTT: Provide free condoms to facilitate safe sex; Provide attractive toothbrush to improved tooth brushing technique).¹

¹Michie, S., M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M. P. Eccles, J. Cane and C. E. Wood (2013). "The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions." Ann Behav Med 46(1): 81-95, supplementary materials
This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding that the addition of a pen is an effective way of increasing response to questionnaire sent to participants in a clinical trial.

Is it appropriate to describe a pen that is sent with a questionnaire as an ‘incentive’? According to the Behaviour Change Technique Taxonomy, an incentive is a where someone is promised in advance that ‘money/vouchers or other valued objects will be delivered if and only if there has been effort and/or progress in performing the behaviour’¹. In this experiment, the pen was provided to all participants, regardless of whether or not they performed the behaviour (return of the questionnaire), and perhaps that would be better described, not as an incentive, but rather as something else, such as ‘adding objects to the environment’, which is defined as ‘Add objects to the environment in order to facilitate performance of the behaviour.’ (Examples from the BCTT: Provide free condoms to facilitate safe sex; Provide attractive toothbrush to improved tooth brushing technique).¹

¹Michie, S., M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M. P. Eccles, J. Cane and C. E. Wood (2013). "The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions." Ann Behav Med 46(1): 81-95, supplementary materials
Competing Interests: I have previously co-authored a paper with one of this paper's authors Close
Report a concern
Discussion is closed on this version, please comment on the latest version above.

Author details Author details

Alex S. Mitchell
Roles: Data Curation, Formal Analysis, Software, Writing – Original Draft Preparation, Writing – Review & Editing

Liz Cook
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Alexandra Dean
Roles: Investigation, Project Administration, Resources, Writing – Review & Editing

Caroline Fairhurst
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – Review & Editing

Matthew Northgraves
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

David J. Torgerson
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Supervision, Writing – Review & Editing

Mike Reed
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The KReBS trial was funded by 3M, while the SWAT was supported by the PROMETHEUS programme (MRC MR/R013748/1).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 07 Sep 2021, 9:321

https://doi.org/10.12688/f1000research.23018.2

version 1

Published: 04 May 2020, 9:321

https://doi.org/10.12688/f1000research.23018.1

© 2021 Mitchell AS et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Mitchell AS, Cook L, Dean A et al. Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial [version 2; peer review: 1 approved, 1 approved with reservations] F1000Research 2021, 9:321 (https://doi.org/10.12688/f1000research.23018.2)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 2

VERSION 2

PUBLISHED 07 Sep 2021

Revised

Views

Reviewer Report 01 Dec 2021

Matthias Briel, Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Basel, Switzerland

Approved with Reservations

https://doi.org/10.5256/f1000research.77323.r100075

Summary of the article:
The article reports the results of a SWAT within the KReBS trial. The SWAT aimed to investigate the effectiveness of the inclusion of a pen with a postal questionnaire compared to no pen being included on the questionnaire return rate in a large orthopaedic trial. The results provide evidence for a higher questionnaire return rate, higher questionnaire completion rate, and shorter time until questionnaire return with the inclusion of a pen.

Abstract:
For the conclusion I would suggest being more precise by rephrasing it as follows:
"This SWAT adds to the growing evidence base for whether pens are effective as an incentive for returning a postal questionnaire and participant retention, and indicates their potential effectiveness."

Introduction:
Did the authors conduct a systematic search for similar SWATs in terms of research question? If yes, please provide search details in an appendix. If no, please do so or explain why you think this is not necessary. In the SWAT store there is SWAT 138, for instance, which could/should also be mentioned in the introduction.

Methods:
Outcomes: Please clarify whether the primary outcome includes also questionnaires that were completed over the phone, because there was no response from participants otherwise. In addition, I think it would be of interest for readers to know how many participants in each group (pen vs no pen) needed a reminder letter, and how many needed to be contacted by telephone to obtain patient-reported outcomes. This could be done in post-hoc analyses and would reveal something about how/at which level the intervention worked.

Results:
Please see my suggestion above for some additional analyses.

Discussion:
I think in terms of generalizability it should be clearly acknowledged that the results are generalizable to other Orthopaedic trials with respect to an increased return rate of a postal questionnaire, but not necessarily in generally increasing retention of participants in orthopaedic trials. If participants need to come to the clinic for a follow-up visit instead of filling a questionnaire, then sending a pen might not be useful.

Conclusion:
Based on what I said in the paragraph above, I suggest rephrasing the conclusion as for the abstract - just to make it clear that results cannot automatically extrapolated to any form of participant retention.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Clinical trial methodology, meta-research, research transparency.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 06 Oct 2021

Frances Shiely, HRB Clinical Research Facility and School of Public Health, University College Cork, Cork, Ireland

Approved

https://doi.org/10.5256/f1000research.77323.r93671

Thank you to the authors for their considered revision of this ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 04 May 2020

Views

Reviewer Report 07 May 2020

Frances Shiely, HRB Clinical Research Facility and School of Public Health, University College Cork, Cork, Ireland

Approved with Reservations

https://doi.org/10.5256/f1000research.25410.r63133

Article Summary
This was a SWAT within the KReBS trial. The purpose of the SWAT was to evaluate the effectiveness of the inclusion of a pen with a postal questionnaire, compared to no pen being included, on the retention rate in a large orthopaedic trial. The authors conclude that pens are potentially effective as an incentive for retention.

Abstract
The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Introduction Section
In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

Methods Section
Intervention
All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.
Outcomes
I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

Randomisation
What do you mean by two batches, and why were there two batches?

Conclusion
Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No source data required
Are the conclusions drawn adequately supported by the results?

No

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: clinical trial methodology

CITE

Report a concern

Author Response 28 May 2020

Alex Mitchell, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

28 May 2020

Author Response

We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will ... Continue reading We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will update the article in response to the points raised after we have received comments from additional reviewers.

The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Please see the response to your final point.

In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.

All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.

Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes.

I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.

We disagree that the completion rate should be removed as an outcome as this was pre-specified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?

This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.

Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Thank you for this comment, however we think we are justified in retaining use of the term ‘retention’ in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community (https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ ).
We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will update the article in response to the points raised after we have received comments from additional reviewers.

The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Please see the response to your final point.

In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.

All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.

Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes.

I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.

We disagree that the completion rate should be removed as an outcome as this was pre-specified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?

This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.

Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Thank you for this comment, however we think we are justified in retaining use of the term ‘retention’ in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community (https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ ).
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 28 May 2020

Alex Mitchell, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

28 May 2020

Author Response

We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will ... Continue reading We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will update the article in response to the points raised after we have received comments from additional reviewers.

The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Please see the response to your final point.

In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.

All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.

Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes.

I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.

We disagree that the completion rate should be removed as an outcome as this was pre-specified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?

This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.

Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Thank you for this comment, however we think we are justified in retaining use of the term ‘retention’ in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community (https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ ).
We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will update the article in response to the points raised after we have received comments from additional reviewers.

The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Please see the response to your final point.

In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.

All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.

Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes.

I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.

We disagree that the completion rate should be removed as an outcome as this was pre-specified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?

This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.

Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Thank you for this comment, however we think we are justified in retaining use of the term ‘retention’ in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community (https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ ).
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (1)

Version 2

VERSION 2 PUBLISHED 07 Sep 2021

Revised

Comment

Version 1

VERSION 1 PUBLISHED 04 May 2020

Discussion is closed on this version, please comment on the latest version above.

Reviewer Response 27 Aug 2020

Sarah Cotterill, Centre for Biostatistics, School of Health Sciences, University of Manchester, Manchester, UK

27 Aug 2020

Reviewer Response

This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding ... Continue reading This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding that the addition of a pen is an effective way of increasing response to questionnaire sent to participants in a clinical trial.

Is it appropriate to describe a pen that is sent with a questionnaire as an ‘incentive’? According to the Behaviour Change Technique Taxonomy, an incentive is a where someone is promised in advance that ‘money/vouchers or other valued objects will be delivered if and only if there has been effort and/or progress in performing the behaviour’¹. In this experiment, the pen was provided to all participants, regardless of whether or not they performed the behaviour (return of the questionnaire), and perhaps that would be better described, not as an incentive, but rather as something else, such as ‘adding objects to the environment’, which is defined as ‘Add objects to the environment in order to facilitate performance of the behaviour.’ (Examples from the BCTT: Provide free condoms to facilitate safe sex; Provide attractive toothbrush to improved tooth brushing technique).¹

¹Michie, S., M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M. P. Eccles, J. Cane and C. E. Wood (2013). "The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions." Ann Behav Med 46(1): 81-95, supplementary materials
This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding that the addition of a pen is an effective way of increasing response to questionnaire sent to participants in a clinical trial.

Is it appropriate to describe a pen that is sent with a questionnaire as an ‘incentive’? According to the Behaviour Change Technique Taxonomy, an incentive is a where someone is promised in advance that ‘money/vouchers or other valued objects will be delivered if and only if there has been effort and/or progress in performing the behaviour’¹. In this experiment, the pen was provided to all participants, regardless of whether or not they performed the behaviour (return of the questionnaire), and perhaps that would be better described, not as an incentive, but rather as something else, such as ‘adding objects to the environment’, which is defined as ‘Add objects to the environment in order to facilitate performance of the behaviour.’ (Examples from the BCTT: Provide free condoms to facilitate safe sex; Provide attractive toothbrush to improved tooth brushing technique).¹

¹Michie, S., M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M. P. Eccles, J. Cane and C. E. Wood (2013). "The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions." Ann Behav Med 46(1): 81-95, supplementary materials
Competing Interests: I have previously co-authored a paper with one of this paper's authors Close
Report a concern
Discussion is closed on this version, please comment on the latest version above.

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 07 Sep 21	read	read
Version 1 04 May 20	read

Frances Shiely, University College Cork, Cork, Ireland
Matthias Briel, University of Basel, Basel, Switzerland

Comments on this article

All Comments(1)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

9 Views

01 Dec 2021 | for Version 2

Matthias Briel, Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Basel, Switzerland

9 Views Cite this report Responses(0)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Clinical trial methodology, meta-research, research transparency.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

8 Views

06 Oct 2021 | for Version 2

Frances Shiely, HRB Clinical Research Facility and School of Public Health, University College Cork, Cork, Ireland

8 Views Cite this report Responses(0)

Approved

Thank you to the authors for their considered revision of this SWAT, which has given clarity on the methods which will allow replication.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

clinical trial methodology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

36 Views

07 May 2020 | for Version 1

Frances Shiely, HRB Clinical Research Facility and School of Public Health, University College Cork, Cork, Ireland

36 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No source data required
Are the conclusions drawn adequately supported by the results?

No

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

clinical trial methodology

Respond to this report

Responses (1)

Author Response

28 May 2020

Alex Mitchell, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

We would like to thank the reviewer for the helpful comments and the interest shown in our paper. We hope that our responses are satisfactory. Please note that we will update the article in response to the points raised after we have received comments from additional reviewers.

The background section needs to be amended as your SWAT did not evaluate retention in clinical trials. Likewise, the conclusion is not supported. Please see the comments on the conclusion section below.

Please see the response to your final point.

In the introduction section, you mention the Bell et al. study and state that the authors say the results are uncertain, and in response to this you conducted your study. However, say what the reason for the uncertainty is. It would be helpful to give support to the necessity for your research study to include the shortcomings of Bell et al. Please amend.

The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.

All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the “intervention” heading? It implies that you are speaking about the intervention group only. Perhaps you’d be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.

Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes.

I question the selection of the secondary outcome, the proportion of participants who completed the questionnaire. In reality, if someone begins, whether it’s the York CTU pen or their own, what does the supply of the York pen have to do with the completion of the questionnaire?
You then go on in your results to state that questionnaire completion was highly correlated with questionnaire return. How useful is this measure then? I would like to see the questionnaire completion rate removed as a focus of this SWAT. It additionally detracts from your main findings.

We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.

We disagree that the completion rate should be removed as an outcome as this was pre-specified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?

This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets (‘batches’) of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.

Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It’s a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Thank you for this comment, however we think we are justified in retaining use of the term ‘retention’ in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community (https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ ).

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Hewitt CE, Kumaravel B, Dumville JC, et al.: Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010; 63(11): 1264–1270. PubMed Abstract | Publisher Full Text

[2] 2. Tudur Smith C, Hickey H, Clarke M, et al.: The trials methodological research agenda: results from a priority setting exercise. Trials. 2014; 15: 32. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15: 399. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Whiteside K, Flett L, Mitchell A, et al.: Using pens as an incentive for trial recruitment of older adults: An embedded randomised controlled trial [version 1; peer review: 2 approved]. F1000Res. 2019; 8: 315. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Bell K, Clark L, Fairhurst C, et al.: Enclosing a pen reduced time to response to questionnaire mailings. J Clin Epidemiol. 2016; 74: 144–150. PubMed Abstract | Publisher Full Text

[6] 6. Cook L, Northgraves MJ, Fairhurst C, et al.: Knee Replacement Bandaging Study (KReBS) evaluating the effect of a two-layer compression bandage system on knee function following total knee arthroplasty: study protocol for a randomised controlled trial. Trials. 2019; 20(1): 261. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Dawson J, Fitzpatrick R, Carr A, et al.: Questionnaire on the perceptions of patients about total hip replacement. J Bone Joint Surg Br. 1996; 78(2): 185–190. PubMed Abstract | Publisher Full Text

[8] 8. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC, 2015.

[9] 9. StataCorp: Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC, 2019. Reference Source

[10] 10. Mitchell A, Cook L, Dean A, et al.: Underlying data and CONSORT checklist for Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial. 2020. http://www.doi.org/10.17605/OSF.IO/BEPN3

Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Design

Participants

Intervention

Outcomes

Sample size

Randomisation

Blinding

Approvals

Statistical analysis

Results

Figure 1. Study flow diagram.

Table 1. Baseline characteristics of the SWAT participants.

Table 2. Descriptive summaries of primary and secondary outcomes.

Discussion

Conclusion

Data availability

Underlying data

Reporting guidelines

References

Comments on this article Comments (1)

Open Peer Review

Comments on this article Comments (1)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated