What are the most important unanswered research questions in trial retention? A James Lind Alliance Priority Setting Partnership: the PRioRiTy II (Prioritising Retention in Randomised Trials) study

Convenience sampling was used to sample survey respondents. Steering Group members, including the patient partners, identified and engaged a wide range of appropriate potential stakeholders through their networks of contacts. The target population mirrored those groups represented by individuals on the Steering Group. Specifically, these were patients or members of the public involved in a trial, frontline staff or other staff involved in trial retention, investigators, and trial methodologists, all of whom needed to be based in the UK or Ireland and be over 18. We developed an eight-question online survey in SurveyMonkey (SurveyMonkey, Palo Alto, CA, USA) to gather uncertainties for our initial stage. This was open for 4 weeks. We also made a paper copy of the survey with pre-paid return envelopes available if required. We set no formal target sample size for the number of responses. The eight questions included six open-ended questions (Appendix 1) that explored the respondent’s views on unanswered questions for trial retention and general comments relating to retention in randomised trials that stakeholders would like to see answered. Based on the experience of the PRioRITy I project and discussion by the project Steering Group, these six open-ended questions were modelled on broad areas of trial retention: why participants stay involved; planning of data collection; processes of collecting data; information provided about data collection; aspects relating to trial staff involved in data collection; any other comments. It was felt that using six questions rather than one generic question may allow broader coverage of all aspects of trial retention. The questions also included an additional two demographic questions about the respondents to help monitor the geographic spread and roles of people responding to the survey. A pilot to test question comprehension and website usability was conducted with a small sample (n = 6) of volunteers from within the Health Services Research Unit (HSRU) but included non-academic staff members. We then distributed a weblink to the survey to the four stakeholder groups (described earlier) and also promoted the survey through social media channels and Twitter hashtags. The initial survey was launched in March 2018 and closed in May 2018 (8 weeks of data collection). We also asked respondents if they would consider attending the final consensus meeting. Electronic data was stored on password-protected university computers supported by secure servers. Paper copies of questionnaires were stored in locked tambour filing systems. The electronic and paper data was held in locked offices and only accessible by key personnel.

The initial survey was hosted by the JLA, who provided the Steering Group with regular updates and the compiled answers once completed. We used samples of responses as they were returned to us to identify key themes and questions. This allowed us to generate a representative series of thematic groups efficiently once the survey closed. The JLA collated the survey responses into a single Excel spreadsheet, and we coded the responses using a process of constant comparison analysis [14] into a thematic group where appropriate. We repeated this process of comparison until the range and number of thematic groups truly reflected the whole data set. The determination of thematic groups was an iterative process and generated through discussion amongst the Aberdeen team members, who also conducted the subsequent analysis (DB, HG, KG, and ST). Where an item did not fit into an existing thematic group, we either expanded a thematic group or created a new one. Each stakeholder response could contain numerous items, and multiple themes may have arisen. We therefore sub-divided responses into constituent parts during coding to allow their mapping across different thematic groups. We did not assign responses that were out of scope to a thematic code; rather, we categorised these responses separately for potential future use. All responses that did not refer to a process (such as recruitment) were assumed to be about retention as per the survey questions and hence were considered to be within scope. This process also involved regular group discussion and consultation to ensure consistency in approach and accuracy of the coding.

Once coded, we analysed the data to determine the initial sub-questions and broader main questions present within each theme as well as how often they occurred over the course of 4 weeks. To guide this process, we used word-for-word responses as a framework for developing the sub-questions, which grew iteratively as the data was analysed. We compiled the broader questions from each theme together and conducted a check to ensure that they remained representative of their respective sub-questions. To evaluate reliability, once we connected all coded data items to sub-questions, a 10% sample was selected at random from each team member’s analysis to compare findings. We held group discussions to identify discrepancies and resolve disagreements. We also conducted a check with the questions identified in the initial scoping survey against existing sources of evidence reporting trial retention research. This ensured that questions raised for the interim stage were unanswered by research. The evidence sources used for checking were:

Together with the Steering Group, we grouped and merged the longlist of broad questions where appropriate and removed duplicates to create a shortlist of questions in advance of the interim stage. Through consultation with the Steering Group, we discussed and sometimes revised the terminology to improve the clarity of the original meaning of the questions whilst ensuring the items remained true to the voices of respondents.

In the interim stage, we conducted a ‘back-categorisation’ on the initial stage shortlist in which we asked for feedback and comments from stakeholders who were not involved in the project. These individuals were identified through two processes: (1) email invitation to members of the HSRU (i.e. people who were familiar with trials); (2) invitation to friends and family (who were not familiar with trials) of the Aberdeen team. This process of back-categorisation involved presenting stakeholders with the shortlist of questions from the previous stage and conducting short individual interviews to assess their understanding of the questions. We also asked individuals involved in the back-categorisation process to provide examples of the types of research activities they would expect to see covered by each question. This process ensured that the language used was broad enough to ensure the correct coding of sub-questions under broader indicative questions. For example, questioning around the broad question ‘How could technology be best used in trial follow-up processes?’ included probing stakeholders on what their understanding of ‘technology’ is in relation to trials. Open questions such as ‘From this question, who might you assume would be using technology within trial follow-up processes?’ were used to gather responses to assess whether the question was unintentionally focussing on one specific group of trial stakeholders (e.g. patients). In this example, individuals explained that technology could be used by both the people involved in doing the trial (e.g. research nurses, clinicians) and the people taking part in the trial, so the language of the question was not changed.

The results of the back-categorisation were combined with the earlier responses from the Steering Group to create the list of indicative questions for the interim survey.

For the interim survey, we used SurveyMonkey to ask stakeholders to choose up to 10 of the questions that they believed were the most important. This survey was open for 6 weeks, and we made paper copies of the survey available if required. Invitations to this survey were open to anyone, and not restricted to the participants from the initial survey. As with the initial survey, no formal target sample size was set. However, the number of respondents within each reported group was checked weekly. This allowed us to target groups with lower representation during the ongoing dissemination of the survey.

We distributed the survey link through email, institution websites, blogs, newsletters, and social media. The HSRU at the University of Aberdeen also issued a press release and coordinated promotion alongside the JLA. The interim survey was launched in August 2018 and closed in September 2018 (6 weeks of data collection).

The online survey included a drop-down menu showing the indicative question list from which no more than 10 could be selected. This generated a total score for each question to represent the overall number of times the question was selected. We also used ranked weighted scores to decide which of the interim survey research questions would be taken forward to the final consensus meeting, using the following standard JLA approach as described in the JLA Guidebook [17] (www.​jla.​nihr.​ac.​uk/​jla-guidebook/​).

Each time a question was chosen, we assigned it one point. To ensure equal influence, points for each stakeholder group were tallied separately, generating separate total scores for each group for the questions. Within each of the four stakeholder groups, the scores for each question were arranged in order from highest to lowest. We then gave these a new score according to their position, from 27 for the most popular question down to 1 for the least popular. This resulted in the lowest ranked question receiving the lowest total score, through to the highest ranked question receiving the highest. The list was then ordered by score from highest to lowest and presented to the Steering Group. In cases when the questions had the same total, we ranked them in joint place. This gave the overall interim ranking to the research questions and the rankings for each of the stakeholder groups, whilst minimising bias owing to numbers of responses from each stakeholder group.

The most frequently reported role amongst initial respondents was a researcher involved in aspects of trials other than retention (22%). The proportion of respondents within each stakeholder group and the geographical spread are shown in Table 2.

Thematic grouping was used to sort and separate data from the 2431 answers from 456 responses into categories, resulting in 3256 individual coded items. Within this there were 372 specific questions about retention that served as templates to create a series of sub-questions which would later be grouped within and constitute the main questions, depending on how often they appeared within the data. This allowed each coded item to be represented with a question and led to the creation of a longlist of 105 total questions through combining overlapping main and sub-questions. Through review and discussion with the Steering Group, we merged questions from this list where there was substantial overlap to a shortlist of 33 questions to be carried over to the interim stage.

With further consultation and the ‘back-categorisation’ process, we reduced the initial question shortlist to an indicative list of 27 questions to be used in the interim survey. The survey received 886 responses overall. Of these, only 1 was on paper and 16 were not answered fully. The following results therefore report data from 870 responses. Of these, 100% made a selection of up to 10 of the 27 questions, and 864 respondents gave information describing their gender, with 650 (75%) selecting female and 205 (24%) selecting male, 1 using their own term and 8 (1%) preferring not to say.

The spread of stakeholder groups and their geographical spread can be seen in Table 3.

Following the standard JLA approach for analysis of the interim survey [17], a ranked weighted score across all stakeholder groups was used to select the top 21 questions to be taken to the consensus meeting. We made this decision as experience from the JLA suggests between 20 and 25 questions are optimal for discussion. We selected 21 rather than 20 because there was a tie between the rankings, where two questions received the same score.