Background
Methods
Delphi Survey
Panel size and membership
-
chief investigators
-
members of the UK Clinical Research Network
-
clinical trials unit (CTU) directors
-
representatives of the main UK clinical trial funding bodies
-
operations managers and directors
-
clinical trial quality assurance managers
-
research associates, fellows and academics
-
research delivery managers
-
trial managers and coordinators
-
sponsors
-
statisticians
-
trial steering committee members
Recruitment of the panel
Distributing the Delphi survey
Conducting the Delphi survey
Consensus criteria
Consensus classification | Description | Definition |
---|---|---|
Consensus in | Consensus that the metric should be included in the performance metric set | ≥70% participants scoring 7 to 9 and < 15% participants scoring 1 to 3 |
Consensus out | Consensus that the metric should not be included in the performance metric set | ≥70% participants scoring 1 to 3 and < 15% of participants scoring 7 to 9 |
No consensus | Uncertainty about the importance of the metric | Anything else |
Consensus meeting
Results
Delphi Survey
Role | All Registered | Round 1 | Round 2 | Round 3 | |||||
---|---|---|---|---|---|---|---|---|---|
n
| % of total |
n
| % of total |
n
| % of total |
n
| % of total | % retained from round 1 | |
Senior trial manager, project lead or manager | 56 | 19.0 | 52 | 18.8 | 49 | 19.5 | 40 | 19.0 | 77 |
Trial or research manager | 52 | 17.7 | 51 | 18.4 | 46 | 18.3 | 40 | 19.0 | 78 |
Trial coordinator | 48 | 16.3 | 44 | 15.9 | 37 | 14.7 | 26 | 12.3 | 59 |
Chief investigator | 34 | 11.6 | 32 | 11.6 | 29 | 11.6 | 27 | 12.8 | 84 |
Academic or research associate or fellow | 18 | 6.1 | 18 | 6.5 | 17 | 6.8 | 16 | 7.6 | 89 |
Statistician | 18 | 6.1 | 17 | 6.1 | 16 | 6.4 | 14 | 6.6 | 82 |
Operations manager or director | 14 | 4.8 | 13 | 4.7 | 13 | 5.2 | 10 | 4.7 | 77 |
UK Clinical Research Network member | 12 | 4.1 | 12 | 4.3 | 11 | 4.4 | 9 | 4.3 | 75 |
Clinical Trials Unit director | 8 | 2.7 | 7 | 2.5 | 6 | 2.4 | 5 | 2.4 | 71 |
Quality assurance manager | 8 | 2.7 | 8 | 2.9 | 8 | 3.2 | 8 | 3.8 | 100 |
Research delivery manager | 4 | 1.4 | 3 | 1.1 | 2 | 0.8 | 2 | 0.9 | 67 |
Funder | 2 | 0.7 | 2 | 0.7 | 2 | 0.8 | 2 | 0.9 | 100 |
Other | 20 | 6.8 | 18 | 6.5 | 15 | 6.0 | 12 | 5.7 | 67 |
Total | 294 | 100 | 277 | 100 | 251 | 100 | 211 | 100 | 76 |
Metric and domain | Round 1 scores | Round 2 scores | Round 3 scores | % of consensus meeting participants voting to retain item | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1–3 | 4–6 | 7–9 | 10 | 1–3 | 4–6 | 7–9 | 10 | 1–3 | 4–6 | 7–9 | 10 | |||
n (%) respondents per rating category (1-3 ‘not important’, 4-6 ‘important but not critical’, 7-9 ‘critical’, 10 unable to score) | ||||||||||||||
Recruitment and retention | 1. Total actual recruitment versus total target recruitment a,c | 1 (0.5) | 36 (17.1) | 172 (81.5) | 2 (0.9) | 0 | 16 (7.6) | 194 (91.9) | 1 (0.5) | 0 | 14 (6.6) | 197 (93.4) | 0 | 100 |
2. Time from the site opening to first participant randomised | 8 (3.8) | 108 (51.2) | 94 (44.5) | 1 (0.5) | 3 (1.4) | 118 (55.9) | 90 (42.7) | 0 | 5 (2.4) | 126 (59.7) | 80 (37.9) | 0 | ||
3. Number of days/weeks since the most recent participant was randomised | 21 (10.0) | 112 (53.1) | 77 (36.5) | 1 (0.5) | 12 (5.7) | 142 (67.3) | 57 (27.0) | 0 | 11 (5.2) | 155 (73.5) | 45 (21.3) | 0 | ||
4. Percentage of potential participants screened who have been randomised | 4 (1.9) | 88 (41.7) | 117 (55.5) | 2 (0.9) | 2 (0.9) | 83 (39.3) | 125 (59.2) | 1 (0.5) | 0 | 76 (36) | 134 (63.5) | 1 (0.5) | ||
5. Percentage of potential participants who could have been screened, who were screened | – | – | – | – | 13 (6.2) | 97 (46.0) | 92 (43.6) | 9 (4.3) | 10 (4.7) | 90 (42.7) | 103 (48.8) | 8 (3.8) | ||
6. Percentage of potential participants screened who were eligible | – | – | – | – | 9 (4.3) | 106 (50.2) | 93 (44.1) | 3 (1.4) | 6 (2.8) | 110 (52.1) | 92 (43.6) | 3 (1.4) | ||
7. Percentage of potential participants eligible who have consented b,c | – | – | – | – | 8 (3.8) | 81 (38.4) | 119 (56.4) | 3 (1.4) | 3 (1.4) | 77 (36.5) | 128 (60.7) | 3 (1.4) | 95 | |
8. Percentage of potential participants who have consented and have been randomiseda | – | – | – | – | 5 (2.4) | 71 (33.6) | 131 (62.1) | 4 (1.9) | 2 (0.9) | 57 (27) | 150 (71.1) | 2 (0.9) | 35 | |
9. Percentage of randomised participants who have withdrawn consent to continue in the studya,c | 8 (3.8) | 76 (36.0) | 125 (59.2) | 2 (0.9) | 4 (1.9) | 60 (28.4) | 147 (69.7) | 0 | 4 (1.9) | 46 (21.8) | 161 (76.3) | 0 | 83 | |
10. Percentage of randomised participants lost to follow-upa | 10 (4.7) | 59 (28.0) | 140 (66.4) | 2 (0.9) | 3 (1.4) | 38 (18) | 169 (80.1) | 1 (0.5) | 3 (1.4) | 24 (11.4) | 183 (86.7) | 1 (0.5) | 22 | |
11. Percentage of screening logs returned on time out of all those that should have been returned | 40 (19.0) | 135 (64.0) | 33 (15.6) | 3 (1.4) | 29 (13.7) | 159 (75.4) | 22 (10.4) | 1 (0.5) | 23 (10.9) | 167 (79.1) | 20 (9.5) | 1 (0.5) | ||
12. Percentage of screening items completed of those required | 32 (15.2) | 105 (49.8) | 67 (31.8) | 7 (3.3) | 20 (9.5) | 114 (54) | 72 (34.1) | 5 (2.4) | 18 (8.5) | 117 (55.5) | 72 (34.1) | 4 (1.9) | ||
13. Percentage of randomised participants with a consent form that is incomplete or inaccuratea | 11 (5.2) | 51 (24.2) | 148 (70.1) | 1 (0.5) | 8 (3.8) | 31 (14.7) | 172 (81.5) | 0 | 9 (4.3) | 14 (6.6) | 187 (88.6) | 1 (0.5) | 13 | |
14. Percentage of all expected forms that have been receiveda | 8 (3.8) | 69 (32.7) | 128 (60.7) | 6 (2.8) | 4 (1.9) | 50 (23.7) | 154 (73) | 3 (1.4) | 39 | |||||
15. Percentage of randomised participants with any issues or problems with consenta | 10 (4.7) | 68 (32.2) | 129 (61.1) | 4 (1.9) | 6 (2.8) | 53 (25.1) | 150 (71.1) | 2 (0.9) | 4 (1.9) | 34 (16.1) | 169 (80.1) | 4 (1.9) | 26 | |
16. Percentage of randomised participants for whom documentation of consent is missing from their medical recordsa | 15 (7.1) | 69 (32.7) | 123 (58.3) | 4 (1.9) | 9 (4.3) | 47 (22.3) | 154 (73) | 1 (0.5) | 7 (3.3) | 31 (14.7) | 172 (81.5) | 1 (0.5) | 0 | |
Data quality | 17. Percentage of randomised participants with the time between data collection and either data entry (electronic case report form) or central receipt of paper case report form within the target timeframe | 12 (5.7) | 129 (61.1) | 66 (31.3) | 4 (1.9) | 8 (3.8) | 156 (73.9) | 45 (21.3) | 2 (0.9) | 7 (3.3) | 170 (80.6) | 32 (15.2) | 2 (0.9) | |
18. Percentage of randomised participants with a query/queries for primary outcome data a,c | 4 (1.9) | 59 (28.0) | 145 (68.7) | 3 (1.4) | 3 (1.4) | 36 (17.1) | 170 (80.6) | 2 (0.9) | 4 (1.9) | 23 (10.9) | 182 (86.3) | 2 (0.9) | 65 | |
19. Percentage of randomised participants with query/queries for secondary outcome data | 16 (7.6) | 128 (60.7) | 65 (30.8) | 2 (0.9) | 8 (3.8) | 156 (73.9) | 46 (21.8) | 1 (0.5) | 8 (3.8) | 162 (76.8) | 40 (19) | 1 (0.5) | ||
20. Time taken between sending a data query and resolution of the query | 17 (8.1) | 140 (66.4) | 52 (24.6) | 2 (0.9) | 10 (4.7) | 164 (77.7) | 36 (17.1) | 1 (0.5) | 9 (4.3) | 167 (79.1) | 34 (16.1) | 1 (0.5) | ||
21. Percentage of randomised participants with complete data for primary and important secondary outcomesa,c | 2(0.9) | 44 (20.9) | 163 (77.3) | 2 (0.9) | 1 (0.5) | 20 (9.5%) | 189 (89.6) | 1 (0.5) | 1 (0.5) | 11 (5.2) | 198 (93.8) | 1 (0.5) | 96 | |
22. Percentage of randomised participants with complete data | 3 (1.4) | 85 (40.3) | 120 (56.9) | 3 (1.4) | 0 | 88 (41.7) | 122 (57.8) | 1 (0.5) | 0 | 91 (43.1) | 119 (56.4) | 1 (0.5) | ||
23. Percentage of unresolved serious adverse event queries > 30 calendar days from the date the query was generated a | 3 (1.4) | 44 (20.9) | 163 (77.3) | 1 (0.5) | 1 (0.5) | 24 (11.4) | 186 (88.2) | 0 | 1 (0.5) | 12 (5.7) | 198 (93.8) | 0 | 9 | |
24. Total number of adverse events and serious adverse events reported per number of randomised participants | 32 (15.2) | 101 (47.9) | 74 (35.1) | 4 (1.9) | 19 (9) | 130 (61.6) | 59 (28.0) | 3 (1.4) | 17 (8.1) | 138 (65.4) | 53 (25.1) | 3 (1.4) | ||
25. Number of serious adverse events reported per number of randomised participants | 25 (11.8) | 84 (39.8) | 98 (46.4) | 4 (1.9) | 16 (7.6) | 90 (42.7) | 102 (48.3) | 3 (1.4) | 15 (7.1) | 102 (48.3) | 91 (43.1) | 3 (1.4) | ||
26. Number of adverse events reported per number of randomised participants b,c | 40 (19) | 106 (50.2) | 60 (28.4) | 5 (2.4) | 27 (12.8) | 136 (64.5) | 45 (21.3) | 3 (1.4) | 24 (11.4) | 148 (70.1) | 36 (17.1) | 3 (1.4) | 81 | |
Protocol compliance | 27. Percentage of randomised participants with at least one protocol violation a,c | 6 (2.8) | 78 (37) | 124 (58.8) | 3 (1.4) | 1 (0.5) | 64 (30.3) | 145 (68.7) | 1 (0.5) | 0 | 47 (22.3) | 163 (77.3) | 1 (0.5) | 76 |
28. Percentage of randomised participants receiving allocated intervention as intended per protocol a,c | 2 (0.9) | 48 (22.7) | 158 (74.9) | 3 (1.4) | 0 | 19 (9.0) | 191 (90.5) | 1 (0.5) | 0 | 11 (5.2) | 199 (94.3) | 1 (0.5) | 100 | |
29. Number of missed visits per number of randomised participants a | 7 (3.3) | 93 (44.1) | 107 (50.7) | 4 (1.9) | 5 (2.4) | 75 (35.5) | 128 (60.7) | 3 (1.4) | 4 (1.9) | 52 (24.6) | 152 (72) | 3 (1.4) | 10 | |
30. Number of late visits per number of randomised participants | 18 (8.5) | 128 (60.7) | 61 (28.9) | 4 (1.9) | 10 (4.7) | 157 (74.4) | 41 (19.4) | 3 (1.4) | 9 (4.3) | 162 (76.8) | 37 (17.5) | 3 (1.4) | ||
31. Number of critical or major audit findings per number of randomised participants a | 6 (2.8) | 43 (20.4) | 152 (72) | 10 (4.7) | 4 (1.9) | 23 (10.9) | 179 (84.8) | 5 (2.4) | 3 (1.4) | 14 (6.6) | 190 (90) | 4 (1.9) | 0 | |
Staff | 32. Number of contacts from site staff to the central trial team within a given time period | 79 (37.4) | 112 (53.1) | 14 (6.6) | 6 (2.8) | 76 (36) | 124 (58.8) | 6 (2.8) | 5 (2.4%) | 81 (38.4) | 120 (56.9) | 6 (2.8) | 4 (1.9) | |
33. Time between protocol amendment being sent and principal investigator sign-off | 22 (10.4) | 110 (52.1) | 73 (34.6) | 6 (2.8) | 16 (7.6) | 127 (60.2) | 65 (30.8) | 3 (1.4) | 15 (7.1) | 140 (66.4) | 53 (25.1) | 3 (1.4) | ||
34. Cumulative number of staff included on the delegation of duties log | – | – | – | – | 105 (49.8) | 87 (41.2) | 12 (5.7) | 7 (3.3) | 116 (55.0) | 83 (39.3) | 7 (3.3) | 5 (2.4) |
Round 1
Round 2
Round 3
Consensus meeting
Domain | Metric: original wording | Definition: original wording | Metric: amended wording | Definition: amended wording |
---|---|---|---|---|
Recruitment and retention | 1. Total actual recruitment versus total target recruitment (%) | The actual number of participants recruited into the trial by the site, versus the target number that was contractually agreed with the site prior to the trial commencing | Current actual recruitment versus target recruitment (%) | The actual number of participants recruited into the trial by the site, at the time of monitoring, versus the target number that was contractually agreed with the site prior to the trial commencing |
7. Percentage of potential participants eligible who have consented | The percentage of potential participants who were eligible to participate in the trial and who consented to participate | Percentage of eligible individuals who have consented | The percentage of individuals who were eligible to participate in the trial and who consented to participate | |
9. Percentage of randomised participants who have withdrawn consent to continue | The percentage of randomised participants who have withdrawn their consent to any further participation in the trial at the site. Collection of any further follow up data is therefore not attempted | |||
Data quality | 18. Percentage of randomised participants with a query for primary outcome data | The percentage of randomised participants at the site for whom the central trial team has sent one or more queries relating to the primary outcome data back to the site staff | ||
21. Percentage of randomised participants with complete data for primary and important secondary outcomes | The percentage of randomised participants at the site with outcome data complete for both the primary outcome and all the agreed important secondary outcomes | Percentage of expected participants with complete data for primary and important secondary outcomes | ||
26. Number of adverse events reported per number of randomised participants | Number of Adverse Events reported per number of randomised participants at the site | Percentage of randomised participants with at least one adverse event reported | The percentage of randomised participants at the site who have reported at least one adverse event | |
Protocol compliance | 27. Percentage of randomised participants with at least one protocol violation | The percentage of randomised participants at the site with any protocol violation/s, as defined by the protocol | ||
28. Percentage of randomised participants receiving allocated intervention as intended per protocol | The percentage of randomised participants at the site who started the allocated intervention, as specified in the protocol | Percentage of randomised participants who started allocated intervention |
Reporting tool
Discussion
Limitations
Conclusions
Acknowledgements
-
Stella Tarr and Chris Rumsey (both Nottingham Clinical Trials Unit) for designing and developing the reporting tool
-
The UK Trial Managers’ Network for dissemination of the Delphi survey to their membership
-
The NIHR Clinical Research Network (Louise Lambert and Kathryn Fairbrother) for sharing their experience and insight on national portfolio management and performance indicators
-
Professor Julia Brown (director of the Leeds Institute of Clinical Trials Research and the director of the UK Clinical Research Collaboration’s CTU Network) and Saeeda Bashir for distributing the Delphi survey on behalf of the UK CTU Network
-
Professor Chris Rogers (Medical Research Council Hubs for Trials Methodology Research) for assistance with identifying consensus meeting participants
-
The focus group and Delphi survey participants