Background
Methods
Survey
Scenarios | |
---|---|
A | A trial with a largea sample size, recruiting in several centres, each with multiple treatment providers |
B | A trial with a smallb sample size, recruiting in several centres, each with multiple treatment providers |
C | A trial that recruits within several centres, where treatment providers treat patients across recruiting centres; i.e. treatment provider is not unique to a centre |
D | A trial recruiting from several centres, each with multiple treatment providers, investigating a surgical intervention |
E | A trial recruiting from several centres, each with multiple treatment providers, investigating substantially different surgical interventions, e.g. a trial comparing surgery to an injection |
Analysis
Results
Unit participation and demographics
Managing effects through design
Clustering
“ … drug trials less effect due to centre compared to say complex or surgical interventions.” [ID23]
Question | Category | Response statistics | ||||
---|---|---|---|---|---|---|
n | N | n/N% | ||||
2 | Does your Unit have any multicentre trials that do not stratify randomisation by centre? | Yes | 25 | 44 | 57% | |
No | 18 | 44 | 41% | |||
No response | 1 | 44 | 2% | |||
3 | In each of the following scenarios, how was the randomisation stratified in trials that your Unit has run? Select all that apply. | |||||
A | Large sample size,a recruiting in several centres, each with multiple treatment providers | Experience in trial type | 39 | 44 | 89% | |
Centre | 34 | 39 | 87% | |||
Treatment provider | 3 | 39 | 8% | |||
Both | 10 | 39 | 26% | |||
Neither | 1 | 39 | 3% | |||
No experience in trial type | 4 | 44 | 9% | |||
No response | 1 | 44 | 2% | |||
B | Small sample size,b recruiting in several centres, each with multiple treatment providers | Experience in trial type | 31 | 44 | 70% | |
Centre | 24 | 31 | 77% | |||
Treatment provider | 2 | 31 | 6% | |||
Both | 2 | 31 | 6% | |||
Neither | 7 | 31 | 23% | |||
No experience in trial type | 12 | 44 | 27% | |||
No response | 1 | 44 | 2% | |||
C | Recruiting in several centres, where treatment providers treat patients across recruiting centres (treatment provider is not unique to a centre) | Experience in trial type | 16 | 44 | 36% | |
Centre | 14 | 16 | 88% | |||
Treatment provider | 3 | 16 | 19% | |||
Both | 1 | 16 | 6% | |||
Neither | 0 | 16 | 0% | |||
No experience in trial type | 27 | 44 | 61% | |||
No response | 1 | 44 | 2% | |||
D | A trial investigating a surgical intervention, recruiting from several centres, each with multiple treatment providers | Experience in trial type | 25 | 44 | 57% | |
Centre | 21 | 25 | 84% | |||
Treatment provider | 3 | 25 | 12% | |||
Both | 5 | 25 | 20% | |||
Neither | 3 | 25 | 12% | |||
No experience in trial type | 17 | 44 | 39% | |||
No response | 2 | 44 | 5% | |||
E | Recruiting from several centres, each with multiple treatment providers, comparing substantially different interventions e.g. surgery to an injection | Experience in trial type | 16 | 44 | 36% | |
Centre | 13 | 16 | 81% | |||
Treatment provider | 0 | 16 | 0% | |||
Both | 2 | 16 | 13% | |||
Neither | 2 | 16 | 13% | |||
No experience in trial type | 26 | 44 | 59% | |||
No response | 2 | 44 | 5% | |||
In scenarios where Unit has experience, approaches to stratification changes across scenario, i.e. within-Unit variation to stratification | Different approaches across scenarios | 20 | 44 | 46% | ||
Same approach across all scenarios | 19 | 44 | 43% | |||
No response to Question 3 | 5 | 44 | 11% | |||
4 | In the trials run by your Unit, have you defined a minimum level of expertise for the health professionals participating in the trial in terms of: | Treating the condition within the patient population | 24 | 44 | 55% | |
Delivering the trial intervention | 31 | 44 | 70% | |||
Setting a minimum professional level of treatment providers | 22 | 44 | 50% | |||
Other approach: | ||||||
Based on paramedic experience (defined by years in service) | 1 | 44 | 2% | |||
Based on surgeon experience (at or beyond a certain level) | 1 | 44 | 2% | |||
Centre required to conduct a certain number of operations per year | 1 | 44 | 2% | |||
Clinical decision for Chief Investigator | 1 | 44 | 2% | |||
Deliverer required to pass surgical and radiotherapy quality assurance | 1 | 44 | 2% | |||
Depends on phase of trial—early or pragmatic require different levels | 1 | 44 | 2% | |||
In our stepwise study, all therapists were experienced but the intervention was brand new | 1 | 44 | 2% | |||
Investigators who define research question are experts in the field and have trained staff to deliver intervention | 1 | 44 | 2% | |||
No consistent approach across all our studies. | 1 | 44 | 2% | |||
No Unit-wide policy—decided trial by trial depending on intervention and setting | 1 | 44 | 2% | |||
Surgeon manuals signed off by ‘senior’ surgeon prior to participation | 1 | 44 | 2% | |||
Surgical team led by consultant, who submits video measured for quality assurance, prior to participation | 1 | 44 | 2% | |||
These have been implicitly taken as a Chief Investigator and Principal Investigator | 1 | 44 | 2% | |||
Training provided to health care professionals in order to participate | 1 | 44 | 2% | |||
No, or no response | 5 | 44 | 11% | |||
5 | Has your Unit conducted trials with an expertise-based design, in which participating treatment providers provide only the intervention in which they have expertise? | Yes, when applicablec | 13 | 44 | 30% | |
No, with justification | 1 | 44 | 2% | |||
No | 26 | 44 | 59% | |||
No response | 4 | 44 | 9% |
Unit has multicentre trials that do not stratify randomisation by centre? | Yes (N = 25) | No (N = 18) | ||
---|---|---|---|---|
Reason(s) provided | ||||
Expected homogeneity of treatment effect across centres | 11 | 44% | 2 | 11% |
No interest in centre effect | 4 | 16% | 1 | 6% |
Lots of centres with few participants per centre | 19 | 76% | 1 | 6% |
Not convinced of appropriateness of either fixed or random effect models for centres in the trial | 1 | 4% | 0 | 0% |
Other reason provided | ||||
Aids in blinding if trial open label | 1 | 4% | 0 | 0% |
Balance against other important factors. Centre effect less important in drug trials compared to complex or surgical interventions | 1 | 4% | 0 | 0% |
Concern that, in an unblinded trial, stratifying by centre would make it easier to predict the treatment allocated to the next patient [12] 16:405). | 1 | 4% | 0 | 0% |
For practical reasons | 0 | 0% | 1 | 6% |
Intervention takes place out of hospital | 1 | 4% | 0 | 0% |
Large sample size with small/moderate number of centres. We expect balance to be achieved with simple randomisation | 1 | 4% | 0 | 0% |
Likely to stratify by geographical region if not by centre | 1 | 4% | 0 | 0% |
Randomisation system defaults to stratifying by centre but one example where minimised trial did not. Need to consider balance of resources and avoid confounding. There is a lot of academic debate. e.g. Torgerson. | 0 | 0% | 1 | 6% |
Sometimes stratify by region | 1 | 4% | 0 | 0% |
Stratified by treatment provider within centres and treatment providers unique within centre | 1 | 4% | 0 | 0% |
Undertaken in limited/exceptional circumstances only, e.g. feasibility studies | 1 | 4% | 0 | 0% |
“This subject gets a lot of academic debate in some academic circles. But: our randomisation defaults to stratifying by centre; need to balance resources—don’t want to give one too many overheads; balancing avoids confounding; other opinions, such as Torgerson, exist.” [ID8]
“Recent conversions between senior statisticians advocate not stratifying by centre in any situation. They cited concerns regarding prediction of allocation.” [ID18]
Learning
Managing effects through analysis
Clustering
“Depends. Either include as a stratifying factor (small number of centres, large patient numbers) or by including centre or treatment provider as a cluster.” [ID32]
Question | Category | Response statistics | ||||
---|---|---|---|---|---|---|
n | N | n/N% | ||||
6 | a | Assuming that you have stratified by centre, do you combine by the stratification factor for the purpose of analysis? If so how? | Yes | 24 | 44 | 55% |
Pre-specified grouping rules at design stage | 19 | 24 | 80% | |||
Ad hoc approach, e.g. determined after design due to small numbers per group | 14 | 24 | 58% | |||
Other grouping rule or further details provided | 6 | 24 | 26% | |||
No | 17 | 44 | 39% | |||
No response | 3 | 44 | 7% | |||
b | Assuming that you have stratified by treatment provider, do you combine by the stratification factor for the purpose of analysis? If so how? | Yes | 16 | 44 | 36% | |
Pre-specified grouping rules at design stage | 12 | 16 | 75% | |||
Ad hoc approach, e.g. determined after design due to small numbers per group | 7 | 16 | 44% | |||
Other grouping rule or further details provided | 5 | 16 | 31% | |||
No | 14 | 44 | 32% | |||
No experience with trials of this type | 1 | 44 | 2% | |||
No response | 13 | 44 | 30% | |||
7 | Does your Unit include centre in the statistical model when comparing treatment? | Yes | 39 | 44 | 89% | |
But only if it was used to stratify randomisation | 18 | 39 | 46% | |||
Always | 6 | 39 | 15% | |||
Sometimesa | 15 | 39 | 38% | |||
No, never | 3 | 44 | 7% | |||
No responseb | 2 | 44 | 5% | |||
a | If yes, and assuming that the sample size allows either, would you treat this effect as fixed or random? | Fixed or random, depending on circumstances | 14 | 39 | 36% | |
Fixed | 11 | 39 | 28% | |||
Random | 12 | 39 | 31% | |||
No response | 2 | 39 | 5% | |||
8 | Does your Unit include treatment provider in the statistical model when comparing treatment? | Yes | 26 | 44 | 59% | |
But only if it was used to stratify randomisation | 8 | 26 | 31% | |||
Always | 0 | 26 | 0% | |||
Sometimesc | 18 | 26 | 69% | |||
No, never | 13 | 46 | 30% | |||
No responsed | 5 | 44 | 11% | |||
a | If yes, and assuming that the sample size allows either, would you treat this effect as fixed or random? | Fixed or random, depending on circumstances | 4 | 26 | 15% | |
Fixed | 2 | 26 | 8% | |||
Random | 18 | 26 | 69% | |||
No response | 2 | 26 | 8% | |||
b | If yes, has this effect ever been treated as time-varying within the statistical model? | Yes | 2 | 26 | 8% | |
No | 21 | 26 | 81% | |||
No response | 3 | 26 | 12% | |||
9 | In each of the following scenarios, regardless of the randomisation stratification approach, has a treatment by centre or surgeon interaction investigated, in trials that your Unit has run? Select all that apply. | |||||
A | Large sample size,e recruiting in several centres, each with multiple treatment providers | Experience in trial type | 35 | 44 | 80% | |
Centre | 16 | 35 | 46% | |||
Treatment provider | 4 | 35 | 11% | |||
Both | 3 | 35 | 9% | |||
Neither | 20 | 35 | 57% | |||
No experience in trial type | 7 | 44 | 16% | |||
No response | 2 | 44 | 5% | |||
B | Small sample size,f With centres each recruiting 2 to 3 patients | Experience in trial type | 30 | 44 | 68% | |
Centre | 5 | 30 | 17% | |||
Treatment provider | 0 | 30 | 0% | |||
Both | 0 | 30 | 0% | |||
Neither | 25 | 30 | 83% | |||
No experience in trial type | 12 | 44 | 27% | |||
No response | 2 | 44 | 5% | |||
C | Recruiting in several centres, where treatment providers treat patients across recruiting centres (treatment provider is not unique to a centre) | Experience in trial type | 15 | 44 | 34% | |
Centre | 4 | 15 | 27% | |||
Treatment provider | 1 | 15 | 7% | |||
Both | 0 | 15 | 0% | |||
Neither | 11 | 15 | 73% | |||
No experience in trial type | 27 | 44 | 61% | |||
No response | 2 | 44 | 5% | |||
D | A trial investigating a surgical intervention, recruiting from several centres, each with multiple treatment providers | Experience in trial type | 21 | 44 | 48% | |
Centre | 5 | 19 | 24% | |||
Treatment provider | 3 | 19 | 14% | |||
Both | 1 | 19 | 5% | |||
Neither | 14 | 19 | 67% | |||
No experience in trial type | 19 | 44 | 43% | |||
No response | 4 | 44 | 9% | |||
E | Recruiting from several centres, each with multiple treatment providers, comparing substantially different interventions, e.g. surgery to an injection | Experience in trial type | 14 | 44 | 32% | |
Centre | 5 | 14 | 36% | |||
Treatment provider | 1 | 14 | 7% | |||
Both | 0 | 14 | 0% | |||
Neither | 9 | 14 | 64% | |||
No experience in trial type | 26 | 44 | 59% | |||
No response | 4 | 44 | 9% | |||
In scenarios where Unit has experience, approaches to stratification changes across scenario, i.e. within-Unit variation to stratification | Different approaches across scenarios | 12 | 44 | 27% | ||
Same approach across all scenarios | 24 | 44 | 55% | |||
No response to Question 9 | 8 | 44 | 18% | |||
10 | a | If a positive treatment effect is found, does your Unit explore heterogeneity of treatment effects by centre? | Yes | 32 | 44 | 73% |
No | 9 | 44 | 20% | |||
No response | 3 | 44 | 7% | |||
i. If yes to a, do you explore by graphical display? | Yes | 31 | 32 | 97% | ||
No | 0 | 32 | 3% | |||
No response | 1 | 32 | 3% | |||
ii. If yes to a, do you explore by analytical methods, e.g. significance testing? | Yes | 22 | 32 | 69% | ||
No | 5 | 32 | 16% | |||
No response | 5 | 32 | 16% | |||
b | If a positive treatment effect is found, does your Unit explore heterogeneity of treatment effects by treatment provider? | Yes | 12 | 44 | 27% | |
No | 23 | 44 | 52% | |||
No response | 9 | 44 | 20% | |||
i. If yes to b, do you explore by graphical display? | Yes | 11 | 12 | 92% | ||
No | 0 | 12 | 0% | |||
No response | 1 | 12 | 8% | |||
ii. If yes to b, would you explore by analytical methods, e.g. significance testing? | Yes | 9 | 12 | 75% | ||
No | 1 | 12 | 8% | |||
No response | 2 | 12 | 17% |
Centre stratified: | |
ID4 | Would normally analyse together but adjust for stratification factors (which normally include centres) in analysis. |
ID7 | There will be instances where we have combined centres at the analysis stage due to small numbers. |
ID8 | Different statisticians/trials do different things. Often site = fixed effect and course within site = random effect. If too few within site, then would combine. |
ID14 | Retain structure at analysis. |
ID15 | Have grouped by region/country where numbers are small. Any adjustment should be documented in the Statistical Analysis Plan, and final decision regarding appropriateness can be discussed during blind review of data. |
ID30 | Have used both pre-specified and ad hoc approaches (due to recruitment issues). |
Not stratified by centre: | |
ID32 | We either include as a stratification factor (small number of centres, large patient numbers) or by including centre/provider as a cluster. |
Treatment provider stratified: | |
ID7 | Thinking about complex intervention studies, we don’t usually allow for a ’provider’ effect in the primary analyses, although not necessarily explicitly stated in protocol—many of these studies effectively have partial clustering. We’ve had recent interesting discussions regarding provider effect in such trials, with Chief Investigators strongly feeling that with standardised/manualised intervention and training, it isn’t relevant. |
ID15 | Any adjustment should be documented in the Statistical Analysis Plan, and final decision regarding appropriateness can be discussed during blind review of data. |
ID24 | Experience with multiple treatment providers is in oncology trials with different doctors delivering protocol treatment, e.g. chemotherapy/radiotherapy. The actual treating doctor has not been recorded on the Case Report Forms, hence all providers implicitly combined within a centre. |
ID30 | Have used both pre-specified and ad hoc approaches (due to recruitment issues). |
ID39 | Treatment providers combined by default—as we don’t routinely distinguish them in the analysis. |
“Usually an underlying assumption that centre may be a surrogate for socioeconomic factors that may affect outcome and/or treatment effect and so often not happy to assume that there is an equal fixed treatment effect across all sites.” [ID15]
“If treatment provider was included as stratification factor it will be because we are concerned that the provider will have an impact on outcome but also because we would expect different population for different treatment providers.” [ID15]
Learning
“Fairly crude by letting the number of procedures in the trial increase the relevant surgeon’s experience (ignoring procedures done outside of the trial of course!)” [ID38]
“Had we found evidence of learning, we would have had awkward additional data summaries and presentations” [ID8]