" ...test of the methods and procedures to be used on a larger scaleif the pilot study demonstrates that the methods and procedures can work" ;
"...investigation designed to test the feasibility of methods and proceduresfor later use on a large scale or to search for possible effects and associationsthat may be worth following up in a subsequent larger study" .
A trial study carried out before a research design is finalised to assist in defining the research question or to test the feasibility, reliability and validity of the proposed study design
A smaller version of a study is carried out before the actual investigation is done. Researchers use information gathered in pilot studies to refine or modify the research methodology for a study and to develop large-scale studies
A small scale study conducted to test the plan and method of a research study.
A small study carried out before a large-scale study to try out a procedure or to test a principle
An experimental use of a treatment in a small group of patients to learn if it will be effective and safe on a broad scale
The initial study examining a new method or treatment
A small study often done to assist the preparation of a larger, more comprehensive investigation.
Small, preliminary test or trial run of an intervention, or of an evaluation activity such as an instrument or sampling procedure. The results of the pilot are used to improve the program or evaluation procedure being piloted before it is used on a larger scale.
2. Narrowing the focus: Pilot studies for randomized studies
3. Reasons for Conducting Pilot Studies
Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc.
Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms.
Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres.
Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance.
Process: This assesses the feasibility of the processes that are key to the success of the main study
• Recruitment rates
• Retention rates
• Refusal rates
• Failure/success rates
• (Non)compliance or adherence rates
• eligibility criteria
- Is it obvious who meets and who does not meet the eligibility requirements?
- Are the eligibility criteria sufficient or too restrictive?
• Understanding of study questionnaires or data collection tools:
- Do subjects provide no answer, multiple answers, qualified answers, or unanticipated answers to study questions?
Resources: This deals with assessing time and resource problems that can occur during the main study
• Length of time to fill out all the study forms
• Determining capacity:
- Will the study participants overload your phone lines or overflow your waiting room?
• Determining process time
- How much time does it take to mail out a thousand surveys?
• Is the equipment readily available when and where it is needed?
• What happens when it breaks down or gets stolen?
• Can the software used for capturing data read and understand the data?
• Determining centre willingness and capacity
- Do the centres do what they committed to doing?
- Do investigators have the time to Perform the tasks they committed to doing?
- Are there any capacity issues at each participating centre?
Management: This covers potential human and data management problems
• What are the challenges that participating centres have with managing the study?
• What challenges do study personnel have?
• Is there enough room on the data collection form for all of the data you receive?
• Are there any problems entering data into the computer?
• Can data coming from different sources be matched?
• Were any important data values forgotten about?
• Do data show too much or too little variability?
Scientific: This deals with the assessment of treatment safety, dose, response, effect and variance of the effect
• Is it safe to use the study drug/intervention?
• What is the safe dose level?
• Do patients respond to the drug?
• What is the estimate of the treatment effect?
• What is the estimate of the variance of the treatment effect?
4. Relationships between Pilot Studies, Proof-of-Concept Studies, and Adaptive Designs
5. Challenges of and Common Misconceptions about Pilot Studies
So-and-so did a similar study with 6 patients and got statistical significance - ours uses 12 patients (double the size)!
We did a similar pilot before (and it was published!)
I have funding for 10 patients only;
I have limited seed (start-up) funding;
This is just a student project!
My supervisor (boss) told me to do it as a pilot.
Sample Size for Pilot Studies
6. How to Interpret the Results of a Pilot Study: Criteria for Success
98.5% of patients had to receive study drug within 12 hours of randomization;
91.7% of patients had to receive every scheduled dose of the study drug in a blinded manner;
90% or more of patients had to have lower limb compression ultrasounds performed at the specified times; and
> 90% of necessary dose adjustments had to have been made appropriately in response to pre-defined laboratory criteria.
one subject per centre per week (i.e., 200 subjects from four centres over 50 weeks) can be recruited;
at least 70% of all eligible patients can be recruited;
no more than 5% of all recruited subjects crossed over from one modality to the other; and
complete follow-up in at least 95% of all recruited subjects.
7. Frequently asked questions about pilot studies
Can I publish the results of a pilot study?
Why is it important to publish the results of pilot studies?
Can I combine data from a pilot with data from the main study?
Can I combine the results of a pilot with the results of another study or in a meta-analysis?
Can the results of the pilot study be valid on their own, without existence of the main study
Can I apply for funding for a pilot study?
Can I randomize patients in a pilot study?
How can I use the information from a pilot to estimate the sample size?
Can I use the results of a pilot study to treat my patients?
What can I do with a failed or bad pilot study?
8. Ethical Aspects of Pilot Studies
9. Recommendation for Reporting the Results of Pilot Studies
Reported on Page #
TITLE and ABSTRACT
Does the title or abstract indicate that the study is a "pilot"?
Scientific background for the main study and explanation of rationale for assessing feasibility through piloting
Participants and setting
• Eligibility criteria for participants in the pilot study (these should be the same as in the main study -- if different, state the differences)
• The settings and locations where the data were collected
Provide precise details of the interventions intended for each group and how and when they were actually administered (if applicable) -- state clearly if any aspects of the intervention are assessed for feasibility
• Specific scientific objectives and hypotheses for the main study
• Specific feasibility objectives
• Clearly defined primary and secondary outcome measures for the main study
• Clearly define the feasibility outcomes and how they were operationalized -- these should include key elements such as recruitment rates, consent rates, completion rates, variance estimates, etc
Describe how sample size was determined
• In general for a pilot of a phase III trial, there is no need for a formal sample size calculation. However, confidence interval approach may be used to calculate and justify the sample size based on key feasibility objective(s).
Clearly describe the criteria for assessing success of feasibility -- these should be based on the feasibility objectives
Describe the statistical methods for the analysis of primary and secondary feasibility outcomes
• State whether the study received research ethics approval
• State how informed consent was handled -- given the feasibility nature of the study
Flow of participants through each stage (a flow-chart is strongly recommended).
• Describe protocol deviations from pilot study as planned, together with reasons
• State the number of exclusions at each stage and reasons for exclusions
Report the dates defining the periods of recruitment and follow-up
Report the baseline demographic and clinical characteristics of the participants
Outcomes and estimation
For each primary and secondary feasibility outcome, report the point estimate of effect and its precision (e.g., 95% confidence interval [CI]) -- if applicable
Interpretation of the results should focus on feasibility, taking into account
• the stated criteria for success of feasibility;
• study hypotheses, sources of potential bias or imprecision -- given the feasibility nature of the study
• the dangers associated with multiplicity of analyses and outcomes
Generalizability (external validity) of the feasibility. State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible
Overall evidence of feasibility
General interpretation of the results in the context of current evidence of feasibility
• Focus should be on feasibility