Separation Tests for Early-Phase Complementary and Alternative Medicine Comparative Trials

Abstract

There are some substantial ways in which the current approach to establishing an evidence base for a treatment effect works against complementary and alternative medicine (CAM) research. The standard statistical method in comparative trials is the null hypothesis test, which requires a decision that the ‘no treatment effect’ hypothesis is confirmed, or it is rejected. While this approach is appropriate for evidence-based medicine, the requirement for a decisive result drives sample sizes upward, or leaves small trials subject to being criticised for being underpowered. Employing the fundamental theory of hypothesis tests, as developed by Neyman and Pearson, it is possible to define a ‘separation test’ that avoids this problem, by invoking a different inferential rationale. The purpose of a separation test is to find an indication that further research is justified, or that it is not. This change in strategy can considerably lower the required sample size. Since many CAM comparative trials are in early phases of research, where both budgetary and safety considerations argue for a small sample size, separation tests may be especially of interest to CAM researchers. There is, therefore, an opportunity for evidence-based medicine to generate a new kind of study, in which the purpose is to assess whether it is worthwhile to pursue research on an alternative treatment, rather than to determine whether it has been proved effective.