Skip to main content
SpringerLink
Log in

Classes of Multiple Test Procedures

  • Chapter
  • First Online:
Simultaneous Statistical Inference

  • 1914 Accesses

Abstract

The aim of this chapter is a systematic overview of different classes of multiple tests. Procedures are distinguished by their structure, by the degree of detail of the underlying statistical model and by the type of error control that they provide. Major categories comprise margin-based multiple tests, multivariate multiple test procedures and closed test procedures. Subcategories are introduced where appropriate. We discuss specific examples and indicate computer implementations by means of flow diagrams and pseudo-code. Applications and references to later chapters illustrate which kind of multiple test procedure can be utilized for some standard types of multiple test problems which are relevant in practice. Precise references to the literature are collected for a deeper study of specific methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Arlot S, Blanchard G, Roquain E (2010) Some nonasymptotic results on resampling in high dimension. II: Multiple tests. Ann Stat 38(1):83–99. doi:10.1214/08-AOS668

    Article  Google Scholar 

  • Bauer P, Röhmel J, Maurer W, Hothorn L (1998) Testing strategies in multi-dose experiments including active control. Stat Med 17(18):2133–2146

    Article  CAS  PubMed  Google Scholar 

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Roy Stat Soc Ser B (Methodol) 57(1):289–300

    Google Scholar 

  • Benjamini Y, Hochberg Y (2000) On the adaptive control of the false discovery rate in multiple testing with independent statistics. J Edu Behav Stat 25:60–83

    Google Scholar 

  • Benjamini Y, Krieger AM, Yekutieli D (2006) Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93(3):491–507

    Article  Google Scholar 

  • Bodnar T, Dickhaus T (2013) False Discovery Rate Control under Archimedean Copula. arXiv:1305.3897

    Google Scholar 

  • Bonferroni CE (1935) Il calcolo delle assicurazioni su gruppi di teste. Studi in onore Salvatore Ortu Carboni 13–60.

    Google Scholar 

  • Bonferroni CE (1936) Teoria statistica delle classi e calcolo delle probabilita. Pubbl. d. R. Ist. Super. di Sci. Econom. e Commerciali di Firenze 8. Firenze: Libr. Internaz. Seeber.

    Google Scholar 

  • Bretz F, Hothorn T, Westfall P (2010) Multiple Comparisons Using R. Chapman and Hall/CRC

    Google Scholar 

  • Dickhaus T (2013) Randomized \(p\)-values for multiple testing of composite null hypotheses. J Stat Plann Infer 143(11):1968–1979

    Article  Google Scholar 

  • Dickhaus T, Gierl J (2013) Simultaneous test procedures in terms of p-value copulae. Global Science and Technology Forum (GSTF). In: Proceedings on the 2nd Annual International Conference on Computational Mathematics, Computational Geometry and Statistics (CMCGS 2013), vol 2, pp 75–80

    Google Scholar 

  • Dickhaus T, Stange J (2013) Multiple point hypothesis test problems and effective numbers of tests for control of the family-wise error rate. Calcutta Statist Assoc Bull, to appear

    Google Scholar 

  • Dickhaus T, Strassburger K, Schunk D, Morcillo-Suarez C, Illig T, Navarro A (2012) How to analyze many contingency tables simultaneously in genetic association studies. Stat Appl Genet Mol Biol 11(4):Article 12

    Google Scholar 

  • Dudoit S, van der Laan MJ (2008) Multiple testing procedures with applications to genomics. Springer Series in Statistics. Springer, New York

    Google Scholar 

  • Farcomeni A (2009) Generalized augmentation to control false discovery exceedance in multiple testing. Scand J Stat 36(3):501–517

    Google Scholar 

  • Fengler MR, Okhrin O (2012) Realized Copula. SFB 649 Discussion Paper 2012–034, Sonderforschungsbereich 649, Humboldt-Universität zu Berlin, Germany, available at http://sfb649.wiwi.hu-berlin.de/papers/pdf/SFB649DP2012-034.pdf

  • Finner H (1999) Stepwise multiple test procedures and control of directional errors. Ann Stat 27(1):274–289. doi:10.1214/aos/1018031111

    Article  Google Scholar 

  • Finner H, Gontscharuk V (2009) Controlling the familywise error rate with plug-in estimator for the proportion of true null hypotheses. J Roy Stat Soc B 71(5):1031–1048

    Article  Google Scholar 

  • Finner H, Strassburger K (2002) The partitioning principle: a powerful tool in multiple decision theory. Ann Stat 30(4):1194–1213

    Article  Google Scholar 

  • Finner H, Gontscharuk V, Dickhaus T (2012) False discovery rate control of step-up-down tests with special emphasis on the asymptotically optimal rejection curve. Scand J Stat 39:382–397

    Article  Google Scholar 

  • Goeman JJ, Finos L (2012) The inheritance procedure: multiple testing of tree-structured hypotheses. Stat Appl Genet Mol Biol 11(1):Article 11

    Google Scholar 

  • Hochberg Y, Tamhane AC (1987) Multiple comparison procedures. Wiley Series in Probability and Mathematical Statistics. Applied Probability and Statistics. Wiley, New York

    Google Scholar 

  • Hofert M, Mächler M, McNeil AJ (2012) Likelihood inference for Archimedean copulas in high dimensions under known margins. J Multivariate Anal 110:133–150. doi:10.1016/j.jmva.2012.02.019

    Article  Google Scholar 

  • Höffding W (1940) Maßstabinvariante Korrelationstheorie. Schr math Inst u Inst angew Math Univ Berlin 5:181–233

    Google Scholar 

  • Holm SA (1979) A simple sequentially rejective multiple test procedure. Scand J Stat Theory Appl 6:65–70

    Google Scholar 

  • Hothorn T, Bretz F, Westfall P (2008) Simultaneous inference in general parametric models. Biom J 50(3):346–363

    Article  PubMed  Google Scholar 

  • Hsu JC (1996) Multiple comparisons: theory and methods. Chapman and Hall, London

    Book  Google Scholar 

  • van der Laan MJ, Dudoit S, Pollard KS (2004), Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat Appl Genet Mol Biol 3: Article15

    Google Scholar 

  • van der Laan MJ, Birkner MD, Hubbard AE (2005), Empirical Bayes and resampling based multiple testing procedure controlling tail probability of the proportion of false positives. Stat Appl Genet Mol Biol 4:Article29

    Google Scholar 

  • Langaas M, Lindqvist BH, Ferkingstad E (2005) Estimating the proportion of true null hypotheses, with application to DNA microarray data. J Roy Stat Soc B 67(4):555–572. doi:10.1111/j.1467-9868.2005.00515.x

    Article  Google Scholar 

  • Lehmann EL, Romano JP (2005) Testing statistical hypotheses. Springer Texts in Statistics, 3rd edn. Springer, New York

    Google Scholar 

  • Marcus R, Peritz E, Gabriel KR (1976) On closed test procedures with special reference to ordered analysis of variance. Biometrika 63(3):655–660

    Article  Google Scholar 

  • Meinshausen N (2008) Hierarchical testing of variable importance. Biometrika 95(2):265–278. doi:10.1093/biomet/asn007

    Article  Google Scholar 

  • Meinshausen N, Maathuis MH, Bühlmann P (2011) Asymptotic optimality of the Westfall-Young permutation procedure for multiple testing under dependence. Ann Stat 39(6):3369–3391. doi:10.1214/11-AOS946

    Article  Google Scholar 

  • Moskvina V, Schmidt KM (2008) On multiple-testing correction in genome-wide association studies. Genet Epidemiol 32:567–573

    Article  PubMed  Google Scholar 

  • Nelsen RB (2006) An introduction to copulas. Springer series in statistics. 2nd edn. Springer,New York

    Google Scholar 

  • Pollard KS, van der Laan MJ (2004) Choice of a null distribution in resampling-based multiple testing. J Stat Plann Infer 125(1–2):85–100. doi:10.1016/j.jspi.2003.07.019

    Article  Google Scholar 

  • Romano JP, Wolf M (2005a) Exact and approximate stepdown methods for multiple hypothesis testing. J Am Stat Assoc 100(469):94–108. doi:10.1198/016214504000000539

    Article  CAS  Google Scholar 

  • Romano JP, Wolf M (2005) Stepwise multiple testing as formalized data snooping. Econometrica 73(4):1237–1282. doi:10.1111/j.1468-0262.2005.00615.x

    Article  Google Scholar 

  • Romano JP, Shaikh AM, Wolf M (2008) Control of the false discovery rate under dependence using the bootstrap and subsampling. Test 17(3):417–442. doi:10.1007/s11749-008-0126-6

    Article  Google Scholar 

  • Scheffé H (1953) A method for judging all contrasts in the analysis of variance. Biometrika 40:87–110

    Google Scholar 

  • Schweder T, Spjøtvoll E (1982) Plots of \(P\)-values to evaluate many tests simultaneously. Biometrika 69:493–502

    Google Scholar 

  • Šidák Z (1967) Rectangular confidence regions for the means of multivariate normal distributions. J Am Stat Assoc 62:626–633. doi:10.2307/2283989

    Google Scholar 

  • Sonnemann E (2008) General solutions to multiple testing problems. Translation of "Sonnemann, E. (1982). Allgemeine Lösungen multipler Testprobleme. EDV in Medizin und Biologie 13(4), 120–128". Biom J 50:641–656

    Article  PubMed  Google Scholar 

  • Storey JD, Taylor JE, Siegmund D (2004) Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. J. Roy. Stat. Soc. B, Stat Methodol 66(1):187–205

    Article  Google Scholar 

  • Tamhane AC, Liu W, Dunnett CW (1998) A generalized step-up-down multiple test procedure. Can J Stat 26(2):353–363. doi:10.2307/3315516

    Article  Google Scholar 

  • Troendle JF (1995) A stepwise resampling method of multiple hypothesis testing. J Am Stat Assoc 90(429):370–378. doi:10.2307/2291163

    Article  Google Scholar 

  • Troendle JF (2000) Stepwise normal theory multiple test procedures controlling the false discovery rate. J Stat Plann Infer 84(1–2):139–158. doi:10.1016/S0378-3758(99)00145-7

    Article  Google Scholar 

  • Van der Vaart A (1998) Asymptotic statistics. Cambridge Series in Statistical and Probabilistic Mathematics, 3. Cambridge University Press: Cambridge. doi:10.1017/CBO9780511802256

  • Westfall PH, Krishen A (2001) Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. J Stat Plann Infer 99(1):25–40. doi:10.1016/S0378-3758(01)00077-5

    Article  Google Scholar 

  • Westfall PH, Tobias RD (2007) Multiple testing of general contrasts: truncated closure and the extended Shaffer-Royen method. J Am Stat Assoc 102(478):487–494. doi:10.1198/016214506000001338

    Article  CAS  Google Scholar 

  • Westfall PH, Young SS (1993) Resampling-based multiple testing: examples and methods for p-value adjustment. Wiley Series in Probability and Mathematical Statistics, Applied Probability and Statistics. Wiley, New York

    Google Scholar 

  • Wiens BL (2003) A fixed sequence Bonferroni procedure for testing multiple endpoints. Pharmaceut Statist 2:211215

    Article  Google Scholar 

  • Yekutieli D, Benjamini Y (1999) Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plann Infer 82(1–2):171–196. doi:10.1016/S0378-3758(99)00041-5

    Article  Google Scholar 

Download references

Acknowledgments

The material on copula-based multiple tests originated from joint work with Taras Bodnar, Jakob Gierl and Jens Stange. Helmut Finner and Klaus Straßburger taught me everything I know about closed test procedures and the partitioning principle. Special thanks are due to Mareile Große Ruse for programming the LaTeX code used for the figures.

Author information

Authors and Affiliations

  1. Research Group “Stochastic Algorithms and Nonparametric Statistics”, Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany

    Thorsten Dickhaus

Authors
  1. Thorsten Dickhaus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thorsten Dickhaus .

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Dickhaus, T. (2014). Classes of Multiple Test Procedures. In: Simultaneous Statistical Inference. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45182-9_3

Download citation

Publish with us

Policies and ethics

Search

Navigation