Jump to ContentJump to Main Navigation
Show Summary Details
In This Section

Statistical Applications in Genetics and Molecular Biology

Editor-in-Chief: Stumpf, Michael P.H.

6 Issues per year

IMPACT FACTOR 2016: 0.646
5-year IMPACT FACTOR: 1.191

CiteScore 2016: 0.94

SCImago Journal Rank (SJR) 2015: 0.954
Source Normalized Impact per Paper (SNIP) 2015: 0.554

Mathematical Citation Quotient (MCQ) 2015: 0.06

See all formats and pricing
In This Section
Volume 9, Issue 1 (Oct 2010)


Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn

Belinda Phipson
  • The Walter and Eliza Hall Institute of Medical Research
/ Gordon K Smyth
  • The Walter and Eliza Hall Institute of Medical Reseach
Published Online: 2010-10-31 | DOI: https://doi.org/10.2202/1544-6115.1585

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.

Keywords: permutation test; Monte Carlo test; p-values; multiple testing; microarray

About the article

Published Online: 2010-10-31

Citation Information: Statistical Applications in Genetics and Molecular Biology, ISSN (Online) 1544-6115, DOI: https://doi.org/10.2202/1544-6115.1585. Export Citation

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Jessica L Larson and Art B Owen
BMC Bioinformatics, 2015, Volume 16, Number 1
A. Hahn, G. S. Kranz, M. Kublbock, U. Kaufmann, S. Ganger, A. Hummer, R. Seiger, M. Spies, D. Winkler, S. Kasper, C. Windischberger, D. F. Swaab, and R. Lanzenberger
Cerebral Cortex, 2014
Jelle J. Goeman and Aldo Solari
Statistics in Medicine, 2014, Volume 33, Number 11, Page 1946

Comments (0)

Please log in or register to comment.
Log in