Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Statistical Applications in Genetics and Molecular Biology

Editor-in-Chief: Sanguinetti, Guido

IMPACT FACTOR 2018: 0.536
5-year IMPACT FACTOR: 0.764

CiteScore 2018: 0.49

SCImago Journal Rank (SJR) 2018: 0.316
Source Normalized Impact per Paper (SNIP) 2018: 0.342

Mathematical Citation Quotient (MCQ) 2017: 0.04

See all formats and pricing
More options …
Volume 9, Issue 1


Volume 10 (2011)

Volume 9 (2010)

Volume 6 (2007)

Volume 5 (2006)

Volume 4 (2005)

Volume 2 (2003)

Volume 1 (2002)

Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn

Belinda Phipson / Gordon K Smyth
Published Online: 2010-10-31 | DOI: https://doi.org/10.2202/1544-6115.1585

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.

Keywords: permutation test; Monte Carlo test; p-values; multiple testing; microarray

About the article

Published Online: 2010-10-31

Citation Information: Statistical Applications in Genetics and Molecular Biology, Volume 9, Issue 1, ISSN (Online) 1544-6115, DOI: https://doi.org/10.2202/1544-6115.1585.

Export Citation

©2011 Walter de Gruyter GmbH & Co. KG, Berlin/Boston.Get Permission

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Shinnosuke Nakayama, Tyrone J. Tolbert, Oded Nov, and Maurizio Porfiri
Journal of the Association for Information Science and Technology, 2018
Christian S. Zang, Susanne Jochner-Oette, José Cortés, Anja Rammig, and Annette Menzel
Climate Dynamics, 2018
Anton Albajes-Eizagirre, Aleix Solanes, Eduard Vieta, and Joaquim Radua
NeuroImage, 2018
Andreas Marzoll, Tan Saygi, and Hubert R. Dinse
Scientific Reports, 2018, Volume 8, Number 1
James A. Brissenden, Sean M. Tobyne, David E. Osher, Emily J. Levin, Mark A. Halko, and David C. Somers
Current Biology, 2018
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Juyoung Lee, Takahide Kagawa, and Yoshifumi Miyazaki
Frontiers in Public Health, 2018, Volume 6
Matthew M. Parks, Benjamin J. Raphael, and Charles E. Lawrence
BMC Bioinformatics, 2018, Volume 19, Number 1
Shu Mei Teo, Howard H.F. Tang, Danny Mok, Louise M. Judd, Stephen C. Watts, Kym Pham, Barbara J. Holt, Merci Kusel, Michael Serralha, Niamh Troy, Yury A. Bochkov, Kristine Grindle, Robert F. Lemanske, Sebastian L. Johnston, James E. Gern, Peter D. Sly, Patrick G. Holt, Kathryn E. Holt, and Michael Inouye
Cell Host & Microbe, 2018, Volume 24, Number 3, Page 341
Daniel Feuerriegel, Hannah A.D. Keage, Bruno Rossion, and Genevieve L. Quek
Biological Psychology, 2018
Christopher Hübel, Virpi Leppä, Gerome Breen, and Cynthia M. Bulik
International Journal of Eating Disorders, 2018
Pamela A. Moyer, Margaret S. Boettcher, Jeffery J. McGuire, and John A. Collins
Journal of Geophysical Research: Solid Earth, 2018
Seung Ah Choi, Jun-Won Yun, Kyeung Min Joo, Ji Yeoun Lee, Pil Ae Kwak, Young Eun Lee, Ji-Ran You, Euna Kwon, Woo Ho Kim, Kyu-Chang Wang, Ji Hoon Phi, Byeong-Cheol Kang, and Seung-Ki Kim
Stem Cells and Development, 2016, Volume 25, Number 12, Page 897
Kirill V. Nourski, Mitchell Steinschneider, Ariane E. Rhone, Hiroto Kawasaki, Matthew A. Howard, and Matthew I. Banks
NeuroImage, 2018
Di Wu and Gordon K. Smyth
Nucleic Acids Research, 2012, Volume 40, Number 17, Page e133
Joseph Cursons, Katherine A. Pillman, Kaitlin G. Scheer, Philip A. Gregory, Momeneh Foroutan, Soroor Hediyeh-Zadeh, John Toubia, Edmund J. Crampin, Gregory J. Goodall, Cameron P. Bracken, and Melissa J. Davis
Cell Systems, 2018
Koseki J. Kobayashi-Kirschvink, Hidenori Nakaoka, Arisa Oda, Ken-ichiro F. Kamei, Kazuki Nosho, Hiroko Fukushima, Yu Kanesaki, Shunsuke Yajima, Haruhiko Masaki, Kunihiro Ohta, and Yuichi Wakamoto
Cell Systems, 2018
Anne Steininger, Grit Ebert, Benjamin V. Becker, Chalid Assaf, Markus Möbs, Christian A. Schmidt, Piotr Grabarczyk, Lars R. Jensen, Grzegorz K. Przybylski, Matthias Port, Andreas W. Kuss, and Reinhard Ullmann
Frontiers in Oncology, 2018, Volume 8
Adrià Tauste Campo, Marina Martinez-Garcia, Verónica Nácher, Rogelio Luna, Ranulfo Romo, and Gustavo Deco
Proceedings of the National Academy of Sciences, 2015, Volume 112, Number 15, Page 4761
Andrew D. Rouillard, Mark R. Hurle, Pankaj Agarwal, and Edwin Wang
PLOS Computational Biology, 2018, Volume 14, Number 5, Page e1006142
Aaron T.L. Lun, Davis J. McCarthy, and John C. Marioni
F1000Research, 2016, Volume 5, Page 2122
Aaron T.L. Lun, Davis J. McCarthy, and John C. Marioni
F1000Research, 2016, Volume 5, Page 2122
Willem van den Boom, Callie Mao, Rebecca A Schroeder, and David B Dunson
Bioinformatics, 2018
Matthew M Parks and Ziv Bar-Joseph
Bioinformatics, 2018, Volume 34, Number 6, Page 971
Bradley P. Harris, Charles F. Adams, and Kevin D. E. Stokesbury
Ecosphere, 2018, Volume 9, Number 3, Page e02133
Xiang Zhang, Adil Mardinoglu, Leo A. B. Joosten, Jan A. Kuivenhoven, Yang Li, Mihai G. Netea, and Albert K. Groen
Frontiers in Physiology, 2018, Volume 9
Jarrod Shilts, Guanhua Chen, and Jacob J. Hughey
PeerJ, 2018, Volume 6, Page e4327
Andrew S. Blaeser, Barry W. Connors, and Arto V. Nurmikko
Journal of Neurophysiology, 2017, Volume 117, Number 4, Page 1581
Ezequiel Mikulan, Eugenia Hesse, Lucas Sedeño, Tristán Bekinschtein, Mariano Sigman, María del Carmen García, Walter Silva, Carlos Ciraolo, Adolfo M. García, and Agustín Ibáñez
NeuroImage, 2017
Jesse Hemerik and Jelle Goeman
TEST, 2017
Hao Tam Ho, Johahn Leung, David C. Burr, David Alais, and Maria Concetta Morrone
Current Biology, 2017
Sarah Fleischer, Stefanie Ries, Ping Shen, Alix Lheritier, Frédéric Cazals, Gerd. R. Burmester, Thomas Dörner, and Simon Fillatreau
European Journal of Immunology, 2017
Edina Nemesházi, Krisztián Szabó, Zoltán Horváth, and Szilvia Kövér
Journal of Ornithology, 2017
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Takahide Kagawa, and Yoshifumi Miyazaki
International Journal of Environmental Research and Public Health, 2017, Volume 14, Number 9, Page 987
Antonio Drago, Concetta Crisafulli, Marco Calabrò, and Alessandro Serretti
Journal of Affective Disorders, 2015, Volume 179, Page 88
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Juyoung Lee, Takahide Kagawa, and Yoshifumi Miyazaki
International Journal of Environmental Research and Public Health, 2017, Volume 14, Number 8, Page 931
Antonio Drago, Barbara Monti, Diana De Ronchi, and Alessandro Serretti
Psychiatry Investigation, 2015, Volume 12, Number 1, Page 118
Christopher Minas and Giovanni Montana
Statistical Analysis and Data Mining: The ASA Data Science Journal, 2014, Volume 7, Number 6, Page 450
Dennis Goldfarb, Bridgid E. Hast, Wei Wang, and Michael B. Major
Journal of Proteome Research, 2014, Volume 13, Number 12, Page 5944
Daniel D. Kinnamon and Eden R. Martin
Genetic Epidemiology, 2014, Volume 38, Number 4, Page 325
Thibault Gayet, Sébastien Devillard, Marlène Gamelon, Serge Brandt, Ludovic Say, and Eric Baubet
Evolution, 2016, Volume 70, Number 6, Page 1386
Anderson M. Winkler, Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols
NeuroImage, 2014, Volume 92, Page 381
Christopher Minas, Edward Curry, and Giovanni Montana
Bioinformatics, 2013, Volume 29, Number 20, Page 2555
George C. Tseng, Debashis Ghosh, and Eleanor Feingold
Nucleic Acids Research, 2012, Volume 40, Number 9, Page 3785
Pashupati Mishra, Petri Törönen, Yrjö Leino, and Liisa Holm
Bioinformatics, 2014, Volume 30, Number 19, Page 2747
Jonas Paulsen, Tonje G. Lien, Geir Kjetil Sandve, Lars Holden, Ørnulf Borgan, Ingrid K. Glad, and Eivind Hovig
Nucleic Acids Research, 2013, Volume 41, Number 10, Page 5164
Aline C Planello, Junfeng Ji, Vivek Sharma, Rajat Singhania, Faridah Mbabaali, Fabian Müller, Javier A Alfaro, Christoph Bock, Daniel D De Carvalho, and Nizar N Batada
Cell Regeneration, 2014, Volume 3, Number 1, Page 3:4
Elena Cardenal-Muñoz, Sonia Arafah, Ana Teresa López-Jiménez, Sébastien Kicka, Alexandra Falaise, Frauke Bach, Olivier Schaad, Jason S. King, Monica Hagedorn, Thierry Soldati, and David M. Lewinsohn
PLOS Pathogens, 2017, Volume 13, Number 4, Page e1006344
AmanPreet Badhwar, Angela Tam, Christian Dansereau, Pierre Orban, Felix Hoffstaedter, and Pierre Bellec
Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 2017, Volume 8, Page 73
Prantik Chatterjee and Nikhil Ranjan Pal
Gene, 2016, Volume 590, Number 2, Page 250
Liang Sun, Yu Yu, Tao Huang, Peng An, Danxia Yu, Zhijie Yu, Huaixing Li, Hongguang Sheng, Lu Cai, Jun Xue, Miao Jing, Yixue Li, Xu Lin, Fudi Wang, and Yiqing Song
PLoS ONE, 2012, Volume 7, Number 6, Page e38845
Concetta Crisafulli, Antonio Drago, Marco Calabrò, Edoardo Spina, and Alessandro Serretti
Progress in Neuro-Psychopharmacology and Biological Psychiatry, 2015, Volume 59, Page 21
Shelbi L Russell, Russell B Corbett-Detig, and Colleen M Cavanaugh
The ISME Journal, 2017, Volume 11, Number 6, Page 1359
A. Smolinska, A. G. L. Bodelier, J. W. Dallinga, A. A. M. Masclee, D. M. Jonkers, F.-J. van Schooten, and M. J. Pierik
Alimentary Pharmacology & Therapeutics, 2017, Volume 45, Number 9, Page 1244
Tyler Funnell, Shinya Tasaki, Arusha Oloumi, Shinsuke Araki, Esther Kong, Damian Yap, Yusuke Nakayama, Christopher S. Hughes, S.-W. Grace Cheng, Hirokazu Tozaki, Misa Iwatani, Satoshi Sasaki, Tomohiro Ohashi, Tohru Miyazaki, Nao Morishita, Daisuke Morishita, Mari Ogasawara-Shimizu, Momoko Ohori, Shoichi Nakao, Masatoshi Karashima, Masaya Sano, Aiko Murai, Toshiyuki Nomura, Noriko Uchiyama, Tomohiro Kawamoto, Ryujiro Hara, Osamu Nakanishi, Karey Shumansky, Jamie Rosner, Adrian Wan, Steven McKinney, Gregg B. Morin, Atsushi Nakanishi, Sohrab Shah, Hiroyoshi Toyoshiba, and Samuel Aparicio
Nature Communications, 2017, Volume 8, Number 1
Peter W. Harrison, Judith E. Mank, and Nina Wedell
Genome Biology and Evolution, 2012, Volume 4, Number 11, Page 1118
Seung Hoan Choi, Adam T. Labadorf, Richard H. Myers, Kathryn L. Lunetta, Josée Dupuis, and Anita L. DeStefano
BMC Bioinformatics, 2017, Volume 18, Number 1
Sjur Reppe, Tonje G. Lien, Yi-Hsiang Hsu, Vigdis T Gautvik, Ole K. Olstad, Rona Yu, Hege G. Bakke, Robert Lyle, Marianne K. Kringen, Ingrid K. Glad, and Kaare M. Gautvik
Epigenetics, 2017, Page 00
Dillon W. Leong, Jasper C. Komen, Chelsee A. Hewitt, Estelle Arnaud, Matthew McKenzie, Belinda Phipson, Melanie Bahlo, Adrienne Laskowski, Sarah A. Kinkel, Gayle M. Davey, William R. Heath, Anne K. Voss, René P. Zahedi, James J. Pitt, Roman Chrast, Albert Sickmann, Michael T. Ryan, Gordon K. Smyth, David R. Thorburn, and Hamish S. Scott
Journal of Biological Chemistry, 2012, Volume 287, Number 24, Page 20652
Nestoras Karathanasis, Ioannis Tsamardinos, Vincenzo Lagani, and Holger Fröhlich
PLOS ONE, 2016, Volume 11, Number 11, Page e0165545
Aarti Gautam, Peter D’Arpa, Duncan E. Donohue, Seid Muhie, Nabarun Chakraborty, Brian T. Luke, Dmitry Grapov, Erica E. Carroll, James L. Meyerhoff, Rasha Hammamieh, Marti Jett, and Alexandra Kavushansky
PLOS ONE, 2015, Volume 10, Number 1, Page e0117092
Nadia Cattane, Alessandra Minelli, Elena Milanesi, Carlo Maj, Stefano Bignotti, Marco Bortolomasi, Luisella Bocchio Chiavetto, and Massimo Gennarelli
PLOS ONE, 2015, Volume 10, Number 2, Page e0116686
Silke Janitza, Ender Celik, and Anne-Laure Boulesteix
Advances in Data Analysis and Classification, 2016
Z Li, K Haynes, D J Pennisi, L K Anderson, X Song, G P Thomas, T Kenna, P Leo, and M A Brown
Genes and Immunity, 2017
Wilson Wen Bin Goh and Limsoon Wong
Journal of Bioinformatics and Computational Biology, 2016, Volume 14, Number 05, Page 1650029
Antonio Drago, Concetta Crisafulli, Antonina Sidoti, Marco Calabrò, and Alessandro Serretti
Journal of Affective Disorders, 2016, Volume 190, Page 429
Calabrò Marco, Drago Antonio, Sidoti Antonina, Serretti Alessandro, and Crisafulli Concetta
Psychiatry Research, 2015, Volume 228, Number 3, Page 945
Scott C. Ritchie, Stephen Watts, Liam G. Fearnley, Kathryn E. Holt, Gad Abraham, and Michael Inouye
Cell Systems, 2016, Volume 3, Number 1, Page 71
Jesse Hemerik and Jelle J. Goeman
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2017
Jiaojiao Jing, Cristian Pattaro, Anselm Hoppmann, Yukinori Okada, Caroline S. Fox, and Anna Köttgen
Kidney International, 2016, Volume 90, Number 4, Page 764
Wenkai Hu, Jiandong Wang, Tongwen Chen, and Sirish L. Shah
Control Engineering Practice, 2017, Volume 64, Page 205
Keedrian I. Olmstead, Michael R. La Frano, Johannes Fahrmann, Dmitry Grapov, Jose A. Viscarra, John W. Newman, Oliver Fiehn, Daniel E. Crocker, Fabian V. Filipp, and Rudy M. Ortiz
Metabolomics, 2017, Volume 13, Number 5
Alok Kumar Dwivedi, Indika Mallawaarachchi, and Luis A. Alvarado
Statistics in Medicine, 2017
Nicholas B. Larson, Shannon McDonnell, Lisa Cannon Albright, Craig Teerlink, Janet Stanford, Elaine A. Ostrander, William B. Isaacs, Jianfeng Xu, Kathleen A. Cooney, Ethan Lange, Johanna Schleutker, John D. Carpten, Isaac Powell, Joan E. Bailey-Wilson, Olivier Cussenot, Geraldine Cancel-Tassin, Graham G. Giles, Robert J. MacInnis, Christiane Maier, Alice S. Whittemore, Chih-Lin Hsieh, Fredrik Wiklund, William J. Catolona, William Foulkes, Diptasri Mandal, Rosalind Eeles, Zsofia Kote-Jarai, Michael J. Ackerman, Timothy M. Olson, Christopher J. Klein, Stephen N. Thibodeau, and Daniel J. Schaid
Genetic Epidemiology, 2017, Volume 41, Number 4, Page 297
Po-Yuan Tung, John D. Blischak, Chiaowen Joyce Hsiao, David A. Knowles, Jonathan E. Burnett, Jonathan K. Pritchard, and Yoav Gilad
Scientific Reports, 2017, Volume 7, Page 39921
Petri Kemppainen, Bernt Rønning, Thomas Kvalnes, Ingerid J. Hagen, Thor Harald Ringsby, Anna M. Billing, Henrik Pärn, Sigbjørn Lien, Arild Husby, Bernt-Erik Saether, and Henrik Jensen
Molecular Ecology Resources, 2017, Volume 17, Number 4, Page 770
Tal Golan, Ido Davidesco, Meir Meshulam, David M Groppe, Pierre Mégevand, Erin M Yeagle, Matthew S Goldfinger, Michal Harel, Lucia Melloni, Charles E Schroeder, Leon Y Deouell, Ashesh D Mehta, and Rafael Malach
eLife, 2016, Volume 5
Timothy West, Simon Farmer, Luc Berthouze, Ashwani Jha, Martijn Beudel, Thomas Foltynie, Patricia Limousin, Ludvic Zrinzo, Peter Brown, and Vladimir Litvak
Frontiers in Human Neuroscience, 2016, Volume 10
Christian Hundt, Andreas Hildebrandt, and Bertil Schmidt
BMC Bioinformatics, 2016, Volume 17, Number 1
Nady El Hajj, Marcus Dittrich, Julia Böck, Theo F. J. Kraus, Indrajit Nanda, Tobias Müller, Larissa Seidmann, Tim Tralau, Danuta Galetzka, Eberhard Schneider, and Thomas Haaf
Epigenetics, 2016, Volume 11, Number 8, Page 563
Nimisha Chaturvedi, Renée X. de Menezes, and Jelle J. Goeman
Biometrical Journal, 2017, Volume 59, Number 1, Page 145
Severin Uebbing, Axel Künstner, Hannu Mäkinen, Niclas Backström, Paulina Bolivar, Reto Burri, Ludovic Dutoit, Carina F. Mugal, Alexander Nater, Bronwen Aken, Paul Flicek, Fergal J. Martin, Stephen M. J. Searle, and Hans Ellegren
Molecular Ecology, 2016, Volume 25, Number 9, Page 2015
Jessica L Larson and Art B Owen
BMC Bioinformatics, 2015, Volume 16, Number 1
Jelle J. Goeman and Aldo Solari
Statistics in Medicine, 2014, Volume 33, Number 11, Page 1946

Comments (0)

Please log in or register to comment.
Log in