Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Statistical Applications in Genetics and Molecular Biology

Editor-in-Chief: Sanguinetti, Guido

6 Issues per year


IMPACT FACTOR 2017: 0.812
5-year IMPACT FACTOR: 1.104

CiteScore 2017: 0.86

SCImago Journal Rank (SJR) 2017: 0.456
Source Normalized Impact per Paper (SNIP) 2017: 0.527

Mathematical Citation Quotient (MCQ) 2017: 0.04

Online
ISSN
1544-6115
See all formats and pricing
More options …
Volume 9, Issue 1

Issues

Volume 10 (2011)

Volume 9 (2010)

Volume 6 (2007)

Volume 5 (2006)

Volume 4 (2005)

Volume 2 (2003)

Volume 1 (2002)

Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn

Belinda Phipson / Gordon K Smyth
Published Online: 2010-10-31 | DOI: https://doi.org/10.2202/1544-6115.1585

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.

Keywords: permutation test; Monte Carlo test; p-values; multiple testing; microarray

About the article

Published Online: 2010-10-31


Citation Information: Statistical Applications in Genetics and Molecular Biology, Volume 9, Issue 1, ISSN (Online) 1544-6115, DOI: https://doi.org/10.2202/1544-6115.1585.

Export Citation

©2011 Walter de Gruyter GmbH & Co. KG, Berlin/Boston.Get Permission

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

[1]
Christian S. Zang, Susanne Jochner-Oette, José Cortés, Anja Rammig, and Annette Menzel
Climate Dynamics, 2018
[2]
Anton Albajes-Eizagirre, Aleix Solanes, Eduard Vieta, and Joaquim Radua
NeuroImage, 2018
[3]
Andreas Marzoll, Tan Saygi, and Hubert R. Dinse
Scientific Reports, 2018, Volume 8, Number 1
[4]
James A. Brissenden, Sean M. Tobyne, David E. Osher, Emily J. Levin, Mark A. Halko, and David C. Somers
Current Biology, 2018
[5]
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Juyoung Lee, Takahide Kagawa, and Yoshifumi Miyazaki
Frontiers in Public Health, 2018, Volume 6
[6]
Matthew M. Parks, Benjamin J. Raphael, and Charles E. Lawrence
BMC Bioinformatics, 2018, Volume 19, Number 1
[7]
Shu Mei Teo, Howard H.F. Tang, Danny Mok, Louise M. Judd, Stephen C. Watts, Kym Pham, Barbara J. Holt, Merci Kusel, Michael Serralha, Niamh Troy, Yury A. Bochkov, Kristine Grindle, Robert F. Lemanske, Sebastian L. Johnston, James E. Gern, Peter D. Sly, Patrick G. Holt, Kathryn E. Holt, and Michael Inouye
Cell Host & Microbe, 2018, Volume 24, Number 3, Page 341
[8]
Daniel Feuerriegel, Hannah A.D. Keage, Bruno Rossion, and Genevieve L. Quek
Biological Psychology, 2018
[9]
Christopher Hübel, Virpi Leppä, Gerome Breen, and Cynthia M. Bulik
International Journal of Eating Disorders, 2018
[10]
Pamela A. Moyer, Margaret S. Boettcher, Jeffery J. McGuire, and John A. Collins
Journal of Geophysical Research: Solid Earth, 2018
[11]
Seung Ah Choi, Jun-Won Yun, Kyeung Min Joo, Ji Yeoun Lee, Pil Ae Kwak, Young Eun Lee, Ji-Ran You, Euna Kwon, Woo Ho Kim, Kyu-Chang Wang, Ji Hoon Phi, Byeong-Cheol Kang, and Seung-Ki Kim
Stem Cells and Development, 2016, Volume 25, Number 12, Page 897
[13]
Kirill V. Nourski, Mitchell Steinschneider, Ariane E. Rhone, Hiroto Kawasaki, Matthew A. Howard, and Matthew I. Banks
NeuroImage, 2018
[14]
Di Wu and Gordon K. Smyth
Nucleic Acids Research, 2012, Volume 40, Number 17, Page e133
[15]
Joseph Cursons, Katherine A. Pillman, Kaitlin G. Scheer, Philip A. Gregory, Momeneh Foroutan, Soroor Hediyeh-Zadeh, John Toubia, Edmund J. Crampin, Gregory J. Goodall, Cameron P. Bracken, and Melissa J. Davis
Cell Systems, 2018
[16]
Koseki J. Kobayashi-Kirschvink, Hidenori Nakaoka, Arisa Oda, Ken-ichiro F. Kamei, Kazuki Nosho, Hiroko Fukushima, Yu Kanesaki, Shunsuke Yajima, Haruhiko Masaki, Kunihiro Ohta, and Yuichi Wakamoto
Cell Systems, 2018
[18]
Anne Steininger, Grit Ebert, Benjamin V. Becker, Chalid Assaf, Markus Möbs, Christian A. Schmidt, Piotr Grabarczyk, Lars R. Jensen, Grzegorz K. Przybylski, Matthias Port, Andreas W. Kuss, and Reinhard Ullmann
Frontiers in Oncology, 2018, Volume 8
[20]
Adrià Tauste Campo, Marina Martinez-Garcia, Verónica Nácher, Rogelio Luna, Ranulfo Romo, and Gustavo Deco
Proceedings of the National Academy of Sciences, 2015, Volume 112, Number 15, Page 4761
[21]
Andrew D. Rouillard, Mark R. Hurle, Pankaj Agarwal, and Edwin Wang
PLOS Computational Biology, 2018, Volume 14, Number 5, Page e1006142
[22]
Aaron T.L. Lun, Davis J. McCarthy, and John C. Marioni
F1000Research, 2016, Volume 5, Page 2122
[23]
Aaron T.L. Lun, Davis J. McCarthy, and John C. Marioni
F1000Research, 2016, Volume 5, Page 2122
[24]
Willem van den Boom, Callie Mao, Rebecca A Schroeder, and David B Dunson
Bioinformatics, 2018
[26]
Matthew M Parks and Ziv Bar-Joseph
Bioinformatics, 2018, Volume 34, Number 6, Page 971
[27]
Bradley P. Harris, Charles F. Adams, and Kevin D. E. Stokesbury
Ecosphere, 2018, Volume 9, Number 3, Page e02133
[28]
Xiang Zhang, Adil Mardinoglu, Leo A. B. Joosten, Jan A. Kuivenhoven, Yang Li, Mihai G. Netea, and Albert K. Groen
Frontiers in Physiology, 2018, Volume 9
[29]
Jarrod Shilts, Guanhua Chen, and Jacob J. Hughey
PeerJ, 2018, Volume 6, Page e4327
[30]
Andrew S. Blaeser, Barry W. Connors, and Arto V. Nurmikko
Journal of Neurophysiology, 2017, Volume 117, Number 4, Page 1581
[31]
[32]
Ezequiel Mikulan, Eugenia Hesse, Lucas Sedeño, Tristán Bekinschtein, Mariano Sigman, María del Carmen García, Walter Silva, Carlos Ciraolo, Adolfo M. García, and Agustín Ibáñez
NeuroImage, 2017
[33]
Jesse Hemerik and Jelle Goeman
TEST, 2017
[34]
Hao Tam Ho, Johahn Leung, David C. Burr, David Alais, and Maria Concetta Morrone
Current Biology, 2017
[35]
Sarah Fleischer, Stefanie Ries, Ping Shen, Alix Lheritier, Frédéric Cazals, Gerd. R. Burmester, Thomas Dörner, and Simon Fillatreau
European Journal of Immunology, 2017
[36]
Edina Nemesházi, Krisztián Szabó, Zoltán Horváth, and Szilvia Kövér
Journal of Ornithology, 2017
[37]
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Takahide Kagawa, and Yoshifumi Miyazaki
International Journal of Environmental Research and Public Health, 2017, Volume 14, Number 9, Page 987
[38]
Antonio Drago, Concetta Crisafulli, Marco Calabrò, and Alessandro Serretti
Journal of Affective Disorders, 2015, Volume 179, Page 88
[39]
Hiromitsu Kobayashi, Chorong Song, Harumi Ikei, Bum-Jin Park, Juyoung Lee, Takahide Kagawa, and Yoshifumi Miyazaki
International Journal of Environmental Research and Public Health, 2017, Volume 14, Number 8, Page 931
[40]
Antonio Drago, Barbara Monti, Diana De Ronchi, and Alessandro Serretti
Psychiatry Investigation, 2015, Volume 12, Number 1, Page 118
[41]
Christopher Minas and Giovanni Montana
Statistical Analysis and Data Mining: The ASA Data Science Journal, 2014, Volume 7, Number 6, Page 450
[42]
Dennis Goldfarb, Bridgid E. Hast, Wei Wang, and Michael B. Major
Journal of Proteome Research, 2014, Volume 13, Number 12, Page 5944
[43]
Daniel D. Kinnamon and Eden R. Martin
Genetic Epidemiology, 2014, Volume 38, Number 4, Page 325
[44]
Thibault Gayet, Sébastien Devillard, Marlène Gamelon, Serge Brandt, Ludovic Say, and Eric Baubet
Evolution, 2016, Volume 70, Number 6, Page 1386
[45]
Anderson M. Winkler, Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols
NeuroImage, 2014, Volume 92, Page 381
[46]
Christopher Minas, Edward Curry, and Giovanni Montana
Bioinformatics, 2013, Volume 29, Number 20, Page 2555
[47]
George C. Tseng, Debashis Ghosh, and Eleanor Feingold
Nucleic Acids Research, 2012, Volume 40, Number 9, Page 3785
[48]
Pashupati Mishra, Petri Törönen, Yrjö Leino, and Liisa Holm
Bioinformatics, 2014, Volume 30, Number 19, Page 2747
[49]
Jonas Paulsen, Tonje G. Lien, Geir Kjetil Sandve, Lars Holden, Ørnulf Borgan, Ingrid K. Glad, and Eivind Hovig
Nucleic Acids Research, 2013, Volume 41, Number 10, Page 5164
[50]
Aline C Planello, Junfeng Ji, Vivek Sharma, Rajat Singhania, Faridah Mbabaali, Fabian Müller, Javier A Alfaro, Christoph Bock, Daniel D De Carvalho, and Nizar N Batada
Cell Regeneration, 2014, Volume 3, Number 1, Page 3:4
[51]
Elena Cardenal-Muñoz, Sonia Arafah, Ana Teresa López-Jiménez, Sébastien Kicka, Alexandra Falaise, Frauke Bach, Olivier Schaad, Jason S. King, Monica Hagedorn, Thierry Soldati, and David M. Lewinsohn
PLOS Pathogens, 2017, Volume 13, Number 4, Page e1006344
[52]
AmanPreet Badhwar, Angela Tam, Christian Dansereau, Pierre Orban, Felix Hoffstaedter, and Pierre Bellec
Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 2017, Volume 8, Page 73
[53]
Prantik Chatterjee and Nikhil Ranjan Pal
Gene, 2016, Volume 590, Number 2, Page 250
[54]
Liang Sun, Yu Yu, Tao Huang, Peng An, Danxia Yu, Zhijie Yu, Huaixing Li, Hongguang Sheng, Lu Cai, Jun Xue, Miao Jing, Yixue Li, Xu Lin, Fudi Wang, and Yiqing Song
PLoS ONE, 2012, Volume 7, Number 6, Page e38845
[55]
Concetta Crisafulli, Antonio Drago, Marco Calabrò, Edoardo Spina, and Alessandro Serretti
Progress in Neuro-Psychopharmacology and Biological Psychiatry, 2015, Volume 59, Page 21
[56]
Shelbi L Russell, Russell B Corbett-Detig, and Colleen M Cavanaugh
The ISME Journal, 2017, Volume 11, Number 6, Page 1359
[57]
A. Smolinska, A. G. L. Bodelier, J. W. Dallinga, A. A. M. Masclee, D. M. Jonkers, F.-J. van Schooten, and M. J. Pierik
Alimentary Pharmacology & Therapeutics, 2017, Volume 45, Number 9, Page 1244
[58]
Tyler Funnell, Shinya Tasaki, Arusha Oloumi, Shinsuke Araki, Esther Kong, Damian Yap, Yusuke Nakayama, Christopher S. Hughes, S.-W. Grace Cheng, Hirokazu Tozaki, Misa Iwatani, Satoshi Sasaki, Tomohiro Ohashi, Tohru Miyazaki, Nao Morishita, Daisuke Morishita, Mari Ogasawara-Shimizu, Momoko Ohori, Shoichi Nakao, Masatoshi Karashima, Masaya Sano, Aiko Murai, Toshiyuki Nomura, Noriko Uchiyama, Tomohiro Kawamoto, Ryujiro Hara, Osamu Nakanishi, Karey Shumansky, Jamie Rosner, Adrian Wan, Steven McKinney, Gregg B. Morin, Atsushi Nakanishi, Sohrab Shah, Hiroyoshi Toyoshiba, and Samuel Aparicio
Nature Communications, 2017, Volume 8, Number 1
[59]
Peter W. Harrison, Judith E. Mank, and Nina Wedell
Genome Biology and Evolution, 2012, Volume 4, Number 11, Page 1118
[60]
Seung Hoan Choi, Adam T. Labadorf, Richard H. Myers, Kathryn L. Lunetta, Josée Dupuis, and Anita L. DeStefano
BMC Bioinformatics, 2017, Volume 18, Number 1
[61]
Sjur Reppe, Tonje G. Lien, Yi-Hsiang Hsu, Vigdis T Gautvik, Ole K. Olstad, Rona Yu, Hege G. Bakke, Robert Lyle, Marianne K. Kringen, Ingrid K. Glad, and Kaare M. Gautvik
Epigenetics, 2017, Page 00
[62]
Dillon W. Leong, Jasper C. Komen, Chelsee A. Hewitt, Estelle Arnaud, Matthew McKenzie, Belinda Phipson, Melanie Bahlo, Adrienne Laskowski, Sarah A. Kinkel, Gayle M. Davey, William R. Heath, Anne K. Voss, René P. Zahedi, James J. Pitt, Roman Chrast, Albert Sickmann, Michael T. Ryan, Gordon K. Smyth, David R. Thorburn, and Hamish S. Scott
Journal of Biological Chemistry, 2012, Volume 287, Number 24, Page 20652
[63]
Nestoras Karathanasis, Ioannis Tsamardinos, Vincenzo Lagani, and Holger Fröhlich
PLOS ONE, 2016, Volume 11, Number 11, Page e0165545
[64]
Aarti Gautam, Peter D’Arpa, Duncan E. Donohue, Seid Muhie, Nabarun Chakraborty, Brian T. Luke, Dmitry Grapov, Erica E. Carroll, James L. Meyerhoff, Rasha Hammamieh, Marti Jett, and Alexandra Kavushansky
PLOS ONE, 2015, Volume 10, Number 1, Page e0117092
[65]
Nadia Cattane, Alessandra Minelli, Elena Milanesi, Carlo Maj, Stefano Bignotti, Marco Bortolomasi, Luisella Bocchio Chiavetto, and Massimo Gennarelli
PLOS ONE, 2015, Volume 10, Number 2, Page e0116686
[66]
Silke Janitza, Ender Celik, and Anne-Laure Boulesteix
Advances in Data Analysis and Classification, 2016
[67]
Z Li, K Haynes, D J Pennisi, L K Anderson, X Song, G P Thomas, T Kenna, P Leo, and M A Brown
Genes and Immunity, 2017
[68]
Wilson Wen Bin Goh and Limsoon Wong
Journal of Bioinformatics and Computational Biology, 2016, Volume 14, Number 05, Page 1650029
[69]
Antonio Drago, Concetta Crisafulli, Antonina Sidoti, Marco Calabrò, and Alessandro Serretti
Journal of Affective Disorders, 2016, Volume 190, Page 429
[70]
Calabrò Marco, Drago Antonio, Sidoti Antonina, Serretti Alessandro, and Crisafulli Concetta
Psychiatry Research, 2015, Volume 228, Number 3, Page 945
[71]
Scott C. Ritchie, Stephen Watts, Liam G. Fearnley, Kathryn E. Holt, Gad Abraham, and Michael Inouye
Cell Systems, 2016, Volume 3, Number 1, Page 71
[72]
Jesse Hemerik and Jelle J. Goeman
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2017
[74]
Jiaojiao Jing, Cristian Pattaro, Anselm Hoppmann, Yukinori Okada, Caroline S. Fox, and Anna Köttgen
Kidney International, 2016, Volume 90, Number 4, Page 764
[75]
Wenkai Hu, Jiandong Wang, Tongwen Chen, and Sirish L. Shah
Control Engineering Practice, 2017, Volume 64, Page 205
[76]
Keedrian I. Olmstead, Michael R. La Frano, Johannes Fahrmann, Dmitry Grapov, Jose A. Viscarra, John W. Newman, Oliver Fiehn, Daniel E. Crocker, Fabian V. Filipp, and Rudy M. Ortiz
Metabolomics, 2017, Volume 13, Number 5
[77]
Alok Kumar Dwivedi, Indika Mallawaarachchi, and Luis A. Alvarado
Statistics in Medicine, 2017
[78]
Nicholas B. Larson, Shannon McDonnell, Lisa Cannon Albright, Craig Teerlink, Janet Stanford, Elaine A. Ostrander, William B. Isaacs, Jianfeng Xu, Kathleen A. Cooney, Ethan Lange, Johanna Schleutker, John D. Carpten, Isaac Powell, Joan E. Bailey-Wilson, Olivier Cussenot, Geraldine Cancel-Tassin, Graham G. Giles, Robert J. MacInnis, Christiane Maier, Alice S. Whittemore, Chih-Lin Hsieh, Fredrik Wiklund, William J. Catolona, William Foulkes, Diptasri Mandal, Rosalind Eeles, Zsofia Kote-Jarai, Michael J. Ackerman, Timothy M. Olson, Christopher J. Klein, Stephen N. Thibodeau, and Daniel J. Schaid
Genetic Epidemiology, 2017, Volume 41, Number 4, Page 297
[79]
Po-Yuan Tung, John D. Blischak, Chiaowen Joyce Hsiao, David A. Knowles, Jonathan E. Burnett, Jonathan K. Pritchard, and Yoav Gilad
Scientific Reports, 2017, Volume 7, Page 39921
[80]
Petri Kemppainen, Bernt Rønning, Thomas Kvalnes, Ingerid J. Hagen, Thor Harald Ringsby, Anna M. Billing, Henrik Pärn, Sigbjørn Lien, Arild Husby, Bernt-Erik Saether, and Henrik Jensen
Molecular Ecology Resources, 2017, Volume 17, Number 4, Page 770
[81]
Tal Golan, Ido Davidesco, Meir Meshulam, David M Groppe, Pierre Mégevand, Erin M Yeagle, Matthew S Goldfinger, Michal Harel, Lucia Melloni, Charles E Schroeder, Leon Y Deouell, Ashesh D Mehta, and Rafael Malach
eLife, 2016, Volume 5
[82]
Timothy West, Simon Farmer, Luc Berthouze, Ashwani Jha, Martijn Beudel, Thomas Foltynie, Patricia Limousin, Ludvic Zrinzo, Peter Brown, and Vladimir Litvak
Frontiers in Human Neuroscience, 2016, Volume 10
[83]
Christian Hundt, Andreas Hildebrandt, and Bertil Schmidt
BMC Bioinformatics, 2016, Volume 17, Number 1
[84]
Nady El Hajj, Marcus Dittrich, Julia Böck, Theo F. J. Kraus, Indrajit Nanda, Tobias Müller, Larissa Seidmann, Tim Tralau, Danuta Galetzka, Eberhard Schneider, and Thomas Haaf
Epigenetics, 2016, Volume 11, Number 8, Page 563
[85]
Nimisha Chaturvedi, Renée X. de Menezes, and Jelle J. Goeman
Biometrical Journal, 2017, Volume 59, Number 1, Page 145
[86]
Severin Uebbing, Axel Künstner, Hannu Mäkinen, Niclas Backström, Paulina Bolivar, Reto Burri, Ludovic Dutoit, Carina F. Mugal, Alexander Nater, Bronwen Aken, Paul Flicek, Fergal J. Martin, Stephen M. J. Searle, and Hans Ellegren
Molecular Ecology, 2016, Volume 25, Number 9, Page 2015
[87]
Jessica L Larson and Art B Owen
BMC Bioinformatics, 2015, Volume 16, Number 1
[88]
Jelle J. Goeman and Aldo Solari
Statistics in Medicine, 2014, Volume 33, Number 11, Page 1946

Comments (0)

Please log in or register to comment.
Log in