Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter November 6, 2015

Substantive Importance and the Veil of Statistical Significance

  • Kelly McCaskey EMAIL logo and Carlisle Rainey


Political science is gradually moving away from an exclusive focus on statistical significance and toward an emphasis on the magnitude and importance of effects. While we welcome this change, we argue that the current practice of “magnitude-and-significance,” in which researchers only interpret the magnitude of a statistically significant point estimate, barely improves the much-maligned “sign-and-significance” approach, in which researchers focus only on the statistical significance of an estimate. This exclusive focus on the point estimate hides the uncertainty behind a veil of statistical significance. Instead, we encourage researchers to explicitly account for uncertainty by interpreting the range of values contained in the confidence interval. Especially when making judgments about the importance of estimated effects, we advise researchers to make empirical claims if and only if those claims hold for the entire confidence interval.

Corresponding author: Kelly McCaskey, PhD Student, Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA, e-mail:


We thank Lisa Hultman, Cindy Kam, Jacob Kathman, and Megan Shannon, and Elizabeth Zechmeister for making their data available to us. The analyses presented here were conducted with R 3.1.0. All data and computer code necessary to reproduce our paper and results are available at


Achen, Christopher H. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.10.4135/9781412984560Search in Google Scholar

Berry, William D., Jacqueline H. R. DeMeritt and Justin Esarey (2010) “Testing for Interaction in Binary Logit and Probit Models: Is a Product Term Essential,” American Journal of Political Science, 54(1):105–119.10.1111/j.1540-5907.2009.00429.xSearch in Google Scholar

Casella, George and Roger L. Berger (2002) Statistical Inference. 2nd ed. Pacific Grove, California: Duxbury.Search in Google Scholar

Cohen, Jacob (1990) “Things I Have Learned (So Far),” American Psychologist, 45(12):1304–1312.10.1037/0003-066X.45.12.1304Search in Google Scholar

Cohen, Jacob (1992) “A Power Primer,” Psychological Bulletin, 112(1):115–159.10.1037/0033-2909.112.1.155Search in Google Scholar

Esarey, Justin and Ahra Wu (2014) The Fault in our Stars: Measuring and Correcting Significance Bias in Political Science. Working paper. Copy at in Google Scholar

Esarey, Justin and Nathan Danneman (2015) “A Quantitative Method for Substantive Robustness Assessment,” Political Science Research and Methods, 3(1):95–111.10.1017/psrm.2014.14Search in Google Scholar

Francis, Gregory (2013) “Replication, Statistical Consistency, and Publication Bias (with Discussion),” Journal of Mathematical Psychology, 57(5):153–169.10.1016/ in Google Scholar

Gelman, Andrew and Eric Loken (2014) “Ethics and Statistics: The AAA Tranche of Subprime Science,” Chance, 27(1):51–56.10.1080/09332480.2014.890872Search in Google Scholar

Gerber, Elisabeth R. and Daniel J. Hopkins (2011) “When Mayors Matter: Estimating the Impact of Mayoral Partisanship on City Policy,” American Journal of Political Science, 55(2):326–339.10.1111/j.1540-5907.2010.00499.xSearch in Google Scholar

Gerber, Alan and Neil Malhotra (2008) “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals,” Quarterly Journal of Political Science, 3(3):313–326.10.1561/100.00008024Search in Google Scholar

Gill, Jeff (1999) “Null Hypothesis Significance Testing,” Political Research Quarterly, 52(3):647–674.10.1177/106591299905200309Search in Google Scholar

Glass, Gene V. (1976) “Primary, Secondary, and Meta-Analysis of Research,” Educational Researcher, 5(10):3–8.10.3102/0013189X005010003Search in Google Scholar

Gross, Justin H. (2015) “Testing What Matters (If You Must Test At All): A Context-Driven Approach to Substantive and Statistical Significance,” American Journal of Political Science, 59(3):775–788.10.1111/ajps.12149Search in Google Scholar

Hanmer, Michael J. and Kerem Ozan Kalkan (2013) “Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models,” American Journal of Political Science, 57(1):263–277.10.1111/j.1540-5907.2012.00602.xSearch in Google Scholar

Hetherington, Marc and Elizabeth Suhay (2011) “Authoritarianism, Threat, and Americans’ Support for the War on Terror,” American Journal of Political Science, 55(3):546–560.10.1111/j.1540-5907.2011.00514.xSearch in Google Scholar

Hill, Jr., Daniel W. and Zachary M. Jones (2014) “An Empirical Evaluation of Explanations for State Repression,” American Political Science Review, 108(3):1–27.10.1017/S0003055414000306Search in Google Scholar

Hultman, Lisa, Jacob Kathman and Megan Shannon (2013) “United Nations Peacekeeping and Civilian Protection in Civil War,” American Journal of Political Science, 57(4):875–891.10.1111/ajps.12036Search in Google Scholar

Imai, Kosuke, Gary King and Olivia Lau (2008) “Toward a Common Framework for Statistical Analysis and Development,” Journal of Computational and Graphical Statistics, 17(4):892–913.10.1198/106186008X384898Search in Google Scholar

Kam, Cindy D. and Carl L. Palmer (2008) “Reconsidering the Effects of Education on Political Participation,” Journal of Politics, 70(2):612–631.10.1017/S0022381608080651Search in Google Scholar

Kam, Cindy D. and Elizabeth J. Zechmeister (2013) “Name Recognition and Candidate Support,” American Journal of Political Science, 57(4):971–986.10.1111/ajps.12034Search in Google Scholar

King, Gary and Langche Zeng (2001) “Logistic Regression in Rare Events Data,” Political Analysis, 9(2):137–163.10.1093/oxfordjournals.pan.a004868Search in Google Scholar

King, Gary, Michael Tomz and Jason Wittenberg (2000) “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science, 44(2):341–355.10.2307/2669316Search in Google Scholar

Kirk, Roger E. (1996) “Practice Signifance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56(5):746–759.10.1177/0013164496056005002Search in Google Scholar

Rainey, Carlisle (2014) “Arguing for a Negligible Effect,” American Journal of Political Science, 58(4):1083–1091.10.1111/ajps.12102Search in Google Scholar

Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2011) “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22(11):1359–1366.10.1177/0956797611417632Search in Google Scholar

Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2014) “P-Curve: A Key to the File Drawer,” Journal of Experimental Psychology: General, 143(2):534–547.10.1037/a0033242Search in Google Scholar

Thompson, Bruce (2001) “Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments to Move the Field,” Journal of Experimental Education, 70(1):80–93.10.1080/00220970109599499Search in Google Scholar

Thompson, Bruce (2002) “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes,” Educational Researcher, 31(3):24–31.10.3102/0013189X031003025Search in Google Scholar

Tomz, Michael R. and Jessica L. P. Weeks (2013) “Public Opinion and the Democratic Peace,” American Political Science Review, 107(4):849–865.10.1017/S0003055413000488Search in Google Scholar

Tomz, Michael, Jason Wittenberg and Gary King (2003) “Clarify: Software for Interpreting and Presenting Statistical Results,” Journal of Statistical Software, 8(1):1–30.10.18637/jss.v008.i01Search in Google Scholar

Tukey, John W. (1962) “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33(1):1–67.10.1214/aoms/1177704711Search in Google Scholar

Tukey, John W. (1991) “The Philosophy of Multiple Comparisons,” Statistical Science, 6(1):100–116.10.1214/ss/1177011945Search in Google Scholar

Yates, F. (1951) “The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics,” American Statistical Association Journal, 46(253):19–34.10.2307/2280090Search in Google Scholar

Published Online: 2015-11-6
Published in Print: 2015-12-1

©2015 by De Gruyter

Downloaded on 23.3.2023 from
Scroll Up Arrow