Jump to ContentJump to Main Navigation
Show Summary Details

Statistics, Politics and Policy

Editor-in-Chief: Wagschal, Uwe

2 Issues per year

Online
ISSN
2151-7509
See all formats and pricing

Substantive Importance and the Veil of Statistical Significance

Kelly McCaskey
  • Corresponding author
  • Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA
  • Email:
/ Carlisle Rainey
  • Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA
Published Online: 2015-11-06 | DOI: https://doi.org/10.1515/spp-2015-0001

Abstract

Political science is gradually moving away from an exclusive focus on statistical significance and toward an emphasis on the magnitude and importance of effects. While we welcome this change, we argue that the current practice of “magnitude-and-significance,” in which researchers only interpret the magnitude of a statistically significant point estimate, barely improves the much-maligned “sign-and-significance” approach, in which researchers focus only on the statistical significance of an estimate. This exclusive focus on the point estimate hides the uncertainty behind a veil of statistical significance. Instead, we encourage researchers to explicitly account for uncertainty by interpreting the range of values contained in the confidence interval. Especially when making judgments about the importance of estimated effects, we advise researchers to make empirical claims if and only if those claims hold for the entire confidence interval.

References

  • Achen, Christopher H. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.

  • Berry, William D., Jacqueline H. R. DeMeritt and Justin Esarey (2010) “Testing for Interaction in Binary Logit and Probit Models: Is a Product Term Essential,” American Journal of Political Science, 54(1):105–119. [Crossref] [Web of Science]

  • Casella, George and Roger L. Berger (2002) Statistical Inference. 2nd ed. Pacific Grove, California: Duxbury.

  • Cohen, Jacob (1990) “Things I Have Learned (So Far),” American Psychologist, 45(12):1304–1312. [Crossref]

  • Cohen, Jacob (1992) “A Power Primer,” Psychological Bulletin, 112(1):115–159. [Crossref]

  • Esarey, Justin and Ahra Wu (2014) The Fault in our Stars: Measuring and Correcting Significance Bias in Political Science. Working paper. Copy at jee3.web.rice.edu/significancebias.pdf.

  • Esarey, Justin and Nathan Danneman (2015) “A Quantitative Method for Substantive Robustness Assessment,” Political Science Research and Methods, 3(1):95–111. [Crossref]

  • Francis, Gregory (2013) “Replication, Statistical Consistency, and Publication Bias (with Discussion),” Journal of Mathematical Psychology, 57(5):153–169. [Crossref] [Web of Science]

  • Gelman, Andrew and Eric Loken (2014) “Ethics and Statistics: The AAA Tranche of Subprime Science,” Chance, 27(1):51–56. [Crossref]

  • Gerber, Elisabeth R. and Daniel J. Hopkins (2011) “When Mayors Matter: Estimating the Impact of Mayoral Partisanship on City Policy,” American Journal of Political Science, 55(2):326–339. [Crossref] [Web of Science]

  • Gerber, Alan and Neil Malhotra (2008) “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals,” Quarterly Journal of Political Science, 3(3):313–326. [Web of Science] [Crossref]

  • Gill, Jeff (1999) “Null Hypothesis Significance Testing,” Political Research Quarterly, 52(3):647–674. [Crossref]

  • Glass, Gene V. (1976) “Primary, Secondary, and Meta-Analysis of Research,” Educational Researcher, 5(10):3–8. [Crossref]

  • Gross, Justin H. (2015) “Testing What Matters (If You Must Test At All): A Context-Driven Approach to Substantive and Statistical Significance,” American Journal of Political Science, 59(3):775–788. [Crossref]

  • Hanmer, Michael J. and Kerem Ozan Kalkan (2013) “Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models,” American Journal of Political Science, 57(1):263–277. [Web of Science] [Crossref]

  • Hetherington, Marc and Elizabeth Suhay (2011) “Authoritarianism, Threat, and Americans’ Support for the War on Terror,” American Journal of Political Science, 55(3):546–560. [Web of Science] [Crossref]

  • Hill, Jr., Daniel W. and Zachary M. Jones (2014) “An Empirical Evaluation of Explanations for State Repression,” American Political Science Review, 108(3):1–27. [Crossref] [Web of Science]

  • Hultman, Lisa, Jacob Kathman and Megan Shannon (2013) “United Nations Peacekeeping and Civilian Protection in Civil War,” American Journal of Political Science, 57(4):875–891. [Web of Science]

  • Imai, Kosuke, Gary King and Olivia Lau (2008) “Toward a Common Framework for Statistical Analysis and Development,” Journal of Computational and Graphical Statistics, 17(4):892–913. [Web of Science] [Crossref]

  • Kam, Cindy D. and Carl L. Palmer (2008) “Reconsidering the Effects of Education on Political Participation,” Journal of Politics, 70(2):612–631. [Web of Science] [Crossref]

  • Kam, Cindy D. and Elizabeth J. Zechmeister (2013) “Name Recognition and Candidate Support,” American Journal of Political Science, 57(4):971–986. [Web of Science]

  • King, Gary and Langche Zeng (2001) “Logistic Regression in Rare Events Data,” Political Analysis, 9(2):137–163. [Crossref]

  • King, Gary, Michael Tomz and Jason Wittenberg (2000) “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science, 44(2):341–355. [Crossref]

  • Kirk, Roger E. (1996) “Practice Signifance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56(5):746–759. [Crossref]

  • Rainey, Carlisle (2014) “Arguing for a Negligible Effect,” American Journal of Political Science, 58(4):1083–1091. [Crossref]

  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2011) “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22(11):1359–1366. [Web of Science] [Crossref]

  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2014) “P-Curve: A Key to the File Drawer,” Journal of Experimental Psychology: General, 143(2):534–547. [Crossref] [Web of Science]

  • Thompson, Bruce (2001) “Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments to Move the Field,” Journal of Experimental Education, 70(1):80–93. [Crossref]

  • Thompson, Bruce (2002) “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes,” Educational Researcher, 31(3):24–31.

  • Tomz, Michael R. and Jessica L. P. Weeks (2013) “Public Opinion and the Democratic Peace,” American Political Science Review, 107(4):849–865. [Web of Science] [Crossref]

  • Tomz, Michael, Jason Wittenberg and Gary King (2003) “Clarify: Software for Interpreting and Presenting Statistical Results,” Journal of Statistical Software, 8(1):1–30. [Crossref]

  • Tukey, John W. (1962) “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33(1):1–67. [Crossref]

  • Tukey, John W. (1991) “The Philosophy of Multiple Comparisons,” Statistical Science, 6(1):100–116. [Crossref]

  • Yates, F. (1951) “The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics,” American Statistical Association Journal, 46(253):19–34.

About the article

Corresponding author: Kelly McCaskey, PhD Student, Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA, e-mail:


Published Online: 2015-11-06

Published in Print: 2015-12-01


Citation Information: Statistics, Politics and Policy, ISSN (Online) 2151-7509, ISSN (Print) 2194-6299, DOI: https://doi.org/10.1515/spp-2015-0001. Export Citation

Comments (1)

Please log in or register to comment.
Log in
  • Confidence intervals that don't cover the null value suffer from problems just as significance values do. See my paper Shaffer, J.P. (2004), Confidence intervals on subsets may be misleading, Journal of Modern Applied Statistical Methods, 3, 261-270, plus the errata (2006), 5, 281.

    posted by: Juliet Shaffer on 2015-12-05 06:53 PM (Europe/Berlin)