Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Statistics, Politics and Policy

Editor-in-Chief: Wagschal, Uwe

2 Issues per year

Online
ISSN
2151-7509
See all formats and pricing
More options …

Substantive Importance and the Veil of Statistical Significance

Kelly McCaskey
  • Corresponding author
  • Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Carlisle Rainey
  • Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2015-11-06 | DOI: https://doi.org/10.1515/spp-2015-0001

Abstract

Political science is gradually moving away from an exclusive focus on statistical significance and toward an emphasis on the magnitude and importance of effects. While we welcome this change, we argue that the current practice of “magnitude-and-significance,” in which researchers only interpret the magnitude of a statistically significant point estimate, barely improves the much-maligned “sign-and-significance” approach, in which researchers focus only on the statistical significance of an estimate. This exclusive focus on the point estimate hides the uncertainty behind a veil of statistical significance. Instead, we encourage researchers to explicitly account for uncertainty by interpreting the range of values contained in the confidence interval. Especially when making judgments about the importance of estimated effects, we advise researchers to make empirical claims if and only if those claims hold for the entire confidence interval.

References

  • Achen, Christopher H. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.Google Scholar

  • Berry, William D., Jacqueline H. R. DeMeritt and Justin Esarey (2010) “Testing for Interaction in Binary Logit and Probit Models: Is a Product Term Essential,” American Journal of Political Science, 54(1):105–119.CrossrefWeb of ScienceGoogle Scholar

  • Casella, George and Roger L. Berger (2002) Statistical Inference. 2nd ed. Pacific Grove, California: Duxbury.Google Scholar

  • Cohen, Jacob (1990) “Things I Have Learned (So Far),” American Psychologist, 45(12):1304–1312.CrossrefGoogle Scholar

  • Cohen, Jacob (1992) “A Power Primer,” Psychological Bulletin, 112(1):115–159.CrossrefGoogle Scholar

  • Esarey, Justin and Ahra Wu (2014) The Fault in our Stars: Measuring and Correcting Significance Bias in Political Science. Working paper. Copy at jee3.web.rice.edu/significancebias.pdf.Google Scholar

  • Esarey, Justin and Nathan Danneman (2015) “A Quantitative Method for Substantive Robustness Assessment,” Political Science Research and Methods, 3(1):95–111.CrossrefGoogle Scholar

  • Francis, Gregory (2013) “Replication, Statistical Consistency, and Publication Bias (with Discussion),” Journal of Mathematical Psychology, 57(5):153–169.CrossrefWeb of ScienceGoogle Scholar

  • Gelman, Andrew and Eric Loken (2014) “Ethics and Statistics: The AAA Tranche of Subprime Science,” Chance, 27(1):51–56.CrossrefGoogle Scholar

  • Gerber, Elisabeth R. and Daniel J. Hopkins (2011) “When Mayors Matter: Estimating the Impact of Mayoral Partisanship on City Policy,” American Journal of Political Science, 55(2):326–339.CrossrefWeb of ScienceGoogle Scholar

  • Gerber, Alan and Neil Malhotra (2008) “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals,” Quarterly Journal of Political Science, 3(3):313–326.Web of ScienceCrossrefGoogle Scholar

  • Gill, Jeff (1999) “Null Hypothesis Significance Testing,” Political Research Quarterly, 52(3):647–674.CrossrefGoogle Scholar

  • Glass, Gene V. (1976) “Primary, Secondary, and Meta-Analysis of Research,” Educational Researcher, 5(10):3–8.CrossrefGoogle Scholar

  • Gross, Justin H. (2015) “Testing What Matters (If You Must Test At All): A Context-Driven Approach to Substantive and Statistical Significance,” American Journal of Political Science, 59(3):775–788.CrossrefGoogle Scholar

  • Hanmer, Michael J. and Kerem Ozan Kalkan (2013) “Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models,” American Journal of Political Science, 57(1):263–277.Web of ScienceCrossrefGoogle Scholar

  • Hetherington, Marc and Elizabeth Suhay (2011) “Authoritarianism, Threat, and Americans’ Support for the War on Terror,” American Journal of Political Science, 55(3):546–560.Web of ScienceCrossrefGoogle Scholar

  • Hill, Jr., Daniel W. and Zachary M. Jones (2014) “An Empirical Evaluation of Explanations for State Repression,” American Political Science Review, 108(3):1–27.CrossrefWeb of ScienceGoogle Scholar

  • Hultman, Lisa, Jacob Kathman and Megan Shannon (2013) “United Nations Peacekeeping and Civilian Protection in Civil War,” American Journal of Political Science, 57(4):875–891.Web of ScienceGoogle Scholar

  • Imai, Kosuke, Gary King and Olivia Lau (2008) “Toward a Common Framework for Statistical Analysis and Development,” Journal of Computational and Graphical Statistics, 17(4):892–913.Web of ScienceCrossrefGoogle Scholar

  • Kam, Cindy D. and Carl L. Palmer (2008) “Reconsidering the Effects of Education on Political Participation,” Journal of Politics, 70(2):612–631.Web of ScienceCrossrefGoogle Scholar

  • Kam, Cindy D. and Elizabeth J. Zechmeister (2013) “Name Recognition and Candidate Support,” American Journal of Political Science, 57(4):971–986.Web of ScienceGoogle Scholar

  • King, Gary and Langche Zeng (2001) “Logistic Regression in Rare Events Data,” Political Analysis, 9(2):137–163.CrossrefGoogle Scholar

  • King, Gary, Michael Tomz and Jason Wittenberg (2000) “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science, 44(2):341–355.CrossrefGoogle Scholar

  • Kirk, Roger E. (1996) “Practice Signifance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56(5):746–759.CrossrefGoogle Scholar

  • Rainey, Carlisle (2014) “Arguing for a Negligible Effect,” American Journal of Political Science, 58(4):1083–1091.CrossrefGoogle Scholar

  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2011) “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22(11):1359–1366.Web of ScienceCrossrefGoogle Scholar

  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2014) “P-Curve: A Key to the File Drawer,” Journal of Experimental Psychology: General, 143(2):534–547.CrossrefWeb of ScienceGoogle Scholar

  • Thompson, Bruce (2001) “Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments to Move the Field,” Journal of Experimental Education, 70(1):80–93.CrossrefGoogle Scholar

  • Thompson, Bruce (2002) “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes,” Educational Researcher, 31(3):24–31.Google Scholar

  • Tomz, Michael R. and Jessica L. P. Weeks (2013) “Public Opinion and the Democratic Peace,” American Political Science Review, 107(4):849–865.Web of ScienceCrossrefGoogle Scholar

  • Tomz, Michael, Jason Wittenberg and Gary King (2003) “Clarify: Software for Interpreting and Presenting Statistical Results,” Journal of Statistical Software, 8(1):1–30.CrossrefGoogle Scholar

  • Tukey, John W. (1962) “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33(1):1–67.CrossrefGoogle Scholar

  • Tukey, John W. (1991) “The Philosophy of Multiple Comparisons,” Statistical Science, 6(1):100–116.CrossrefGoogle Scholar

  • Yates, F. (1951) “The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics,” American Statistical Association Journal, 46(253):19–34.Google Scholar

About the article

Corresponding author: Kelly McCaskey, PhD Student, Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA, e-mail:


Published Online: 2015-11-06

Published in Print: 2015-12-01


Citation Information: Statistics, Politics and Policy, ISSN (Online) 2151-7509, ISSN (Print) 2194-6299, DOI: https://doi.org/10.1515/spp-2015-0001.

Export Citation

©2015 by De Gruyter. Copyright Clearance Center

Comments (1)

Please log in or register to comment.
Log in
  • Confidence intervals that don't cover the null value suffer from problems just as significance values do. See my paper Shaffer, J.P. (2004), Confidence intervals on subsets may be misleading, Journal of Modern Applied Statistical Methods, 3, 261-270, plus the errata (2006), 5, 281.

    posted by: Juliet Shaffer on 2015-12-05 06:53 PM (Europe/Berlin)