Substantive Importance and the Veil of Statistical Significance

Kelly McCaskey 1  and Carlisle Rainey 1
  • 1 Department of Political Science, Texas A&M University, 2010 Allen Building, College Station, TX 77843, USA
Kelly McCaskey and Carlisle Rainey

Abstract

Political science is gradually moving away from an exclusive focus on statistical significance and toward an emphasis on the magnitude and importance of effects. While we welcome this change, we argue that the current practice of “magnitude-and-significance,” in which researchers only interpret the magnitude of a statistically significant point estimate, barely improves the much-maligned “sign-and-significance” approach, in which researchers focus only on the statistical significance of an estimate. This exclusive focus on the point estimate hides the uncertainty behind a veil of statistical significance. Instead, we encourage researchers to explicitly account for uncertainty by interpreting the range of values contained in the confidence interval. Especially when making judgments about the importance of estimated effects, we advise researchers to make empirical claims if and only if those claims hold for the entire confidence interval.

  • Achen, Christopher H. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.

    • Crossref
    • Export Citation
  • Berry, William D., Jacqueline H. R. DeMeritt and Justin Esarey (2010) “Testing for Interaction in Binary Logit and Probit Models: Is a Product Term Essential,” American Journal of Political Science, 54(1):105–119.

    • Crossref
    • Export Citation
  • Casella, George and Roger L. Berger (2002) Statistical Inference. 2nd ed. Pacific Grove, California: Duxbury.

  • Cohen, Jacob (1990) “Things I Have Learned (So Far),” American Psychologist, 45(12):1304–1312.

    • Crossref
    • Export Citation
  • Cohen, Jacob (1992) “A Power Primer,” Psychological Bulletin, 112(1):115–159.

    • Crossref
    • Export Citation
  • Esarey, Justin and Ahra Wu (2014) The Fault in our Stars: Measuring and Correcting Significance Bias in Political Science. Working paper. Copy at jee3.web.rice.edu/significancebias.pdf.

  • Esarey, Justin and Nathan Danneman (2015) “A Quantitative Method for Substantive Robustness Assessment,” Political Science Research and Methods, 3(1):95–111.

    • Crossref
    • Export Citation
  • Francis, Gregory (2013) “Replication, Statistical Consistency, and Publication Bias (with Discussion),” Journal of Mathematical Psychology, 57(5):153–169.

    • Crossref
    • Export Citation
  • Gelman, Andrew and Eric Loken (2014) “Ethics and Statistics: The AAA Tranche of Subprime Science,” Chance, 27(1):51–56.

    • Crossref
    • Export Citation
  • Gerber, Elisabeth R. and Daniel J. Hopkins (2011) “When Mayors Matter: Estimating the Impact of Mayoral Partisanship on City Policy,” American Journal of Political Science, 55(2):326–339.

    • Crossref
    • Export Citation
  • Gerber, Alan and Neil Malhotra (2008) “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals,” Quarterly Journal of Political Science, 3(3):313–326.

    • Crossref
    • Export Citation
  • Gill, Jeff (1999) “Null Hypothesis Significance Testing,” Political Research Quarterly, 52(3):647–674.

    • Crossref
    • Export Citation
  • Glass, Gene V. (1976) “Primary, Secondary, and Meta-Analysis of Research,” Educational Researcher, 5(10):3–8.

    • Crossref
    • Export Citation
  • Gross, Justin H. (2015) “Testing What Matters (If You Must Test At All): A Context-Driven Approach to Substantive and Statistical Significance,” American Journal of Political Science, 59(3):775–788.

    • Crossref
    • Export Citation
  • Hanmer, Michael J. and Kerem Ozan Kalkan (2013) “Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models,” American Journal of Political Science, 57(1):263–277.

    • Crossref
    • Export Citation
  • Hetherington, Marc and Elizabeth Suhay (2011) “Authoritarianism, Threat, and Americans’ Support for the War on Terror,” American Journal of Political Science, 55(3):546–560.

    • Crossref
    • Export Citation
  • Hill, Jr., Daniel W. and Zachary M. Jones (2014) “An Empirical Evaluation of Explanations for State Repression,” American Political Science Review, 108(3):1–27.

    • Crossref
    • Export Citation
  • Hultman, Lisa, Jacob Kathman and Megan Shannon (2013) “United Nations Peacekeeping and Civilian Protection in Civil War,” American Journal of Political Science, 57(4):875–891.

    • Crossref
    • Export Citation
  • Imai, Kosuke, Gary King and Olivia Lau (2008) “Toward a Common Framework for Statistical Analysis and Development,” Journal of Computational and Graphical Statistics, 17(4):892–913.

    • Crossref
    • Export Citation
  • Kam, Cindy D. and Carl L. Palmer (2008) “Reconsidering the Effects of Education on Political Participation,” Journal of Politics, 70(2):612–631.

    • Crossref
    • Export Citation
  • Kam, Cindy D. and Elizabeth J. Zechmeister (2013) “Name Recognition and Candidate Support,” American Journal of Political Science, 57(4):971–986.

    • Crossref
    • Export Citation
  • King, Gary and Langche Zeng (2001) “Logistic Regression in Rare Events Data,” Political Analysis, 9(2):137–163.

    • Crossref
    • Export Citation
  • King, Gary, Michael Tomz and Jason Wittenberg (2000) “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science, 44(2):341–355.

    • Crossref
    • Export Citation
  • Kirk, Roger E. (1996) “Practice Signifance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56(5):746–759.

    • Crossref
    • Export Citation
  • Rainey, Carlisle (2014) “Arguing for a Negligible Effect,” American Journal of Political Science, 58(4):1083–1091.

    • Crossref
    • Export Citation
  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2011) “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22(11):1359–1366.

    • Crossref
    • Export Citation
  • Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2014) “P-Curve: A Key to the File Drawer,” Journal of Experimental Psychology: General, 143(2):534–547.

    • Crossref
    • Export Citation
  • Thompson, Bruce (2001) “Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments to Move the Field,” Journal of Experimental Education, 70(1):80–93.

    • Crossref
    • Export Citation
  • Thompson, Bruce (2002) “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes,” Educational Researcher, 31(3):24–31.

    • Crossref
    • Export Citation
  • Tomz, Michael R. and Jessica L. P. Weeks (2013) “Public Opinion and the Democratic Peace,” American Political Science Review, 107(4):849–865.

    • Crossref
    • Export Citation
  • Tomz, Michael, Jason Wittenberg and Gary King (2003) “Clarify: Software for Interpreting and Presenting Statistical Results,” Journal of Statistical Software, 8(1):1–30.

    • Crossref
    • Export Citation
  • Tukey, John W. (1962) “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33(1):1–67.

    • Crossref
    • Export Citation
  • Tukey, John W. (1991) “The Philosophy of Multiple Comparisons,” Statistical Science, 6(1):100–116.

    • Crossref
    • Export Citation
  • Yates, F. (1951) “The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics,” American Statistical Association Journal, 46(253):19–34.

    • Crossref
    • Export Citation
Purchase article
Get instant unlimited access to the article.
$42.00
Log in
Already have access? Please log in.


or
Log in with your institution

Journal + Issues

Statistics, Politics, and Policy studies the ways that statistical analysis drives public policy decisions, and publishes significant research on the application of statistical ideas to problems that relate to policy implementation.

Search