Abstract
Political science is gradually moving away from an exclusive focus on statistical significance and toward an emphasis on the magnitude and importance of effects. While we welcome this change, we argue that the current practice of “magnitude-and-significance,” in which researchers only interpret the magnitude of a statistically significant point estimate, barely improves the much-maligned “sign-and-significance” approach, in which researchers focus only on the statistical significance of an estimate. This exclusive focus on the point estimate hides the uncertainty behind a veil of statistical significance. Instead, we encourage researchers to explicitly account for uncertainty by interpreting the range of values contained in the confidence interval. Especially when making judgments about the importance of estimated effects, we advise researchers to make empirical claims if and only if those claims hold for the entire confidence interval.
Acknowledgments
We thank Lisa Hultman, Cindy Kam, Jacob Kathman, and Megan Shannon, and Elizabeth Zechmeister for making their data available to us. The analyses presented here were conducted with R 3.1.0. All data and computer code necessary to reproduce our paper and results are available at https://github.com/carlislerainey/meaningful-inferences.
References
Achen, Christopher H. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.10.4135/9781412984560Search in Google Scholar
Berry, William D., Jacqueline H. R. DeMeritt and Justin Esarey (2010) “Testing for Interaction in Binary Logit and Probit Models: Is a Product Term Essential,” American Journal of Political Science, 54(1):105–119.10.1111/j.1540-5907.2009.00429.xSearch in Google Scholar
Casella, George and Roger L. Berger (2002) Statistical Inference. 2nd ed. Pacific Grove, California: Duxbury.Search in Google Scholar
Cohen, Jacob (1990) “Things I Have Learned (So Far),” American Psychologist, 45(12):1304–1312.10.1037/0003-066X.45.12.1304Search in Google Scholar
Cohen, Jacob (1992) “A Power Primer,” Psychological Bulletin, 112(1):115–159.10.1037/0033-2909.112.1.155Search in Google Scholar
Esarey, Justin and Ahra Wu (2014) The Fault in our Stars: Measuring and Correcting Significance Bias in Political Science. Working paper. Copy at jee3.web.rice.edu/significancebias.pdf.Search in Google Scholar
Esarey, Justin and Nathan Danneman (2015) “A Quantitative Method for Substantive Robustness Assessment,” Political Science Research and Methods, 3(1):95–111.10.1017/psrm.2014.14Search in Google Scholar
Francis, Gregory (2013) “Replication, Statistical Consistency, and Publication Bias (with Discussion),” Journal of Mathematical Psychology, 57(5):153–169.10.1016/j.jmp.2013.02.003Search in Google Scholar
Gelman, Andrew and Eric Loken (2014) “Ethics and Statistics: The AAA Tranche of Subprime Science,” Chance, 27(1):51–56.10.1080/09332480.2014.890872Search in Google Scholar
Gerber, Elisabeth R. and Daniel J. Hopkins (2011) “When Mayors Matter: Estimating the Impact of Mayoral Partisanship on City Policy,” American Journal of Political Science, 55(2):326–339.10.1111/j.1540-5907.2010.00499.xSearch in Google Scholar
Gerber, Alan and Neil Malhotra (2008) “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals,” Quarterly Journal of Political Science, 3(3):313–326.10.1561/100.00008024Search in Google Scholar
Gill, Jeff (1999) “Null Hypothesis Significance Testing,” Political Research Quarterly, 52(3):647–674.10.1177/106591299905200309Search in Google Scholar
Glass, Gene V. (1976) “Primary, Secondary, and Meta-Analysis of Research,” Educational Researcher, 5(10):3–8.10.3102/0013189X005010003Search in Google Scholar
Gross, Justin H. (2015) “Testing What Matters (If You Must Test At All): A Context-Driven Approach to Substantive and Statistical Significance,” American Journal of Political Science, 59(3):775–788.10.1111/ajps.12149Search in Google Scholar
Hanmer, Michael J. and Kerem Ozan Kalkan (2013) “Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models,” American Journal of Political Science, 57(1):263–277.10.1111/j.1540-5907.2012.00602.xSearch in Google Scholar
Hetherington, Marc and Elizabeth Suhay (2011) “Authoritarianism, Threat, and Americans’ Support for the War on Terror,” American Journal of Political Science, 55(3):546–560.10.1111/j.1540-5907.2011.00514.xSearch in Google Scholar
Hill, Jr., Daniel W. and Zachary M. Jones (2014) “An Empirical Evaluation of Explanations for State Repression,” American Political Science Review, 108(3):1–27.10.1017/S0003055414000306Search in Google Scholar
Hultman, Lisa, Jacob Kathman and Megan Shannon (2013) “United Nations Peacekeeping and Civilian Protection in Civil War,” American Journal of Political Science, 57(4):875–891.10.1111/ajps.12036Search in Google Scholar
Imai, Kosuke, Gary King and Olivia Lau (2008) “Toward a Common Framework for Statistical Analysis and Development,” Journal of Computational and Graphical Statistics, 17(4):892–913.10.1198/106186008X384898Search in Google Scholar
Kam, Cindy D. and Carl L. Palmer (2008) “Reconsidering the Effects of Education on Political Participation,” Journal of Politics, 70(2):612–631.10.1017/S0022381608080651Search in Google Scholar
Kam, Cindy D. and Elizabeth J. Zechmeister (2013) “Name Recognition and Candidate Support,” American Journal of Political Science, 57(4):971–986.10.1111/ajps.12034Search in Google Scholar
King, Gary and Langche Zeng (2001) “Logistic Regression in Rare Events Data,” Political Analysis, 9(2):137–163.10.1093/oxfordjournals.pan.a004868Search in Google Scholar
King, Gary, Michael Tomz and Jason Wittenberg (2000) “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science, 44(2):341–355.10.2307/2669316Search in Google Scholar
Kirk, Roger E. (1996) “Practice Signifance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56(5):746–759.10.1177/0013164496056005002Search in Google Scholar
Rainey, Carlisle (2014) “Arguing for a Negligible Effect,” American Journal of Political Science, 58(4):1083–1091.10.1111/ajps.12102Search in Google Scholar
Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2011) “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science, 22(11):1359–1366.10.1177/0956797611417632Search in Google Scholar
Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn (2014) “P-Curve: A Key to the File Drawer,” Journal of Experimental Psychology: General, 143(2):534–547.10.1037/a0033242Search in Google Scholar
Thompson, Bruce (2001) “Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments to Move the Field,” Journal of Experimental Education, 70(1):80–93.10.1080/00220970109599499Search in Google Scholar
Thompson, Bruce (2002) “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes,” Educational Researcher, 31(3):24–31.10.3102/0013189X031003025Search in Google Scholar
Tomz, Michael R. and Jessica L. P. Weeks (2013) “Public Opinion and the Democratic Peace,” American Political Science Review, 107(4):849–865.10.1017/S0003055413000488Search in Google Scholar
Tomz, Michael, Jason Wittenberg and Gary King (2003) “Clarify: Software for Interpreting and Presenting Statistical Results,” Journal of Statistical Software, 8(1):1–30.10.18637/jss.v008.i01Search in Google Scholar
Tukey, John W. (1962) “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33(1):1–67.10.1214/aoms/1177704711Search in Google Scholar
Tukey, John W. (1991) “The Philosophy of Multiple Comparisons,” Statistical Science, 6(1):100–116.10.1214/ss/1177011945Search in Google Scholar
Yates, F. (1951) “The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics,” American Statistical Association Journal, 46(253):19–34.10.2307/2280090Search in Google Scholar
©2015 by De Gruyter
