Abstract
As public confidence in polling has been waning in the wake of recent elections (Narea, N. 2016. After 2016, Can we Ever Trust the Polls Again? New Republic), many researchers have been seeking to diagnose the shortcomings in these data (Gelman, A., and J. Azari. 2017. “19 Things We Learned from the 2016 Election.” Statistics and Public Policy 4 (1): 1–10; Kennedy, C., M. Blumenthal, S. Clement, J. D. Clinton, C. Durand, C. Franklin, K. McGeeney, L. Miringoff, K. Olson, D. Rivers, L. Saad, G. E. Witt, and C. Wlezien. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33; Mercer, A., C. Deane, and K. McGeeney. 2016. Why 2016 Election Polls Missed Their Mark. Pew Research Center. Also available at https://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/). One conjecture stems from observed differences between polling results based on the methodological choice between live and non-live modes of survey administration (Enten, H. 2015. The Future of Polling May Depend on Donald Trumps Fate. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-future-of-polling-may-depend-on-donald-trumps-fate/). While it has become commonplace to discuss “mode effect” on surveys, it reemerged in the political zeitgeist as the “Shy Trump” supporter hypothesis leading up to the 2016 U.S. Presidential Election (Edsall, T. B. 2016. How Many People Support Trump but Dont want to Admit it. New York Times). Motivated by the conflicting evidence for (Enns, P. K., J. Lagodny, and J. P. Schuldt. 2017. “Understanding the 2016 US Presidential Polls: The Importance of Hidden Trump Supporters.” Statistics, Politics, and Policy 8 (1): 41–63) and against (Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics, and Policy 8 (1): 29–40) this hypothesis, we built a complex statistical model that pools together results across multiple pollsters and throughout the election cycle while accounting for the nuances of these data. Specifically, we explored election data for the presence of mode effect using time series with a general additive mixed model (GAMM). We estimated mode effect at state and national levels to perform statistical mode adjustments, which we then compared to observed election results. In this paper, we utilized polling results from the United States Presidential Elections in 2016 (4208 polls) and 2020 (4133 polls). Using these data, we identified spatial trends and areas where mode effect was statistically significant at a 0.05 level. In summary, we make three contributions to the literature on mode effect adjustment in the poll aggregation setting. First, we present a straightforward and flexible statistical approach to estimating mode effect using time series data. In doing so, we help to bridge the gap between theory-focused statistical work and the social sciences. Second, we apply this method to two recent presidential elections, providing insight into the significance of mode effect. Third, we provide evidence for spatial mode effect trends suggesting regional voting behaviors that future scholars can explore.
References
Bailik, C., and H. Enten. 2016. The Polls Missed Trump. We Asked Pollsters why. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/.Search in Google Scholar
Bowling, A. 2005. “Mode of Questionnaire Administration Can Have Serious Effects on Data Quality.” Journal of Public Health 27 (3): 281–91, https://doi.org/10.1093/pubmed/fdi031.Search in Google Scholar
Brownback, A., and A. Novotny. 2018. “Social Desirability Bias and Polling Errors in the 2016 Presidential Election.” Journal of Behavioral and Experimental Economics 74: 38–56, https://doi.org/10.1016/j.socec.2018.03.001.Search in Google Scholar
Cassino, D. 2016. “How Todays Political Polling Works.” Harvard Business Review 1: 2016.Search in Google Scholar
Claassen, R. L., and J. B. Ryan. 2016. “Social Desirability, Hidden Biases, and Support for Hillary Clinton.” PS: Political Science & Politics 49 (4): 730–5, https://doi.org/10.1017/s1049096516001414.Search in Google Scholar
Cohn, N. 2015. Online Polls are Rising. So are Concerns about their Results. New York, USA: The New York Times. https://www.nytimes.com/2015/11/28/upshot/online-polls-are-rising-so-are-concerns-about-their-results.html.Search in Google Scholar
Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics, and Policy 8 (1): 29–40, https://doi.org/10.1515/spp-2016-0005.Search in Google Scholar
Couper, M. P. 2005. “Technology Trends in Survey Data Collection.” Social Science Computer Review 23 (4): 486–501, https://doi.org/10.1177/0894439305278972.Search in Google Scholar
Couper, M. P. 2011. “The Future of Modes of Data Collection.” Public Opinion Quarterly 75 (5): 889–908, https://doi.org/10.1093/poq/nfr046.Search in Google Scholar
Deming, W. E. 1944. “On Errors in Surveys.” American Sociological Review 9 (4): 359–69, https://doi.org/10.2307/2085979.Search in Google Scholar
Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, Vol. 2. New York: Wiley.Search in Google Scholar
Dillman, D. A., J. D. Smyth, and L. M. Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley & Sons.Search in Google Scholar
Dropp, K. 2016. How we Conducted our Shy Trumper Study. Morning Consult. Also available at https://morningconsult.com/2016/11/03/shy-trump-social-desirability-undercover-voter-study/.Search in Google Scholar
Dropp, K., R. Graziano, L. Maxwell, and S. Weiss. 2020. Are there “Shy Trump” Voters in 2020? Morning Consult. Also available at https://morningconsult.com/form/shy-trump-2020/.Search in Google Scholar
Edsall, T. B. 2016. How Many People Support Trump but Dont want to Admit it. New York, USA: The New York Times. https://www.nytimes.com/2016/05/11/opinion/campaign-stops/how-many-people-support-trump-but-dont-want-to-admit-it.html.Search in Google Scholar
Enns, P. K., J. Lagodny, and J. P. Schuldt. 2017. “Understanding the 2016 US Presidential Polls: The Importance of Hidden Trump Supporters.” Statistics, Politics, and Policy 8 (1): 41–63, https://doi.org/10.1515/spp-2017-0003.Search in Google Scholar
Enten, H. 2015. The Future of Polling May Depend on Donald Trumps Fate. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-future-of-polling-may-depend-on-donald-trumps-fate/.Search in Google Scholar
Enten, H. 2016. Shy Voters Probably arent why the Polls Missed Trump. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/shy-voters-probably-arent-why-the-polls-missed-trump/.Search in Google Scholar
Faraway, J. J. 2016. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models. Boca Raton: Chapman & Hall/CRC.10.1201/9781315382722Search in Google Scholar
FiveThirtyEight 2020. Data. Also available at https://github.com/fivethirtyeight/data.Search in Google Scholar
Gelman, A. 2016. “A 2% Swing: The Poll-Based Forecast did Fine (On Average) in Blue States; they Blew it in the Red States.” In Statistical Modeling, Causal Inference, and Social Science. Also available at https://statmodeling.stat.columbia.edu/2016/11/09/polls-just-fine-blue-states-blew-red-states/.Search in Google Scholar
Gelman, A., and J. Azari. 2017. “19 Things we Learned from the 2016 Election.” Statistics and Public Policy 4 (1): 1–10, https://doi.org/10.1080/2330443x.2017.1356775.Search in Google Scholar
Hox, J. J., E. D. De Leeuw, and E. A. O. Zijlmans. 2015. Measurement Equivalence in Mixed Mode Surveys, edited by, R. van de Schoot, P. Schmidt and A. de beuckelaer, 115–23. Measurement Invariance, Frontiers in Psychology.10.3389/fpsyg.2015.00087Search in Google Scholar
Keeter, S. 2015. From Telephone to the Web: The Challenge of Mode of Interview Effects in Public Opinion Polls. Pew Research Center. Also available at https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/.Search in Google Scholar
Kellner, P. 2004. “Can Online Polls Produce Accurate Findings?” International Journal of Market Research 46 (1): 3–22, https://doi.org/10.1177/147078530404600102.Search in Google Scholar
Kennedy, C., M. Blumenthal, S. Clement, J. D. Clinton, C. Durand, C. Franklin, K. McGeeney, L. Miringoff, K. Olson, D. Rivers, L. Saad, G. E. Witt, and C. Wlezien. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33, https://doi.org/10.1093/poq/nfx047.Search in Google Scholar
Kim, S. T., D. Weaver, and L. Willnat. 2000. “Media Reporting and Perceived Credibility of Online Polls.” Journalism & Mass Communication Quarterly 77 (4): 846–64, https://doi.org/10.1177/107769900007700408.Search in Google Scholar
Klausch, T., J. J. Hox, and B. Schouten. 2013. “Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions.” Sociological Methods & Research 42 (3): 227–63, https://doi.org/10.1177/0049124113500480.Search in Google Scholar
Lohr, S. L. 2022. Sampling: Design and Analysis. Boca Raton: CRC Press.Search in Google Scholar
Mercer, A., C. Deane, and K. McGeeney. 2016. Why 2016 Election Polls Missed their Mark. Pew Research Center. Also available at https://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/.Search in Google Scholar
Narea, N. 2016. After 2016, Can we Ever Trust the Polls Again? The New Republic. https://newrepublic.com/article/139158/2016-can-ever-trust-polls-again.Search in Google Scholar
Shepard, S. 2016. Poll: ‘Shy Trump’ Voters are a Mirage. Politico. Also available at https://www.politico.com/story/2016/11/poll-shy-voters-trump-230667.Search in Google Scholar
Smalley, H. K., and C. Wolf. 2021. Pollgamm. Also available at https://github.com/kitadasmalley/pollGAMM.Search in Google Scholar
Stanton, Z. 2020. ‘People Are Going To Be Shocked’: Return of the ‘Shy’ Trump Voter? Politico: Politico. https://www.politico.com/news/magazine/2020/10/29/2020-polls-trump-biden-prediction-accurate-2016-433619.Search in Google Scholar
Tourangeau, R., and T. W. Smith. 1996. “Asking Sensitive Questions the Impact of Data Collection Mode, Question Format, and Question Context.” Public Opinion Quarterly 60 (2): 275–304, https://doi.org/10.1086/297751.Search in Google Scholar
Walker, A. H., and J. D. Restuccia. 1984. “Obtaining Information on Patient Satisfaction with Hospital Care: Mail versus Telephone.” Health Services Research 19 (3): 291.Search in Google Scholar
© 2022 Walter de Gruyter GmbH, Berlin/Boston
