Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter February 2, 2022

Building a Framework for Mode Effect Estimation in United States Presidential Election Polls

  • Heather Kitada Smalley EMAIL logo and Christopher Wolf

Abstract

As public confidence in polling has been waning in the wake of recent elections (Narea, N. 2016. After 2016, Can we Ever Trust the Polls Again? New Republic), many researchers have been seeking to diagnose the shortcomings in these data (Gelman, A., and J. Azari. 2017. “19 Things We Learned from the 2016 Election.” Statistics and Public Policy 4 (1): 1–10; Kennedy, C., M. Blumenthal, S. Clement, J. D. Clinton, C. Durand, C. Franklin, K. McGeeney, L. Miringoff, K. Olson, D. Rivers, L. Saad, G. E. Witt, and C. Wlezien. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33; Mercer, A., C. Deane, and K. McGeeney. 2016. Why 2016 Election Polls Missed Their Mark. Pew Research Center. Also available at https://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/). One conjecture stems from observed differences between polling results based on the methodological choice between live and non-live modes of survey administration (Enten, H. 2015. The Future of Polling May Depend on Donald Trumps Fate. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-future-of-polling-may-depend-on-donald-trumps-fate/). While it has become commonplace to discuss “mode effect” on surveys, it reemerged in the political zeitgeist as the “Shy Trump” supporter hypothesis leading up to the 2016 U.S. Presidential Election (Edsall, T. B. 2016. How Many People Support Trump but Dont want to Admit it. New York Times). Motivated by the conflicting evidence for (Enns, P. K., J. Lagodny, and J. P. Schuldt. 2017. “Understanding the 2016 US Presidential Polls: The Importance of Hidden Trump Supporters.” Statistics, Politics, and Policy 8 (1): 41–63) and against (Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics, and Policy 8 (1): 29–40) this hypothesis, we built a complex statistical model that pools together results across multiple pollsters and throughout the election cycle while accounting for the nuances of these data. Specifically, we explored election data for the presence of mode effect using time series with a general additive mixed model (GAMM). We estimated mode effect at state and national levels to perform statistical mode adjustments, which we then compared to observed election results. In this paper, we utilized polling results from the United States Presidential Elections in 2016 (4208 polls) and 2020 (4133 polls). Using these data, we identified spatial trends and areas where mode effect was statistically significant at a 0.05 level. In summary, we make three contributions to the literature on mode effect adjustment in the poll aggregation setting. First, we present a straightforward and flexible statistical approach to estimating mode effect using time series data. In doing so, we help to bridge the gap between theory-focused statistical work and the social sciences. Second, we apply this method to two recent presidential elections, providing insight into the significance of mode effect. Third, we provide evidence for spatial mode effect trends suggesting regional voting behaviors that future scholars can explore.


Corresponding author: Heather Kitada Smalley, Willamette University, Salem, USA, E-mail:

References

Bailik, C., and H. Enten. 2016. The Polls Missed Trump. We Asked Pollsters why. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/.Search in Google Scholar

Bowling, A. 2005. “Mode of Questionnaire Administration Can Have Serious Effects on Data Quality.” Journal of Public Health 27 (3): 281–91, https://doi.org/10.1093/pubmed/fdi031.Search in Google Scholar

Brownback, A., and A. Novotny. 2018. “Social Desirability Bias and Polling Errors in the 2016 Presidential Election.” Journal of Behavioral and Experimental Economics 74: 38–56, https://doi.org/10.1016/j.socec.2018.03.001.Search in Google Scholar

Cassino, D. 2016. “How Todays Political Polling Works.” Harvard Business Review 1: 2016.Search in Google Scholar

Claassen, R. L., and J. B. Ryan. 2016. “Social Desirability, Hidden Biases, and Support for Hillary Clinton.” PS: Political Science & Politics 49 (4): 730–5, https://doi.org/10.1017/s1049096516001414.Search in Google Scholar

Cohn, N. 2015. Online Polls are Rising. So are Concerns about their Results. New York, USA: The New York Times. https://www.nytimes.com/2015/11/28/upshot/online-polls-are-rising-so-are-concerns-about-their-results.html.Search in Google Scholar

Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics, and Policy 8 (1): 29–40, https://doi.org/10.1515/spp-2016-0005.Search in Google Scholar

Couper, M. P. 2005. “Technology Trends in Survey Data Collection.” Social Science Computer Review 23 (4): 486–501, https://doi.org/10.1177/0894439305278972.Search in Google Scholar

Couper, M. P. 2011. “The Future of Modes of Data Collection.” Public Opinion Quarterly 75 (5): 889–908, https://doi.org/10.1093/poq/nfr046.Search in Google Scholar

Deming, W. E. 1944. “On Errors in Surveys.” American Sociological Review 9 (4): 359–69, https://doi.org/10.2307/2085979.Search in Google Scholar

Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, Vol. 2. New York: Wiley.Search in Google Scholar

Dillman, D. A., J. D. Smyth, and L. M. Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley & Sons.Search in Google Scholar

Dropp, K. 2016. How we Conducted our Shy Trumper Study. Morning Consult. Also available at https://morningconsult.com/2016/11/03/shy-trump-social-desirability-undercover-voter-study/.Search in Google Scholar

Dropp, K., R. Graziano, L. Maxwell, and S. Weiss. 2020. Are there “Shy Trump” Voters in 2020? Morning Consult. Also available at https://morningconsult.com/form/shy-trump-2020/.Search in Google Scholar

Edsall, T. B. 2016. How Many People Support Trump but Dont want to Admit it. New York, USA: The New York Times. https://www.nytimes.com/2016/05/11/opinion/campaign-stops/how-many-people-support-trump-but-dont-want-to-admit-it.html.Search in Google Scholar

Enns, P. K., J. Lagodny, and J. P. Schuldt. 2017. “Understanding the 2016 US Presidential Polls: The Importance of Hidden Trump Supporters.” Statistics, Politics, and Policy 8 (1): 41–63, https://doi.org/10.1515/spp-2017-0003.Search in Google Scholar

Enten, H. 2015. The Future of Polling May Depend on Donald Trumps Fate. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/the-future-of-polling-may-depend-on-donald-trumps-fate/.Search in Google Scholar

Enten, H. 2016. Shy Voters Probably arent why the Polls Missed Trump. FiveThirtyEight. Also available at https://fivethirtyeight.com/features/shy-voters-probably-arent-why-the-polls-missed-trump/.Search in Google Scholar

Faraway, J. J. 2016. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models. Boca Raton: Chapman & Hall/CRC.10.1201/9781315382722Search in Google Scholar

FiveThirtyEight 2020. Data. Also available at https://github.com/fivethirtyeight/data.Search in Google Scholar

Gelman, A. 2016. “A 2% Swing: The Poll-Based Forecast did Fine (On Average) in Blue States; they Blew it in the Red States.” In Statistical Modeling, Causal Inference, and Social Science. Also available at https://statmodeling.stat.columbia.edu/2016/11/09/polls-just-fine-blue-states-blew-red-states/.Search in Google Scholar

Gelman, A., and J. Azari. 2017. “19 Things we Learned from the 2016 Election.” Statistics and Public Policy 4 (1): 1–10, https://doi.org/10.1080/2330443x.2017.1356775.Search in Google Scholar

Hox, J. J., E. D. De Leeuw, and E. A. O. Zijlmans. 2015. Measurement Equivalence in Mixed Mode Surveys, edited by, R. van de Schoot, P. Schmidt and A. de beuckelaer, 115–23. Measurement Invariance, Frontiers in Psychology.10.3389/fpsyg.2015.00087Search in Google Scholar

Keeter, S. 2015. From Telephone to the Web: The Challenge of Mode of Interview Effects in Public Opinion Polls. Pew Research Center. Also available at https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/.Search in Google Scholar

Kellner, P. 2004. “Can Online Polls Produce Accurate Findings?” International Journal of Market Research 46 (1): 3–22, https://doi.org/10.1177/147078530404600102.Search in Google Scholar

Kennedy, C., M. Blumenthal, S. Clement, J. D. Clinton, C. Durand, C. Franklin, K. McGeeney, L. Miringoff, K. Olson, D. Rivers, L. Saad, G. E. Witt, and C. Wlezien. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33, https://doi.org/10.1093/poq/nfx047.Search in Google Scholar

Kim, S. T., D. Weaver, and L. Willnat. 2000. “Media Reporting and Perceived Credibility of Online Polls.” Journalism & Mass Communication Quarterly 77 (4): 846–64, https://doi.org/10.1177/107769900007700408.Search in Google Scholar

Klausch, T., J. J. Hox, and B. Schouten. 2013. “Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions.” Sociological Methods & Research 42 (3): 227–63, https://doi.org/10.1177/0049124113500480.Search in Google Scholar

Lohr, S. L. 2022. Sampling: Design and Analysis. Boca Raton: CRC Press.Search in Google Scholar

Mercer, A., C. Deane, and K. McGeeney. 2016. Why 2016 Election Polls Missed their Mark. Pew Research Center. Also available at https://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/.Search in Google Scholar

Narea, N. 2016. After 2016, Can we Ever Trust the Polls Again? The New Republic. https://newrepublic.com/article/139158/2016-can-ever-trust-polls-again.Search in Google Scholar

Shepard, S. 2016. Poll: ‘Shy Trump’ Voters are a Mirage. Politico. Also available at https://www.politico.com/story/2016/11/poll-shy-voters-trump-230667.Search in Google Scholar

Smalley, H. K., and C. Wolf. 2021. Pollgamm. Also available at https://github.com/kitadasmalley/pollGAMM.Search in Google Scholar

Stanton, Z. 2020. ‘People Are Going To Be Shocked’: Return of the ‘Shy’ Trump Voter? Politico: Politico. https://www.politico.com/news/magazine/2020/10/29/2020-polls-trump-biden-prediction-accurate-2016-433619.Search in Google Scholar

Tourangeau, R., and T. W. Smith. 1996. “Asking Sensitive Questions the Impact of Data Collection Mode, Question Format, and Question Context.” Public Opinion Quarterly 60 (2): 275–304, https://doi.org/10.1086/297751.Search in Google Scholar

Walker, A. H., and J. D. Restuccia. 1984. “Obtaining Information on Patient Satisfaction with Hospital Care: Mail versus Telephone.” Health Services Research 19 (3): 291.Search in Google Scholar

Received: 2021-08-03
Revised: 2021-12-18
Accepted: 2022-01-12
Published Online: 2022-02-02
Published in Print: 2022-03-28

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 27.2.2024 from https://www.degruyter.com/document/doi/10.1515/spp-2021-0024/html
Scroll to top button