Debate Reaction Ideal Points: Political Ideology Measurement Using Real-Time Reaction Data

1 Obstacles to Measuring Ideology in the Public

Existing research on ideology in the mass public uses both self-reported attitudinal (Ansolabehere, Rodden, and Snyder 2008; Barber and Pope 2018) and observed behavioral (Barberá 2015; Bonica 2019) measures, at the individual (Jessee 2016) and aggregate (Abramowitz and Saunders 2008; Tausanovich and Warshaw 2013) levels. There are several challenges to the uses of these measures, and the different approaches result in different conclusions about the distribution of ideology in the public over time (cf. Abramowitz 2010; Fiorina and Abrams 2009). Therefore, it is important to think carefully about both the practical and theoretical implications of various measurement strategies.

For decades, the most common and straightforward measure of ideology in the United States has been to ask people to describe themselves on a scale ranging from “extremely conservative” to “extremely liberal.” This single-item self-reported ideology (alternatively called symbolic ideology or ideological identification) demonstrates high within-subject reliability over time and consistency with party identification or other aspects of individual identity (Kinder and Kalmoe 2017). In a thorough examination, Kinder and Kalmoe (2017) conclude that ideological self-identification only holds modest predictive power with regards to presidential voting, even less for down-ballot races, and very little constraining relationship with policy attitudes (Ellis and Stimson 2012; Kinder and Kalmoe 2017; Malka and Lelkes 2010). At best, direct self-identification of ideology can be understood as a general summary judgment of a range of policy- and identity-based considerations (Kinder and Kalmoe 2017), which reflect a variety of social cues beyond politics and policy (Ellis and Stimson 2012).

Because self-identification of ideology lacks the necessary consistent and constraining relationship with policy views, some researchers have instead turned to measuring ideology through a combination of multiple survey questions asking about specific issue positions. Averaging across multiple measures provides estimates that reveal more structure and stability in individual preferences than using single-item survey questions (Ansolabehere, Rodden, and Snyder 2008). Moving beyond averages, scholars use a variety of methods to generate a single estimate of ideology from a range of policy issue questions. For example, Abramowitz (2010), Abramowitz and Saunders (2008), and the Pew Research Center (2017) use a count of policy responses that are consistent with commonly understood “liberal” and “conservative” ideological positions. Baldassarri and Gelman (2008) use correlations between survey items on the ANES. Still others use factor analysis or item response theory to extract latent ideology from the set of policy responses (Hill and Tausanovitch 2015; Jacobson 2012; Tausanovitch and Warshaw 2013; Treier and Hillygus 2009; Ura and Ellis 2012).

However, survey responses have a number of weaknesses for measurement of political attitudes generally that are not clearly overcome by asking more policy questions or combining them using more complex algorithms. First, public opinion research has a long history of demonstrating that individuals are often unable or unwilling to accurately report their political opinions in a meaningful or consistent way (stemming from Converse 1964). Indeed, even small variations in question wording, order, and response options can produce sizeable variance in the attitudes people report on surveys (e.g. Smith 1989), with substantial implications for the perceived level of polarization (Fiorina and Abrams 2009). Survey responses based on policy information are impacted by dual concerns of framing and information. As researchers provide additional information to contextualize policy debates for less informed respondents, they risk influencing the responses through priming or framing processes (Smith 1989).

Second, declining response rates further exacerbate the potential for disconnect between the willing survey taker and the general public (Schill 2017). As response rates decline, a growing divide between politically interested survey takers and the general public may lead to less representative survey data. Additionally, as a specific manifestation of survey response error, many uninterested or uninformed survey respondents may pick moderate response options to avoid the cognitive burden of figuring out their position. It is therefore difficult to distinguish people who genuinely have no opinion from those who are ideological moderates (Fiorina and Abrams 2008; Kinder and Kalmoe 2017).

Third, measurement error can also be introduced by researchers through the process of creating and administering the survey. Due to space and cost constraints inherent in surveys, researchers must make selective decisions about the particular set of policy issue questions to include on the survey (Schill 2017), which can influence the resulting ideology measurements. Researcher priorities may not reflect the most salient political issues at the time when the respondent takes the survey. Furthermore, the connection of abstract ideologies to specific issue positions is not itself stable over time (Lewis 2019). Even survey-based measurements that rely on a well-established standard set of questions may not be reliable over time as the meaning of policy alternatives shift relative the current political climate and context.

Another strategy for measuring ideological preferences relies on observed political outcomes, such as election returns (Abramowitz 2010), vote totals, or candidate approval ratings (Kimball and Gross 2007) rather than self-reported policy opinions. These measures have the advantage of minimizing error due to the willingness or ability of respondents to provide accurate answers, either because they are entirely behavioral or because they have a lower cognitive and informational burden. Furthermore, they are deeply embedded in the reality of the contemporaneous political climate. However, using observed choices (such as votes) when the set of discrete choice options has itself grown more polarized may create the appearance of increased ideological consistency or extremity even when the underlying opinions have not changed (Fiorina and Abrams 2009; Fiorina, Abrams, and Pope 2008; Prior 2013). These measures also conflate ideological views with partisan choices (Fiorina, Abrams, and Pope 2008).

More recently, scholars have begun to mine available non-survey behavioral data to estimate ideology. Barberà (2015) uses the known ideology of political elites on Twitter to infer the ideology of the non-elite Twitter users who follow their accounts. Bonica (2019) uses political donation patterns to estimate the ideology of individual campaign donors. These approaches have the many advantages of being behavioral, contextual, real-world data from which to measure ideology, and are not influenced by researcher survey design choices or reliant on what people are willing or able to respond about their own preferences. However, they are necessarily limited to relatively narrow segments of the general population whose level of political interest and preferences are substantially different from the general public (for campaign donors, see Broockman and Malhotra 2020; for Twitter users, see Pew Research Center 2019).

While each of these approaches to measuring ideology have significant strengths, they also each face notable limitations. Because there is no ground truth measure for ideology against which to compare these various approaches and determine which is most accurate, researchers must rely on a varied toolkit of approaches and carefully consider which is most suited to a particular research approach. We next introduce our addition to this toolkit—debate reaction ideal points—and discuss how this approach mitigates many of the weaknesses of previous measures, along with consideration of its limitations.

2 Ideal Point Modeling Using Real-Time Presidential Debate Reactions

Real-time response data are gaining prominence as an alternative to survey methods in public opinion research (Schill 2017). Communication scholars have used real-time reactions in both laboratory and real-world settings to evaluate viewer reactions to the content of media material for nearly a century (Schill 2017). Political scientists have applied such techniques to political media, including advertisements (Iyengar, Jackman, and Hahn 2017; Kaid 2009; Maier, Hampe, and Jahn 2016), debates (Boydstun et al. 2014b; Hughes and Bucy 2017; Jarman 2005; Jasperson, Gollins, and Walls 2017; Schill and Kirk 2014), and multi-person panels (Waldvogel 2020). Real-time reaction approaches build on psychological theories of information processing to capture attitudes at the moment of exposure, capitalize on high frequency of measurement to provide additional statistical leverage, and, because of their breadth, can “capture the richness of individuals’ views” (Schill 2017, p. 19).

Here, we argue that viewer reactions to political debates tell us not only about the effectiveness of various aspects of the message, but also something about the viewers themselves. As viewers react positively to some messages and negatively to others over the course of the debate, we can model their political predispositions.

Presidential debates are a unique, information-rich political event, and real-time reactions to debates are well-suited to understanding viewers’ political attitudes. Debates provide an exceptional opportunity for ideal-point estimation because candidates in debates cover a wide range of policy topics using language and style designed to appeal to the lay voter (Benoit 2013). The real-time element of the debate reaction is particularly important, since post-debate coverage has a sizeable demonstrated impact on public opinion surrounding presidential debates (Fridkin et al. 2008).

Audience reactions to statements in a debate reflect, for the most part, reactions to salient policy topics that can have an impact on vote choice (Benoit 2013). In the 2012 presidential debates, 76% of statements were about policy rather than either candidate’s personal character (Benoit 2013), and the policy topics were driven more by public opinion salience than what the candidates might select based on their relative issue strengths (Boydstun, Glazier, and Pietryka 2013). Although the policy discussion in debates is sometimes criticized for lacking complexity (Rowland 2013), there is evidence that watching debates increases both issue knowledge and the importance that viewers place on policy positions in evaluating candidates (Benoit, Mckinney, and Holbert 2001, 2003). Debate content is a part of the actual information used by debate-watchers to make their voting decisions (Benoit 2013; Benoit, Hansen, and Verser 2003; Blais and Perrella 2008).^[2] Thus, ideology measurements based on reactions of prospective voters to real-time statements in live debates represent a snapshot of political judgments as they would actually occur in the course of a developing political campaign, rather than being influenced by survey design or researcher biases.

Ideal point models are commonly used to provide an objective, data-driven, and scalable characterization of the ideological positions of political actors (Laver 2014). While these models were originally developed to generate ideal point estimates of elected officials (McCarty, Poole, and Rosenthal 1997; Shor and McCarty 2011), more recent scholars have applied them to members of the general public by using their survey responses (Bafumi and Herron 2010; Hill and Tausanovitch 2015; Jessee 2012, 2016; Tausanovitch and Warshaw 2013), donations to candidates (Bonica 2019), or decisions of who to follow on Twitter (Barberá 2015). More finely articulated computational models of legislative behavior have allowed for the generation of issue-specific ideal points (Gerrish and Blei 2012; Lauderdale and Clark 2014) and accounted for text-based framing (Nguyen et al. 2015). Recently, text-based ideal points have been estimated using only language inputs without additional behavioral signals (Vafa, Naidu, and Blei 2020). Along similar lines, we posit that reacting positively or negatively to a political statement is a choice amenable to ideal-point analysis. To that end, we use real-time responses to statements made in the 2012 presidential and vice-presidential debates collected by an online response app (Boydstun et al. 2014b; Resnik et al. 2017).

Ideal point modeling using real-time debate reactions overcomes many of the challenges faced by existing public ideology measurement strategies.

First, the topics of the debates are wide-ranging, allowing for a far more robust ideology measure than is available through analysis of a single vote choice or a limited set of issue positions on a survey. Additionally, these topics are directly tied to the salient issues in the political context and campaign cycle (Boydstun, Glazier, and Pietryka 2013), and so reflect the political reality as it exists at the moment of measurement. Rather than survey measures decided in advance by the researcher and subject to misinterpretation or shifting meaning over time, the debate moderator sets the agenda and the candidates themselves determine the salient aspects of the policy questions through their responses (Nguyen, Boyd-Graber, and Resnik 2012). It is true that the topics covered in a debate are not comprehensive, meaning debate data may not provide a comprehensive ideology measure based on the full range of the political landscape or subsets of issues that are not prominent on the agenda during the debate. However, debate reaction ideal points can be considered a snapshot of ideology that reflects the contemporaneous national political context.

Second, reaction data avoid some of the pitfalls of self-reported ideological or issue-position data, including social desirability biases or satisficing in survey responses. Real-time response data for live viewing are demonstrated to have high reliability and validity, while avoiding biases of survey responses (Waldvogel 2020). Rather than complex questions with many response options, the reactions involve simple choices in response to policy discussion presented in an accessible visual and auditory format directly from the candidates. Where surveys must decide between offering explanatory information and potentially influencing responses, or not providing background and increasing uninformed responses, debate candidates often present the relevant background information and counterarguments, informing voters as they make their cases.

Third, ideal point estimates are an established way of understanding ideology and polarization for political actors. Although the ideal points we estimate are not on a directly comparable policy space or an identical task to politicians considering proposed legislation or judicial cases, using an ideal point estimation model provides an important comparability in method to the estimates of ideology commonly used for elites. Additionally, ideal point estimation generates an ideology measure that is more meaningful and stable than alternatives based only on one or a few survey questions (Ansolabehere, Rodden, and Snyder 2008).

Fourth, respondents can choose which statements to respond to during the debates. Patterns of response and non-response provide meaningful information about a debate-watcher’s prioritization of different policy issues (Boydstun et al. 2014b). We use a Bayesian ordinal logit implementation of the ideal-point model, which, unlike the typical binary logit-based model, allows us to include information about both the choice to respond and the direction of response in the ideal point model.

Fifth, ideology estimation and conceptualization for moderates is particularly difficult using existing measures (Ahler and Broockman 2018), in part because people who are uninterested lack the necessary information to form a meaningful survey response (Fiorina and Abrams 2009), or are unlikely to be included in behavioral samples (i.e. campaign donors, political Twitter users). Debates are helpfully geared towards prime time viewers (Benoit 2013) and ideal points can be estimated for anyone who can be incentivized to watch and react. Additionally, while still subject to selection and non-response biases, real-time debate reaction data have potential to incorporate more uninterested or uninformed respondents than traditional survey methods (Waldvogel and Metz 2020).

Sixth, although the partisanship of the candidates is a highly salient cue in the debates, respondents can provide a mixture of many responses to statements by both candidates, instead of making a single binary choice between them (see the Online Appendix for the distribution of responses to in-party and out-party candidates). This helps mitigate the appearance of polarization that can arise when the measurement is based on a single binary choice between two increasingly polarized alternatives.

Finally, the availability of debate transcripts allows for the inclusion of textual information into the estimation and interpretation of the ideal points. Specifically, the researcher is able to remove statements that are likely to elicit non-policy reactions (e.g. insults or one-liners), or limit the ideology estimations to particular subsets of policy considerations. Additionally, the Bayesian ideal point estimation strategy produces additional information about the statements themselves that can be used to qualitatively evaluate the performance of the estimation procedure.

3 Data and Methods

During the October 2012 U.S. presidential and vice presidential debates, 6560 college students were recruited to watch and react to the debates using a browser-based mobile application (Resnik et al. 2017). Subjects were recruited by their instructors and received course credit for participation (Boydstun et al. 2014a). This method of recruitment generated a large and diverse respondent pool, with sample demographics that are highly comparable to a variety of national metrics and a sizeable number of respondents in rare demographic categories, such as African-American Republicans (Boydstun et al. 2014a). The sample as a whole is younger, somewhat more religious, and slightly more liberal than the general public (53–59% planned to vote for Obama). College students are distinct from the general population in important ways, including the noted instability of political ideology during the impressionable years (Krosnick and Alwin 1989). While the nature of the sample prevents us from making inferences about the distribution of ideology in the general population, the large sample size and nationwide recruitment provide ample variation to establish the relationships needed for validation of the method. Additional information about the recruitment process and sample demographics is available in the Online Appendix.

Notably, the process motivated respondents to participate in debate watching who might not have otherwise chosen to do so for their own purposes, which increases our ability to estimate ideology for people with a broad range of political interest and engagement. Note that Waldvogel and Metz (2020) also report broad reach with real-time debate response data in Germany. Real-time debate reaction data, then, provide an important potential for reaching otherwise uninterested or disengaged segments of the population.

Prior to each debate, participants completed a brief pre-debate survey that included demographic and political attitude questions. During the debate, the user could register a reaction by tapping a target (one of the candidates or the moderator) to indicate to whom they were reacting, followed by tapping one of Agree, Disagree, Spin, or Dodge to record a reaction. These responses were considered self-explanatory and participants were not given additional definitions.

Using time-stamped transcripts, we identify distinct statements made by the candidates and moderator during the four debates. We use the timestamps of the debate viewers’ reactions to link reactions to specific debate statements, and consider the reaction to be a “vote” in favor of or against that statement.^[3] Agree reactions are considered a positive reaction to the statement, and coded as 1. Disagree, Spin, and Dodge reactions are considered a negative reaction to the statement, and coded as −1.^[4] Statements to which a respondent did not react are coded as 0. Consistent with best practices in ideal point estimation, we also drop statements that have unanimous or near-unanimous response patterns (more than 99% of viewers have the same code). We also only include respondents who reacted to at least 10 statements. This results in 5119 debate watchers, reacting to 523 debate statements, with a median of 30 reactions per person. We pool the data for all four debates into a single dataset to estimate the ideal points.

We estimate a Bayesian ordinal logit parameterization of an ideal point model, based on respondents’ patterns of positive, negative, and no reactions to the debate statements.^[5] Using an ordinal logit allows for explicit modeling of the decision not to respond as a separate choice from agreeing or disagreeing. Practically speaking, this increases the information available to estimate each individual’s ideal point, resulting in estimates that are more precise than with a standard logit implementation.^[6]

4 Results

A kernel density plot of the estimated ideal points for all respondents is presented in Figure 1. The ideal points are normalized to have a mean of zero and variance of one. The distribution is notably bimodal, where there are peaks on both the liberal (negative) and conservative (positive) sides of the mean.^[7] The majority of responses fall within one standard deviation of the mean indicating a substantial amount of moderation. The liberal peak is somewhat larger than the conservative peak, reflecting the somewhat higher number of self-described Democratic respondents in our sample (see Boydstun et al. 2014a). However, as we will demonstrate, there is substantial within-party variation in the ideal point estimates indicating that the measure is not merely a proxy for partisanship.

Figure 1:

Kernel density plot of estimated ideal points.

Kernel density plots of debate reaction ideal points estimated using a Bayesian ordinal logit model. Normalized to mean = 0, standard deviation = 1. Negative values represent more liberal ideologies.

Next, we validate these ideal points by performing a series of comparisons between the estimated ideal points and other metrics from the pre-debate survey, for which we expect a higher (partisanship, political interest) or lower (self-reported ideology) level of correlation. There is some difficulty inherent in validation of the measure, as there is no ground truth measurement of ideology with which to compare our metric. Therefore, the tests we conduct are based on the estimated ideal points relating in expected ways with other measures based on what we know from the existing literature on ideology.

The first test is whether the estimated debate reaction ideal points correlate with self-reported ideology. As previously discussed, self-reported ideology typically has modest correlation with policy-preference based measures, but it is not a strongly predictive of vote choice or consistently indicative of policy preferences across topic areas. The pre-debate survey asked respondents to report their ideology on a 101-point scale with zero being the most liberal and 100 being the most conservative. Figure 2 displays a scatterplot of the correlation between the ideal point estimate (y axis) and self-reported ideology measure (x axis). The Pearson’s correlation between self-reported ideology and estimated ideal points is 0.69.

Figure 2:

Scatter plot correlation of ideal points with self-reported ideology.

Self-reported ideology ranges from 0 (most liberal) to 100 (most conservative). Observations are also distinguished by self-reported partisanship.

However, prior scholars have noted that the overall correlation between ideology and partisan or ideological-identity measures can be deceptive because it leverages the distinction between liberal and conservative categories but may have limited predictive value within party (see Bonica 2019; Hill and Huber 2017), and increasingly stark cross-party differences can mask important intra-party divisions (Nguyen et al. 2015). Our measure is no exception; restricted to subsamples by party, the correlation between self-reported and estimated ideology for Democrats is 0.28 and for Republicans is 0.38. As Kinder and Kalmoe (2017) argue, we expect this is because strength of ideological self-identification likely reflects a stronger identity-based attachment to the ideological label as a social identity attachment, rather than indicating a more extreme set of policy preferences. Modest correlation with ideological self-identification and, in particular, low correlation within parties, conforms to the expected pattern of relationships between self-reported ideology and ideology estimated from a set of policy positions.

We also note that the self-reported ideology distribution in Figure 2 visibly demonstrates one of the drawbacks of single-item survey-based measures, with clear clustering at the values of 0, 50, and 100. The estimated ideal-points, by contrast, are more continuous in nature.

Ideology has become increasingly aligned with party identification in recent years, so we next examine the distribution of ideal points by respondents’ self-reported party identification. We expect respondents who identify as closest to the Republican or Democratic parties to have the most extreme ideologies, with respondents who lean towards the Democratic or Republican parties closer to the middle, and independents to be the most centrist and also have the highest variance. Figure 3 shows a ridge plot of the kernel density of ideal points, broken down by party affiliation. As expected, those with stronger attachments to the Republican (Democratic) party have a more extreme median ideal point, 1.17 (−0.65) than those with weaker attachments to the Republican (Democratic) party, 0.89 (−0.42).

Figure 3:

Distribution of ideal points by party affiliation.

Kernel density plots of the ideal points in Figure 1, separated by respondents’ self-reported party identification (right) and campaign interest (left). Vertical dashed lines indicate median values for each subgroup. Party ID is measured using a single question self-reported five-category scale.

Another common test of ideology measures is their predictive power regarding vote choice. Because we do not have actual individual-level post-election vote choice, and because individual evaluations of candidates immediately prior to the debate may be too closely connected to the debate reactions measure to provide a meaningful test, we test the correlation between ideal points and vote choice at an aggregate level. Figure 4 displays the correlation between debate-watchers’ average ideal points and actual 2012 presidential vote totals, by state. As expected, states that voted for Obama at higher rates also have more liberal mean ideal point scores. With one exception, states that had less than 50% voting for Obama have a mean ideal point greater than zero, indicating a more conservative average ideology. The Pearson’s correlation (omitting DC as a potential outlier) between these two measures is −0.74.

Figure 4:

Scatterplot correlation between average ideal point and state vote totals.

Mean ideal point for all respondents in the state on the X axis; negative values are more liberal. State-level percent of vote received by Obama on the Y axis. States with fewer than 50 respondents are omitted from the plot. Because DC is a clear outlier, it is included on the graph but is not included in the correlation calculations.

Next, we expect that respondents who are more politically engaged will have more extreme responses (Barber and Pope 2018; Jennings 1992). Figure 5 shows ideal point distributions by quartiles of a survey question asking respondents to evaluate their level of self-reported campaign interest (on a 0–100 scale), with dashed lines indicating the median of Democratic, Independent, and Republican subgroups, respectively. The lowest two quartiles look very similar, with some separation between the parties and more respondents in the moderate space between the two peaks (Q1 Democratic median = −0.47, Republican median = 0.93). For the highest two quartiles, the median value for each party becomes more extreme, and the trough between parties becomes more pronounced, indicating more ideological division among those who have high levels of political interest (Q4 Democratic median = −0.75, Republican median = 1.22).

Figure 5:

Distribution of ideal points by political interest.

Kernel density plots of the ideal points in Figure 1, separated by respondents’ campaign interest. Dashed vertical lines indicate median values for Democratic, Independent, and Republican subgroups, respectively. Interest measured using a self-reported thermometer rating, divided into quartiles. Q1: <50, Q2: 50–67, Q3: 67–88, Q4: >88.

Further demographic correlations by race, gender, and age, are available in the Online Appendix.

An additional advantage of debate reaction ideal point estimation is the ability to disaggregate ideal point estimation by policy subtopics of the debate statements, which allows for estimation of policy preferences for specific policy areas. This is beneficial for both practical and theoretical purposes.

Practically speaking, it is possible that estimating ideal points using the full range of policy and non-policy statements in the debate could overestimate the number of ideological moderates due to averaging across policy domains. In this case, people with debate reaction ideal points near zero may not be those who prefer traditionally moderate or centrist policies, but rather those who have a strong liberal position in some issue areas and a strong conservative position in other areas (Ahler and Broockman 2018). On the other hand, it could underestimate the number of ideological moderates if respondents react strongly or more frequently to partisan, non-policy cues.

Theoretically, one of the challenges in conceptualizing and estimating ideology is that people rarely have ideologically principled logic for holding particular policy beliefs that remain consistent across a variety of issue areas (Converse 1964; Kinder and Kalmoe 2017). While a lengthy survey could plumb the depths of opinion on one or more issue areas, that is a cognitively intensive task that stretches the limits of respondents willingness and ability to respond. By contrast, the nature of the high-frequency real-time reaction to debate statements allows us to estimate a separate ideology measure for different policy areas with lower demand on the respondent (Boydstun et al. 2014b). The estimation of separate ideal points by topic area allows for exploration of ideological principles that are more limited in scope. While such topic-specific estimation has been conducted for political actors (Gerrish and Blei 2012; Lauderdale and Clark 2014), there is little work extending the practice to the general public.

To account for the practical possibilities of bias due to non-policy statements in the debate, and as proof of concept for the single-topic ideal points, we estimate three separate ideal points based on issue area. We categorized a single primary topic for each debate statement using the Policy Agendas coding framework, and aggregated into three subsets of policy statements: Domestic Policy, Economy, and Foreign Policy (coding details in the Online Appendix). Figure 6 displays the distribution of ideal points based only on the statements made in each topic area.

Figure 6:

Topic specific ideal point distributions.

Kernel density plots of the ideal points estimated using only the subtopics of policy statements. Domestic: 2636 respondents, 148 statements; Economy: 2726 respondents, 125 statements; Foreign: 2000 respondents, 178 statements.

The distribution of domestic and economic policy-specific ideal points is very similar to the overall distribution, indicating that concerns about bias from non-policy statements are minimal. The distribution is noticeably less polarized when estimated using only statements about foreign policy, a result reinforced by a “dip” test for bimodality (Hartigan 1985). We are able to reject the null hypothesis of unimodality for both Domestic and Economic policy (p-value < 0.001 in both cases). We fail to reject a null of unimodality in the Foreign case (p-value 0.10). This is consistent with prior conclusions that although foreign policy and defense attitudes are increasingly aligned with party positions, sorting on foreign policy attitudes occurred later and to a lesser extent than other issues (Fiorina and Abrams 2008).

5 Popularity and Polarity of Debate Statements

As a further validation of the model specification, we examine the results of two additional parameters that are estimated in the Bayesian ideal point model: popularity (discrimination parameter) and polarity (difficulty parameter). Rather than describing the people who react to the statements, these metrics use the aggregate patterns of reactions to the statements to characterize features of the debate statements themselves. Popularity indicates how likely all respondents are to agree with the statement, and is higher if respondents from across the ideal point space agree with the statement. Polarity represents how strongly a statement splits respondents, and therefore represents the conservative or liberal extremity of a speaker’s statement. Statements that a range of respondents agree with or disagree with will have lower polarity; statements that are primarily agreed with by respondents on one end of the policy space and disagreed with by respondents on the other end of the policy space will have high polarity scores. These metrics provide an opportunity to examine the content of the statements, and ensure that the model output conforms to qualitative expectations about the kinds of statements that would be more (or less) popular or distinguish between liberal and conservative voters.

Table 1 shows the average polarity and popularity for the statements made by each speaker, across all debates. As expected, the Republican ticket has a more positive (conservative) polarity and the Democratic ticket has a more negative (liberal) polarity. On both tickets, the presidential candidates made statements that were, on average, more ideologically extreme than the vice-presidential candidates. Notably, the moderator had an average polarity very close to zero, and approximately equidistant from the two slates of candidates, demonstrating the moderators competently managed the debates in an unbiased manner.

Obama had the highest average popularity, while Ryan was lowest. While this is expected due to the somewhat more liberal nature of the sample, it also conforms to expectations based on public perceptions and media discourse regarding the candidates’ debate performances.^[8] The full text of the 10 highest and lowest popularity and polarity statements are in the Online Appendix.

6 Conclusion

We have introduced a new measure of individual ideology that uses real-time reactions to political debates as the basis for ideal point estimation. Debate reaction ideal points have the advantage of providing robust, continuous scales that yield more information than traditional limited scales of self-reported ideology, and provide the flexibility to estimate based on a content subset. Using reactions to a live debate, the estimation is grounded in real-world policy content and issue presentation that directly reflects the current political context. Being real time, it reduces potential biases introduced by conscious reflection, satisficing, or question wording. Additionally, respondents can choose which statements to respond to and which to ignore, so the method can cover the broad range of issues in a debate, avoid researcher influence about what survey questions to ask, and distinguish lack of interest from moderate positions.

We find that more partisan and more interested study participants are more ideologically extreme. Unless they intentionally account for how to gather meaningful attitudinal data from less engaged citizens, estimates of public ideology are likely to overstate levels of polarization in the mass public, precisely because such practices remove an ideologically moderate segment of the American population. Estimation strategies, such as the one presented here, that provide a meaningful way to measure ideology for the least engaged are essential for accurate portrayal of ideology and polarization in the American public.

An important contribution of this method is the ability to use subsets of the debate statements to estimate ideal points within separate policy areas. This allows for a more robust understanding of how ideology varies across issue areas in the United States. As expected based on public opinion research, we find that attitudes about foreign policy issues are less polarized than attitudes about domestic and economic policy.

The biggest challenge facing widespread use of debate reaction ideal points is data collection. While our own available data limit the present study to the 2012 election, real-time reactions data are increasingly common and facilitated by innovations in virtual technology (Maier, Hampe, and Jahn 2016; Waldvogel 2020; Waldvogel and Metz 2020). Future research should consider ways to harness real-time reactions already conducted by a variety of news media outlets, to link real-time reaction data with other sources of public opinion information, or to integrate real-time reaction measures into surveys or other platforms. While we believe there are benefits to response data collected synchronously from debates in real time, before pundits have a chance to influence viewers’ interpretations (Fridkin et al. 2008) or political priorities shift (Boydstun, Glazier, and Pietryka 2013), future researchers may find applications for response data based on debate recordings or other stimuli that can be provided on-demand through survey platforms or in a lab. The ideal point estimation method described here is flexible enough for application to any reaction data, although the nature of the stimulus material may change the substantive meaning of the ideal points.

Additionally, linking real-time reactions with other estimates of public ideology (e.g. Twitter or campaign donations) would provide useful insights into the validity of these approaches. Such innovations promise to enhance the estimation of public ideology. Repetition of debate reaction measurement over time would be particularly valuable, as it would allow for examination of changes in ideology over time and across respondent characteristics. Continued use of this estimation strategy may provide valuable insight into the nature and extent of political polarization in the mass public.