Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter November 17, 2015

Assessing Robustness of Findings About Racial Redistricting’s Effect on Southern House Delegations

Carlos A. Sanchez-Martinez and Kenneth W. Shotts


We assess whether racial redistricting increases the number of Southern representatives to the left of the US House median. Our results, which are based on Monte Carlo simulations and an alternative measure of representatives’ preferences, are generally null findings. The data do not support the claim that racial redistricting promotes liberal policy outcomes; nor do they support the claim that it promotes conservative policy outcomes. At a methodological level, we suggest techniques that researchers can use to assess how robust their findings are to noise in variables based on estimated values like DW-Nominate scores.

Corresponding author: Kenneth W. Shotts, David S. and Ann M. Barlow Professor of Political Economy at the Stanford Graduate School of Business, Stanford, CA, USA, e-mail:

Appendix: Registered Research Design

On July 9, 2015, before beginning any analysis, we pre-registered our study with EGAP, under the project title “Assessing Robustness of Findings About Racial Redistricting and Southern House Members” (ID #20150709AA). What follows is the content of that document.

Background information: Shotts (2003a) analyzed the effect of racial redistricting in the 1980s and 1990s, using a dependent variable that was measured using DW-Nominates. Simons and Mallinson (2015) extended this analysis to include more recent Congresses. Along the way they replicated Shotts (2003a). To do this, they used the current DW-Nominate estimates, and the key result in Shotts (2003a) no longer holds (i.e. is no longer statistically significant). The reason for the discrepancy is that as more votes happen over the years, each representative’s DW-Nominate score for previous years moves a bit. The Shotts (2003a) dependent variable – the proportion of members of a state’s delegation to the left of the House median – apparently was sensitive to small movements in DW-Nominate estimates for individuals near the median.

Our goal is to lay out simple tools that can be used to assess whether Shotts’s results are robust to a small amount of error in the DW-Nominates. The motivation for our analysis is methodological, i.e. to identify things that an applied researcher like Shotts (or Simons and Mallinson) could and should have done to take into account the possibility of measurement error.

Our analysis will use two different techniques.

The first technique is reestimation of the Shotts (2003a) model using data produced from Monte Carlo simulations, allowing for errors in DW-Nominates. This technique will enable us to check whether the results in Shotts (2003a) are robust to reasonable levels of error. This method is far from perfect, because errors in DW-Nominates presumably are correlated both across time and across members. However, to the best of our knowledge, no technique has been developed that would deal with this issue in DW-Nominate error. Hence, we will treat the errors as being independent.

The second technique is to use a different measure of the ideology of elected officials, based on the Bonica (2014) estimates.

Most of the input data for this study already exist and are readily available. All data used in Shotts’s original study are available in his replication archive. Current DW-Nominate estimates, along with estimated standard errors are available from Keith Poole’s website. The Bonica scores are available from his website. Simons and Mallinson have graciously provided the data they used in their extension of Shotts.

As of the time that this document is being submitted to EGAP, we have not conducted any analysis of the data.

Part 1: Re-Analysis of Shotts (2003a)

1-1. Plot of raw DW-Nominates. To give a sense of the basic data before and after the 1990s redistricting, we will plot the Old and Current DW-Nominates for Southern Representatives and the House Median in the 102nd, 103rd, and 104th Congresses. This will give a rough sense of whether the count of the number of Southern representatives to the left of the median is sensitive to errors in measurement for DW-Nominates.

1-2. Reestimation of the original model using samples obtained from Monte Carlo simulations that allow for different levels of error in Old-DW-Nominates (i.e. the ones used in Shotts 2003a). We will perform regressions for Shotts’s key empirical specification (Table 1, p. 221) in a fashion similar to bootstrap resampling procedures. All variables except the dependent variable Left of Median and its lag will be exactly as in Shotts (2003a). For the Left of Median variable, each iteration of the simulation will draw for each House Member in each year a DW-Nominate score from a normal distribution, with mean at his/her estimated mean. The standard deviation of the draws will range from 0 to 0.2, in 0.01 unit increments. Based on the work of Carroll et al. (pp. 270–271) we believe that the most relevant standard deviation will be one around 0.05. However, we are examining smaller and larger values as well.

We will perform 1000 iterations for each standard deviation between 0 and 0.2. The product of this analysis will be a figure. On the horizontal axis will be the standard deviation. On the vertical axis will be the proportion of the 1000 iterations that resulted in a statistically-significant (at the conventional 0.05 level) positive coefficient on the key variable of interest in Shotts (2003a), namely Fraction Majority-Minority. We also will plot the proportion of iterations that resulted in a statistically-significant negative coefficient on Fraction Majority-Minority, and report the mean and standard error from our iteration estimates.

Given that Shotts’s results were no longer statistically significant in Simons and Mallinson’s analysis (which used DW-Nominates that they downloaded recently from Poole’s website), we expect that a small amount of noise will dramatically reduce the proportion of the time that there is a statistically-significant positive coefficient on Fraction Majority-Minority. However, to be clear, we do not have strong priors on how large the effect will be for different levels of the standard deviation.

We do not expect that there will be many statistically-significant negative coefficients on Fraction Majority-Minority, at least for small standard deviations.

1-3. Monte Carlo using Carroll et al. (2009) estimated standard errors in Current-DW-Nominates for individual members. This analysis will be very similar to part 1-2. However, there will be only one level of standard deviation in the DW-Nominate for each member of Congress. This will vary across members. Also, the DW-Nominates that we will use will be Current-DW-Nominates, because those are the ones for which estimated standard errors are available.

We will conduct 1000 iterations of the simulated regression, and will report the proportion of the time that the coefficient on Fraction Majority-Minority was positive and statistically significant and the proportion of the time that it was negative and statistically significant at the conventional 0.05 level.

1-4. Using Bonica scores to measure Left of Median. Another, very different, way of assessing the robustness of Shotts’s results is to use a different measure of politicians’ ideal points. We will do this using Bonica’s (2014) measures, which are based on interest group contributions. For this analysis we will replicate Table 1 from Shotts (2003a), using the same data for every variable except Fraction Left of Median and its lag, both of which will be calculated using Bonica’s estimates of members’ ideological positions.

Part 2: Application to Simons and Mallinson

2-1. Monte Carlo simulation for different levels of error in Current-DW-Nominates. As above, in part 1-2, we will re-run Simons and Mallinson’s model (the “Conditional Effects Model” in Table 3 of the March 2015 version of their paper) using a Monte Carlo Simulation with the standard deviation of the DW-Nominate estimates allowed to vary from 0 to 0.2 in 0.01 unit increments. For each of the key independent variables (Fraction Majority-Minority, Unified Republican Control, Bipartisan Control, Majority-MinorityX Unified Republican, and Majority-Minority X Bipartisan), we will plot the proportion of iterations that resulted in a statistically-significant positive coefficient as well as the proportion that resulted in a statistically-significant negative coefficient.

2-2. Monte Carlo simulation using Carroll et al. (2009) estimated standard errors. As in part 1-3, we will re-run Simons and Mallinson’s model using a simulation with estimated standard errors for Current-DW-Nominates. For each of the key independent variables, we will report the proportion of iterations in which the coefficient was positive and statistically significant as well as the proportion in which it was negative and statistically significant.


Bonica, A. (2014) “Mapping the Ideological Marketplace,” American Journal of Political Science, 58:367–386.10.1111/ajps.12062Search in Google Scholar

Cameron, C., D. Epstein and S. O’Halloran (1996) “Do Majority-Minority Districts Maximize Substantive Black Representation in Congress?,” American Political Science Review, 90:794–812.10.2307/2945843Search in Google Scholar

Carroll, R., J. B. Lewis, J. Lo, K. T. Poole and H. Rosenthal (2009) “Measuring Bias and Uncertainty in DW-NOMINATE Ideal Point Estimates via the Parametric Bootstrap,” Political Analysis, 17:261–275.10.1093/pan/mpp005Search in Google Scholar

Clinton, J., S. Jackman and D. Rivers (2004) “The Statistical Analysis of Roll-Call Data,” American Political Science Review, 98:355–370.10.1017/S0003055404001194Search in Google Scholar

Grofman, B., R. Griffin and A. Glazer (1992) “The Effect of Black Population on Electing Democrats and Liberals to the House of Representatives,” Legislative Studies Quarterly, 17:365–79.10.2307/439735Search in Google Scholar

Krehbiel, K. and Z. Peskowitz (2015) “Legislative Organization and Ideal-Point Bias,” Journal of Theoretical Politics, 27:673–703.10.1177/0951629814562290Search in Google Scholar

Lublin, D. I. (1997) The Paradox of Representation. Princeton, New Jersey: Princeton University Press.Search in Google Scholar

Lublin, D. and D. S. Voss (2003) “The Missing Middle: Why Median-Voter Theory Can’t Save Democrats From Singing the Boll-Weevil Blues,” Journal of Politics, 65:227–237.10.1111/1468-2508.t01-1-00011Search in Google Scholar

Nakao, K. (2011) “Racial Redistricting for Minority Representation Without Partisan Bias: A Theoretical Approach,” Economics and Politics, 23:132–151.10.1111/j.1468-0343.2010.00379.xSearch in Google Scholar

Poole, K. T. and H. Rosenthal (1997) Congress: A Political-Economic History of Roll Call Voting. New York, New York: Oxford University Press.Search in Google Scholar

Poole, K. T. and H. Rosenthal (2001) “D-NOMINATE After 10 Years: An Update to Congress: A Political-Economic History of Roll Call Voting,” Legislative Studies Quarterly, 26:5–29.10.2307/440401Search in Google Scholar

Shotts, K. W. (2001) “The Effects of Majority-Minority Mandates on Partisan Gerrymandering,” American Journal of Political Science, 54:120–135.10.2307/2669363Search in Google Scholar

Shotts, K. W. (2002) “Gerrymandering, Legislative Composition, and National Policy Outcomes,” American Journal of Political Science, 46:398–414.10.2307/3088384Search in Google Scholar

Shotts, K. W. (2003a) “Does Racial Redistricting Cause Conservative Policy Outcomes? Policy Preferences of Southern Representatives in the 1980s and 1990s,” Journal of Politics, 65:216–226.10.1111/1468-2508.t01-1-00010Search in Google Scholar

Shotts, K. W. (2003b) “Racial Redistricting’s Alleged Perverse Effects: Theory, Data, and ‘Reality,’” Journal of Politics, 65:238–243.10.1111/1468-2508.t01-1-00012Search in Google Scholar

Simons, J. and D. J. Mallinson (2015) “Party Control and Perverse Effects in Majority-Minority Districting: Replication Challenges When Using DW-Nominate,” Statistics, Politics, and Policy, 6:19–37.10.1515/spp-2014-0005Search in Google Scholar

Washington, E. (2012) “Do Majority Black Districts Limit Blacks’ Representation? The Case of the 1990s Redistricting,” Journal of Law and Economics, 55:251–274.10.1086/661991Search in Google Scholar

Article note:

We thank Joseph Simons and Daniel Mallinson for graciously sharing their data with us and for engaging in a collegial and productive scholarly exchange. Any omissions or errors are our own. Part of Ken Shotts’s work on this project was completed during a sabbatical at the Center for Advanced Study in the Behavioral Sciences at Stanford University. Replication data and code are available at

Published Online: 2015-11-17
Published in Print: 2015-12-1

©2015 by De Gruyter