Assessing Robustness of Findings About Racial Redistricting’s Effect on Southern House Delegations

Appendix: Registered Research Design

On July 9, 2015, before beginning any analysis, we pre-registered our study with EGAP, under the project title “Assessing Robustness of Findings About Racial Redistricting and Southern House Members” (ID #20150709AA). What follows is the content of that document.

Background information: Shotts (2003a) analyzed the effect of racial redistricting in the 1980s and 1990s, using a dependent variable that was measured using DW-Nominates. Simons and Mallinson (2015) extended this analysis to include more recent Congresses. Along the way they replicated Shotts (2003a). To do this, they used the current DW-Nominate estimates, and the key result in Shotts (2003a) no longer holds (i.e. is no longer statistically significant). The reason for the discrepancy is that as more votes happen over the years, each representative’s DW-Nominate score for previous years moves a bit. The Shotts (2003a) dependent variable – the proportion of members of a state’s delegation to the left of the House median – apparently was sensitive to small movements in DW-Nominate estimates for individuals near the median.

Our goal is to lay out simple tools that can be used to assess whether Shotts’s results are robust to a small amount of error in the DW-Nominates. The motivation for our analysis is methodological, i.e. to identify things that an applied researcher like Shotts (or Simons and Mallinson) could and should have done to take into account the possibility of measurement error.

Our analysis will use two different techniques.

The first technique is reestimation of the Shotts (2003a) model using data produced from Monte Carlo simulations, allowing for errors in DW-Nominates. This technique will enable us to check whether the results in Shotts (2003a) are robust to reasonable levels of error. This method is far from perfect, because errors in DW-Nominates presumably are correlated both across time and across members. However, to the best of our knowledge, no technique has been developed that would deal with this issue in DW-Nominate error. Hence, we will treat the errors as being independent.

The second technique is to use a different measure of the ideology of elected officials, based on the Bonica (2014) estimates.

Most of the input data for this study already exist and are readily available. All data used in Shotts’s original study are available in his replication archive. Current DW-Nominate estimates, along with estimated standard errors are available from Keith Poole’s website. The Bonica scores are available from his website. Simons and Mallinson have graciously provided the data they used in their extension of Shotts.

As of the time that this document is being submitted to EGAP, we have not conducted any analysis of the data.

Part 1: Re-Analysis of Shotts (2003a)

1-1. Plot of raw DW-Nominates. To give a sense of the basic data before and after the 1990s redistricting, we will plot the Old and Current DW-Nominates for Southern Representatives and the House Median in the 102nd, 103rd, and 104th Congresses. This will give a rough sense of whether the count of the number of Southern representatives to the left of the median is sensitive to errors in measurement for DW-Nominates.

1-2. Reestimation of the original model using samples obtained from Monte Carlo simulations that allow for different levels of error in Old-DW-Nominates (i.e. the ones used in Shotts 2003a). We will perform regressions for Shotts’s key empirical specification (Table 1, p. 221) in a fashion similar to bootstrap resampling procedures. All variables except the dependent variable Left of Median and its lag will be exactly as in Shotts (2003a). For the Left of Median variable, each iteration of the simulation will draw for each House Member in each year a DW-Nominate score from a normal distribution, with mean at his/her estimated mean. The standard deviation of the draws will range from 0 to 0.2, in 0.01 unit increments. Based on the work of Carroll et al. (pp. 270–271) we believe that the most relevant standard deviation will be one around 0.05. However, we are examining smaller and larger values as well.

We will perform 1000 iterations for each standard deviation between 0 and 0.2. The product of this analysis will be a figure. On the horizontal axis will be the standard deviation. On the vertical axis will be the proportion of the 1000 iterations that resulted in a statistically-significant (at the conventional 0.05 level) positive coefficient on the key variable of interest in Shotts (2003a), namely Fraction Majority-Minority. We also will plot the proportion of iterations that resulted in a statistically-significant negative coefficient on Fraction Majority-Minority, and report the mean and standard error from our iteration estimates.

Given that Shotts’s results were no longer statistically significant in Simons and Mallinson’s analysis (which used DW-Nominates that they downloaded recently from Poole’s website), we expect that a small amount of noise will dramatically reduce the proportion of the time that there is a statistically-significant positive coefficient on Fraction Majority-Minority. However, to be clear, we do not have strong priors on how large the effect will be for different levels of the standard deviation.

We do not expect that there will be many statistically-significant negative coefficients on Fraction Majority-Minority, at least for small standard deviations.

1-3. Monte Carlo using Carroll et al. (2009) estimated standard errors in Current-DW-Nominates for individual members. This analysis will be very similar to part 1-2. However, there will be only one level of standard deviation in the DW-Nominate for each member of Congress. This will vary across members. Also, the DW-Nominates that we will use will be Current-DW-Nominates, because those are the ones for which estimated standard errors are available.

We will conduct 1000 iterations of the simulated regression, and will report the proportion of the time that the coefficient on Fraction Majority-Minority was positive and statistically significant and the proportion of the time that it was negative and statistically significant at the conventional 0.05 level.

1-4. Using Bonica scores to measure Left of Median. Another, very different, way of assessing the robustness of Shotts’s results is to use a different measure of politicians’ ideal points. We will do this using Bonica’s (2014) measures, which are based on interest group contributions. For this analysis we will replicate Table 1 from Shotts (2003a), using the same data for every variable except Fraction Left of Median and its lag, both of which will be calculated using Bonica’s estimates of members’ ideological positions.

Part 2: Application to Simons and Mallinson

2-1. Monte Carlo simulation for different levels of error in Current-DW-Nominates. As above, in part 1-2, we will re-run Simons and Mallinson’s model (the “Conditional Effects Model” in Table 3 of the March 2015 version of their paper) using a Monte Carlo Simulation with the standard deviation of the DW-Nominate estimates allowed to vary from 0 to 0.2 in 0.01 unit increments. For each of the key independent variables (Fraction Majority-Minority, Unified Republican Control, Bipartisan Control, Majority-MinorityX Unified Republican, and Majority-Minority X Bipartisan), we will plot the proportion of iterations that resulted in a statistically-significant positive coefficient as well as the proportion that resulted in a statistically-significant negative coefficient.

2-2. Monte Carlo simulation using Carroll et al. (2009) estimated standard errors. As in part 1-3, we will re-run Simons and Mallinson’s model using a simulation with estimated standard errors for Current-DW-Nominates. For each of the key independent variables, we will report the proportion of iterations in which the coefficient was positive and statistically significant as well as the proportion in which it was negative and statistically significant.