Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter January 11, 2013

Risk-limiting Audits and the Margin of Victory in Nonplurality Elections

Anand D. Sarwate, Stephen Checkoway and Hovav Shacham

Abstract

Post-election audits are an important method for verifying the outcome of an election. Recent work on risk-limiting, post-election audits has focused almost exclusively on plurality elections. Several organization and municipalities use nonplurality methods such as range voting, the Borda count, and instant-runoff voting (IRV). We believe that it is crucial to develop effective methods of performing risk-limiting, post-election audits for these methods. We define a general notion of the margin of victory and develop risk-limiting auditing procedures for these nonplurality methods. For scored systems, we show how to adapt methods from plurality auditing. For IRV, the situation is markedly different. We provide a risk-limiting method for auditing the candidate elimination order. We provide a more efficient audit for the elections in which the margin of the IRV election can be efficiently calculated or bounded. We provide efficiently computable upper and lower bounds on the margin and, where possible, compare them to the exact margins for a large number of real elections.


Corresponding author: Anand D. Sarwate, Toyota Technological Institute at Chicago, 6045 S. Kenwood Ave, Chicago, IL 60637, USA

We thank Eric Rescorla and Joseph Lorenzo Hall for numerous helpful discussions. Philip Stark for suggesting a universal definition of margin of victory, Mike LaBonte for providing Aspen election data and Michael Byrne, David Cary, Stephen Goggin, Thomas R. Magrino, Ronald L. Rivest, Emily Shen, and David Wager for sharing advance copies of their papers. The manuscript improved greatly from the careful reading and comments of the reviewers. We thank the editor, Justin H. Gross, for his extreme patience during our revision process.

This material is based upon work supported by National Science Foundation under grants CNS-0831532 and CNS-0963702 and by the MURI program under AFOSR Grant No. FA9550-08-1-0352.

  1. 1
  2. 2

    This is Hare’s rule for ballot transfers (Tideman 1995).

  3. 3

    San Francisco allows voters to rank no more than three of the candidates for each race.

  4. 4

    In a presentation at the EVN 2011 conference, Emily Shen gave another such example.

  5. 5

    A simpler form of auditing simply recounts ballots to confirm the winner, called a ballot polling audit in Lindeman and Stark (2012).

  6. 6

    The remainder of this subsection is adapted from the authors’ earlier work on risk-limiting, post-election audits (Checkoway, Sarwate, and Shacham, 2010).

  7. 7

    Intermediate sub-precinct audit units, such as individual voting machines, appear to provide littlegain in statistical power, but may reduce the cost of locating the ballots to audit.

  8. 8

    This is not without loss of generality—if more information is known about the reported margins,more targeted sampling can be more efficient (Stark 2009b).

  9. 9

    To use ɛ in Definition 3, the ballots must first be converted from ordered lists to pairs of scores: (1,2), (1) → (1,0); (2,1), (2) → (0,1); and ( ) → (0,0) which is to say that only the top-ranked candidate on the ballot gets a score of 1.

  10. 10

    S.F., Cal., Charter art. XIII, § 13.102(e) (Mar. 2002), “If the total number of votes of the two or more candidates credited with the lowest number of votes is less than the number of votes credited to the candidate with the next highest number of votes, those candidates with the lowest number of votes shall be eliminated simultaneously and their votes transferred to the next-ranked continuing candidate on each ballot in a single counting operation.”

  11. 11

    For example, with the base IRV elimination rule, if the two candidates with the fewest number of top-choice votes in a round have the same number of votes, then the candidate to be eliminated may be chosen by some other mechanism such as a coin flip.

  12. 12
  13. 13

    All of our code is available at https://www.cs.jhu.edu/∼s/elections/irv.html.

  14. 14

    A priority queue is an abstract data type which is conceptually a set of elements each of which has an associated priority. Common implementations of priority queues support fast insertion of a new element with arbitrary priority and fast removal of the element with the highest priority.

  15. 15

    Cf., Ala. Code §17-16-20 (2010) or Fla. Stat. §102.141 (2010).

  16. 16

    I. D. Hill describes a slightly different example of instability in a real Single Transferable Vote election—the multiseat analogue of instant-runoff voting. Hill points out that a change in a single ballot’s 15th choice (out of 23) would result in a different winner. In this case, it was the difference between voting for one of the (eventual) winners and the closest runner up rather than between two losers (Hill 2004).

References

Arrow, K. J. 1951. Social Choice and Individual Values. Yale University Press, second edition.Search in Google Scholar

Aslam, J. A., R. A. Popa, and R. L. Rivest. 2007. “On Estimating the Size and Confidence of a Statistical Audit.” Martinez, R. and D. Wagner, eds. Proceedings of EVT 2007, USENIX and ACCURATE. 8–8.Search in Google Scholar

Aslam, J. A., R. A. Popa, and R. L. Rivest. 2008. “On Auditing Elections When Precincts Have Different Sizes.” in D. Dill and T. Kohno, eds., Proceedings of EVT 2008, USENIX and ACCURATE, 1–1.Search in Google Scholar

Australian Electoral Commission 2011. “Counting the Votes – House of Representatives.” (http://www.aec.gov.au/Voting/counting/vid_hor.htm).Search in Google Scholar

Baldwin, J. M. 1926. “The Technique of the Nanson Preferential Majority System of Election.” Proceedings of the Royal Society of Victoria 39:42–52.Search in Google Scholar

Bartholdi III, J. J. and J. B. Orlin. 1991. “Single Transferable Vote Resists Strategic Voting.” Social Choice and Welfare 8:341–54.Search in Google Scholar

Bartholdi III, J. J., C. A. Tovey, and M. A. Trick. 1989. “The Computational Difficulty of Manipulating an Election.” Social Choice and Welfare 6:227–241.Search in Google Scholar

Bartholdi III, J. J., C. A. Tovey, and M. A. Trick. 1992. “How Hard is it to Control an Election.” Mathematical Computer Modelling 16:27–40 (http://dx. doi.org/10.1016/0895-7177(92)90085-Y).Search in Google Scholar

Benaloh, J., D. Jones, E. L. Lazarus, M. Lindeman, and P. B. Stark. 2011. “SOBA: Secrecy-Preserving Observable Ballot-Level Audit.”in Shacham, H. and V. Teague, eds. Proceedings of EVT/WOTE 2011, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Betzler, N., R. Niedermeier, and G. J. Woeginger. 2011. “Unweighted Coalitional Manipulation Under the Borda Rule is NP-hard.” in Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, 55–60 (http://ijcai.org/papers11/Papers/IJCAI11-021.pdf).Search in Google Scholar

Black, D. 1958. The Theory of Committees and Elections. Cambridge: Cambridge University Press.Search in Google Scholar

Brams, S. J. and P. C. Fishburn. 1983. Approval Voting. Cambridge, MA: Birkhäuser Boston.Search in Google Scholar

Brams, S. J. and P. C. Fishburn. 2005. “Going from Theory to Practice: The Mixed Success of Approval Voting.” Social Choice and Welfare 25:457–474 (http://dx.doi.org/10.1007/s00355-005-0013-y).Search in Google Scholar

Brelsford, E., P. Faliszewski, E. Hemaspaandra, H. Schnoor, and I. Schnoor. 2008. “Approximability of Manipulating Elections.” in D. Fox and C. P. Gomes, eds., AAAI, AAAI Press, 44–49.Search in Google Scholar

Calandrino, J. A., J. A. Halderman, and E. W. Felten. 2007. “Machine-Assisted Election Auditing.” in Martinez, R. and D. Wagner, eds. Proceedings of EVT 2007, USENIX and ACCURATE.Search in Google Scholar

Checkoway, S., A. Sarwate, and H. Shacham. 2010. “Single-Ballot Risk-Limiting Audits Using Convex Optimization.” in Jones, D., J.-J. Quisquater, and E. Rescorla, eds. Proceedings of EVT/WOTE 2010, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Conitzer, V., T. Sandholm, and J. Lang. 2007. “When are Elections with Few Candidates Hard to Manipulate?” Journal of the ACM, 54 (http://dx. doi.org/10.1145/1236457.1236461).Search in Google Scholar

Consortium for Elections and Political Process Strengthening 2011. “ElectionGuide.” (http://www.electionguide.org/).Search in Google Scholar

Davies, J., G. Katsirelos, N. Narodytska, and T. Walsh. 2011. “Complexity of and Algorithms for Borda Manipulation.” in Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence.Search in Google Scholar

Dopp, K. 2008. “History of Confidence Election Auditing Development (1975 to 2008) & Overview of Election Auditing Fundamentals.” (http://electionmathematics.org/ucvAnalysis/US/paperaudits/History-of-Election-Auditing-Development.pdf, online).Search in Google Scholar

Doron, G. and R. Kronick. 1977. “Single Transferable Vote: An Example of a Perverse Social Choice Functioned.” American Journal of Political Science 21:303–311. (http://www.jstor.org/stable/2110496).Search in Google Scholar

Dummett, M. 1984. Voting Procedures, Clarendon: Oxford.Search in Google Scholar

Faliszewski, P., E. Hemaspaandra, and L. A. Hemaspaandra. 2010. “Using Complexity to Protect Elections.” Communications of the ACM 53:74–82. (http://dx.doi.org/10.1145/1839676.1839696).Search in Google Scholar

Faliszewski, P., E. Hemaspaandra, and L. A. Hemaspaandra. 2011. “Multimode Control Attacks on Elections.” Journal of Artificial Intelligence Research 40:305–351. (http://dx.doi.org/10.1613/jair.3136).Search in Google Scholar

Farvaque, E., H. Jayet, and L. Ragot. 2009. “A “Winner” Under Any Voting Rule ? an Experiment on the Single Transferable Vote.” Documents de travail du Centre d’Economie de la Sorbonne 09067, Université Panthéon-Sorbonne (Paris 1), Centre d’Economie de la Sorbonne.Search in Google Scholar

Fishburn, P. C. 1977. “Condorcet Social Choice Functions.” SIAM Journal on Applied Mathematics 33:469–489. (http://dx.doi.org/10.1137/0133030).Search in Google Scholar

Friedgut, E., G. Kalai, and N. Nisan. 2008. “Elections Can be Manipulated Often.” in FOCS, IEEE Computer Society 243–249. (http://dx.doi.org/10.1109/FOCS.2008.87).Search in Google Scholar

Gardenfors, P. 1976. “Manipulation of Social Choice Functions.” Journal of Economic Theory 13:217–28.Search in Google Scholar

Gibbard, A. 1973. “Manipulation of Voting Schemes: A General Result.” Econometrica 41:587–601.Search in Google Scholar

Higgins, M. J., R. L. Rivest, and P. B. Stark. 2011. “Sharper p-Values for Stratified Election Audits.” Statistics, Politics, and Policy 2. (http://dx.doi.org/10.2202/2151-7509.1031).Search in Google Scholar

Hill, I. D. 2004. “An Odd Feature in a Real STV Election.” Voting Matters 18:9.Search in Google Scholar

Isaksson, M., G. Kindler, and E. Mossel. 2010. “The Geometry of Manipulation: A Quantitative Proof of the Gibbard-Satterthwaite Theorem.” in FOCS, IEEE Computer Society 319–328. (http://dx.doi.org/).Search in Google Scholar

Johnson, K. C. 2004. “Election Certification by Statistical Audit of Voter-Verified Paper Ballots.” (http://dx.doi.org/10.2139/ssrn.640943, sSRN Working Paper).Search in Google Scholar

Jones, D., J.-J. Quisquater, and E. Rescorla, eds. 2010. Proceedings of EVT/WOTE 2010, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Kaplan, H. M. 1987. “A Method of One-Sided Nonparametric Inference for the Mean of a Nonnegative Population.” The American Statistician 41:157–158. (http://dx.doi.org/10.1080/00031305.1987.10475470).Search in Google Scholar

Lindeman, M. and P. B. Stark. 2012. “A Gentle Introduction to Risk-Limiting Audits.” IEEE Security and Privacy. (http://dx.doi.org/10.1109/MSP.2012.56).Search in Google Scholar

Magrino, T. R., R. L. Rivest, E. Shen, and D. Wagner. 2011. “Computing the Margin of Victory in IRV elections.” in Shacham, H. and V. Teague, eds. Proceedings of EVT/WOTE 2011, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Martinez, R. and D. Wagner, eds. 2007. Proceedings of EVT 2007, USENIX and ACCURATE.Search in Google Scholar

Nanson, E. J. 1882. “Methods of election.” Transactions and Proceedings of the Royal Society of Victoria 18:197–240.Search in Google Scholar

Neff, C. A. 2003. “Election Confidence: A Comparison of Methodologies and their Relative Effectiveness at Achieving it (revision 6).” (www.verifiedvoting.org/downloads/20031217.neff.electionconfidence.pdf).Search in Google Scholar

O’Neill, J. 2006. “Fast Algorithms for Counting Ranked Ballots.” Voting Matters 21:1–5.Search in Google Scholar

Poundstone, W. 2008. Gaming the Vote: Why Elections Aren’t Fair (and What We Can Do About It). New York: Hill and Wang. ISBN 978-0-8090-4892-2.Search in Google Scholar

Procaccia, A. D. and J. S. Rosenschein. 2007. “Junta Distributions and the Averagecase Complexity of Manipulating Elections.” Journal of Artificial Intelligence Research 28:157–181. (http://dx.doi.org/10.1613/jair.2148).Search in Google Scholar

Reilly, B. 2002. “Social Choice in the South Seas: Electoral Innovation and the Borda Count in the Pacific Island Countries.” International Political Science Review 23:355–72.Search in Google Scholar

Roberts, S. 2010. “Minor Parties See Threat in Ballot Quirk.” The New York Times, October 26, A24.Search in Google Scholar

Saari, D. 1995. Basic Geometry of Voting, Berlin: Springer.Search in Google Scholar

Saari, D. G. 2001. Decisions And Elections: Explaining The Unexpected. Cambridge, UK: Cambridge University Press.Search in Google Scholar

Saltman, R. G. 1975. “Effective use of computing technology in vote-tallying.” Technical Report Tech. Rep. NBSIR 75–687, National Bureau of Standards (Information Technology Division), Washington, D.C., USA. (URL http://csrc.nist.gov/publications/nistpubs/NBS_SP_500-30.pdf).Search in Google Scholar

San Francisco Voting Systems Task Force 2011. “Recommendations on Voting Systems for the City and County of San Francisco (draft).” (http://www.sfgov2.org/Modules/ShowDocument.aspx?documentid=155).Search in Google Scholar

Satterthwaite, M. A. 1975. “Strategy-Proofness and Arrow’s Conditions: Existence and Correspondence Theorems for Voting and Social Welfare Functions.” Journal of Economic Theory 10:187–217.Search in Google Scholar

Schulze, M. 1997. “Condorect Sub-Cycle Rule.” The Election-Methods Mailing List. (http://lists.electorama.com/pipermail/electionmethods-electorama.com/1997-October/001570.html).Search in Google Scholar

Schulze, M. 2011. “A New Monotonic, Clone-Independent, Reversal Symmetric, and Condorcet-Consistent Single-Winner Election Method.” Social Choice and Welfare 36:267–303. (http://dx.doi.org/10.1007/s00355-010-0475-4).Search in Google Scholar

Shacham, H. and V. Teague, eds. 2011. Proceedings of EVT/WOTE2011, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Stanislevic, H. 2006. “Random Auditing of E-Voting Systems: How Much is Enough?” (http://www.votetrustusa.org/pdfs/VTTF/EVEPAuditing.pdf).Search in Google Scholar

Stark, P. B. 2008a. “Conservative Statistical Post-Election Audits.” Annals of Applied Statistics 2:550–81. (http://dx.doi.org/10.1214/08-AOAS161).Search in Google Scholar

Stark, P. B. 2008b. “Election Audits by Sampling with Probability Proportional to an Error Bound: dealing with Discrepancies.” (http://www.stat.berkeley.edu/∼stark/Preprints/ppebwrwd08.pdf, draft).Search in Google Scholar

Stark, P. B. 2008c. “A Sharper Discrepancy Measure for Post-Election Audits.” Annals of Applied Statistics 2:982–985. (http://dx.doi.org/10.1214/08-AOAS171).Search in Google Scholar

Stark, P. B. 2009a. “Auditing a Collection of Races Simultaneously.” (URL http://arxiv.org/abs/0905.1422v1, arXiV Preprint arXiv:0905.1422v1 [stat.AP]).Search in Google Scholar

Stark, P. B. 2009b. “CAST: Canvass Audits by Sampling and Testing.” IEEE Transactions on Information Forensics and Security 4:708–717. (http://dx.doi.org/10.1109/TIFS.2009.2034210).Search in Google Scholar

Stark, P. B. 2009c. “Efficient Post-Election Audits of Multiple Contests: 2009 California tests.” in Proceedings of the 4th Annual Conference on Empirical Legal Studies. (http://dx.doi.org/10.2139/ssrn.1443314).Search in Google Scholar

Stark, P. B. 2009d. “Risk-Limiting Postelection Audits: Conservative P-Values from Common Probability Inequalities.” IEEE Transactions on Information Forensics and Security 4:1005–1014. (http://dx.doi.org/10.1109/TIFS.2009.2034190).Search in Google Scholar

Stark, P. B. 2010. “Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits.” in Jones, D., J.-J. Quisquater, and E. Rescorla, eds. Proceedings of EVT/WOTE 2010, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Sturton, C., E. Rescorla, and D. Wagner. 2009. “Weight, Weight, Don’t Tell Me: Using Scales to Select Ballots for Auditing,” in D. Jefferson, J. L. Hall, and T. Moran, eds., Proceedings of EVT/WOTE 2009, USENIX, ACCURATE, and IAVoSS.Search in Google Scholar

Szpiro, G. 2010. Numbers Rule: The Vexing Mathematics Of Democracy, From Plato To The Present. Princeton, NJ: Princeton University Press.Search in Google Scholar

The Heisman Memorial Trophy 2011. “Heisman Trophy Balloting.” (http://www.heisman.com/history/balloting.php).Search in Google Scholar

Tideman, N. 1995. “The Single Transferable Vote.” The Journal of Economic Perspectives 9:27–38. (http://dx.doi.org/10.1257/jep.9.1.27).Search in Google Scholar

Tideman, T. N. 1987. “Independence of Clones as a Criterion for Voting Rules.” Social Choice and Welfare 4:185–206.Search in Google Scholar

Woodall, D. R. 1997. “Monotonicity of Single-Seat Preferential Election Rules.” Discrete Applied Mathematics 77:81–98. (http://dx.doi.org/10.1016/S0166-218X(96)00100-X).Search in Google Scholar

Appendix

A IRV tabulation algorithm

One way to perform IRV tabulation is described in Algorithm 4. It takes as input the set of candidates A, the maximum number of rounds of the algorithm to perform ρ, and the set of ballots It iteratively eliminates the candidates with the fewest top choice votes – yi(1) is the top, continuing candidate on ballot i – by removing the candidates from every ballot on which they appear. As output, it produces the winner, if any, after p rounds, the set of candidates eliminated in each round, and the modified set of ballots after candidates have been eliminated.

As discussed in Section 5.3, there several rules for choosing which candidates to eliminate in each round. By abstracting the choice of the elimination set, all varieties of IRV can be described at once. The function takes the set of candidates A and the set of ballots B and returns the set of candidates to be eliminated next. For example, using the base IRV elimination rule, returns a singleton set consisting of the candidate with the fewest top-choice votes. The SF RCV elimination rule returns largest set of candidates E such that the sum of the top-choice votes for all candidates in E is less than the number of top-choice votes for all of the candidates not in E, S.F., Cal., Charter art. XIII, § 13.102(e) (2002).

Rather than iterating over each ballot every time, one can pick smarter representations such as keeping track of how many ballots with each particular candidate ranking exist or using tree data structure in which paths from the root to a node correspond to candidate rankings (O’Neill 2006). Using a tree, eliminating a candidate involves recursively removing nodes corresponding to that candidate and merging their children.

The function in Algorithm 5 takes a set of candidates A and a set of ballots B and returns all sets of candidates E that satisfy (18). This function is used in the construction of the IRV lower bound in Section 6.2 as well as auditing the elimination order in Section 5.3. Under the base IRV elimination rule, returns the smallest element of whereas under the SF RCV rule, it returns the largest.

B Examples for IRV margins

In this appendix we give some toy examples of IRV elections that illustrate two points. First, a small number of errors can dramatically change the outcome of an IRV election, even when the final round margin between the final two candidates is quite large. Secondly, the IRV margin can be smaller than the Condorcet margin lower bound, even when IRV elects the Condorcet winner.

B.1 IRV can be sensitive to small errors

IRV is sensitive to errors, in the following sense: switching even a single vote from one losing candidate to another (or fabricating a vote for a losing candidate) may be enough to change the winner of an election.16 We illustrate this via a simple example. Consider the six candidate, 1000 ballot election in Table 3. Zoë has the fewest votes of any candidate. She is eliminated in the first round and ultimately Velma wins with 496 votes. Ulric comes in second with 379 votes. Naively, one might say that Velma won with a margin greater than 10% (either 11.7% or about 13.4% depending on whether the denominator is 1000 or 379+496=876).

Table 3

Unmodified six candidate, 1000 ballot IRV election.

Table 3 Unmodified six candidate, 1000 ballot IRV election.

If an adversary is able to arrange for a single Y X V ballot to be counted as a Z Y ballot, then we get the election in Table 4. Here, the small error cascades through the rest of the rounds and Ulric, who previously came in second, is the winner with 379 votes. The correct winner, Velma, does not even make it to the final round. Instead, Xavier, who was previously eliminated in the second round makes it all the way to the final round to lose with 295 votes. Again, naively, the margin appears to be quite large (either 8.4% or about 12.5%). This example shows that intuition about margin calculations in plurality elections may not be applicable to IRV elections.

Table 4

IRV election in Table 3 with a single Y X V ballot changed to Z Y.

Table 4 IRV election in Table 3 with a single Y X V ballot changed to Z Y.

B.2 Margins for Condorcet versus IRV

The margin for IRV may be smaller than the Condorcet margin. Consider an election between Xavier, Yolanda, and Zoë. Only 36 ballots were cast in this election, and the results are summarized in Table 5.

Table 4

IRV election where the IRV margin is smaller than the Condorcet margin.

Table 4 IRV election where the IRV margin is smaller than the Condorcet margin.

Under IRV, in the first round Xavier gets 11 votes, Yolanda 15 votes, and Zoë 10 votes, so Zoë is eliminated. However, the supporters of Zoë break unanimously for Xavier over Yolanda, so in the final round Xavier defeats Yolanda 21 votes to 15 and Xavier is the IRV winner. The simple lower bound for the margin of this election is one vote, the gap between Xavier and Zoë in the first round. Note that Xavier is also the Condorcet winner of this election – voters prefer Xavier to both Yolanda and Zoë by 21 to 15. The Condorcet margin is therefore six votes. Further, voters also prefer Yolanda to Zoë 21 to 15 so the minimum difference in preference between candidates is also six. However, the IRV margin really is two since one ballot shifted from Xavier to Zoë will cause Xavier to be eliminated in the first round and Yolanda to win.

Published Online: 2013-01-11

©2013 by Walter de Gruyter Berlin Boston