Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Proceedings on Privacy Enhancing Technologies

4 Issues per year

Open Access
See all formats and pricing
More options …

Automated Experiments on Ad Privacy Settings

A Tale of Opacity, Choice, and Discrimination

Amit Datta / Michael Carl Tschantz / Anupam Datta
Published Online: 2015-04-18 | DOI: https://doi.org/10.1515/popets-2015-0007


To partly address people’s concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google’s ads, and Ad Settings interact. AdFisher can run browser-based experiments and analyze data using machine learning and significance tests. Our tool uses a rigorous experimental design and statistical analysis to ensure the statistical soundness of our results. We use AdFisher to find that the Ad Settings was opaque about some features of a user’s profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse changed the ads shown but not the settings page. We also found that setting the gender to female resulted in getting fewer instances of an ad related to high paying jobs than setting it to male. We cannot determine who caused these findings due to our limited visibility into the ad ecosystem, which includes Google, advertisers, websites, and users. Nevertheless, these results can form the starting point for deeper investigations by either the companies themselves or by regulatory bodies.

Keywords : blackbox analysis; information flow; behavioral advertising; transparency; choice; discrimination


  • [1] J. R. Mayer and J. C. Mitchell, “Third-party web tracking: Policy and technology,” in IEEE Symposium on Security and Privacy, 2012, pp. 413-427.Google Scholar

  • [2] B. Ur, P. G. Leon, L. F. Cranor, R. Shay, and Y. Wang, “Smart, useful, scary, creepy: Perceptions of online behavioral advertising,” in Proceedings of the Eighth Symposium on Usable Privacy and Security. ACM, 2012, pp. 4:1-4:15.Google Scholar

  • [3] Google, “About ads settings,” https://support.google.com/ ads/answer/2662856, accessed Nov. 21, 2014.Google Scholar

  • [4] Yahoo!, “Ad interest manager,” https://info.yahoo.com/ privacy/us/yahoo/opt_out/targeting/details.html, accessed Nov. 21, 2014.Google Scholar

  • [5] Microsoft, “Microsoft personalized ad preferences,” http: //choice.microsoft.com/en-us/opt-out, accessed Nov. 21, 2014.Google Scholar

  • [6] Executive Office of the President, “Big data: Seizing opportunities, preserving values,” Posted at http://www. whitehouse.gov/sites/default/files/docs/big_data_privacy_ report_may_1_2014.pdf, 2014, accessed Jan. 26, 2014.Google Scholar

  • [7] R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning fair representations,” in Proceedings of the 30th International Conference on Machine Learning (ICML-13), S. Dasgupta and D. Mcallester, Eds., vol. 28. JMLR Workshop and Conference Proceedings, May 2013, pp. 325-333. [Online]. Available: http: //jmlr.org/proceedings/papers/v28/zemel13.pdfGoogle Scholar

  • [8] Google, “Privacy policy,” https://www.google.com/intl/en/ policies/privacy/, accessed Nov. 21, 2014.Google Scholar

  • [9] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.Google Scholar

  • [10] E. Jones, T. Oliphant, P. Peterson et al., “SciPy: Open source scientific tools for Python,” 2001, http://www.scipy.org/.Google Scholar

  • [11] M. C. Tschantz, A. Datta, A. Datta, and J. M. Wing, “A methodology for information flow experiments,” ArXiv, Tech. Rep. arXiv:1405.2376v1, 2014.Google Scholar

  • [12] P. Good, Permutation, Parametric and Bootstrap Tests of Hypotheses. Springer, 2005.Google Scholar

  • [13] C. E. Wills and C. Tatar, “Understanding what they do with what they know,” in Proceedings of the 2012 ACM Workshop on Privacy in the Electronic Society, 2012, pp. 13-18.Google Scholar

  • [14] P. Barford, I. Canadi, D. Krushevskaja, Q. Ma, and S. Muthukrishnan, “Adscape: Harvesting and analyzing online display ads,” in Proceedings of the 23rd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2014, pp. 597-608.Google Scholar

  • [15] B. Liu, A. Sheth, U. Weinsberg, J. Chandrashekar, and R. Govindan, “AdReveal: Improving transparency into online targeted advertising,” in Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks. ACM, 2013, pp. 12:1-12:7.Google Scholar

  • [16] M. Lécuyer, G. Ducoffe, F. Lan, A. Papancea, T. Petsios, R. Spahn, A. Chaintreau, and R. Geambasu, “XRay: Increasing the web’s transparency with differential correlation,” in Proceedings of the USENIX Security Symposium, 2014.Google Scholar

  • [17] S. Englehardt, C. Eubank, P. Zimmerman, D. Reisman, and A. Narayanan, “Web privacy measurement: Scientific principles, engineering platform, and new results,” Manuscript posted at http://randomwalker.info/ publications/WebPrivacyMeasurement.pdf, 2014, accessed Nov. 22, 2014.Google Scholar

  • [18] S. Guha, B. Cheng, and P. Francis, “Challenges in measuring online advertising systems,” in Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, 2010, pp. 81-87.Google Scholar

  • [19] R. Balebako, P. Leon, R. Shay, B. Ur, Y. Wang, and L. Cranor, “Measuring the effectiveness of privacy tools for limiting behavioral advertising,” in Web 2.0 Security and Privacy Workshop, 2012.Google Scholar

  • [20] L. Sweeney, “Discrimination in online ad delivery,” Commun. ACM, vol. 56, no. 5, pp. 44-54, 2013.Google Scholar

  • [21] R. A. Fisher, The Design of Experiments. Oliver & Boyd, 1935.Google Scholar

  • [22] S. Greenland and J. M. Robins, “Identifiability, exchangeability, and epidemiological confounding,” International Journal of Epidemiology, vol. 15, no. 3, pp. 413-419, 1986.Google Scholar

  • [23] T. M. Mitchell, Machine Learning. McGraw-Hill, 1997.Google Scholar

  • [24] D. D. Jensen, “Induction with randomization testing: Decision-oriented analysis of large data sets,” Ph.D. dissertation, Sever Institute of Washington University, 1992.Google Scholar

  • [25] Alexa, “Is popularity in the top sites by category directory based on traffic rank?” https://support.alexa.com/hc/enus/ articles/200461970, accessed Nov. 21, 2014.Google Scholar

  • [26] C. M. Bishop, Pattern Recognition and Machine Learning. Springer, 2006.Google Scholar

  • [27] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65-70, 1979.Google Scholar

  • [28] Google, “Google privacy and terms,” http://www.google. com/policies/technologies/ads/, accessed Nov. 22, 2014.Google Scholar

  • [29] H. Abdi, “Bonferroni and Šidák corrections for multiple comparisons,” in Encyclopedia of Measurement and Statistics, N. J. Salkind, Ed. Sage, 2007.Google Scholar

  • [30] D. Hume, A Treatise of Human Nature: Being an Attempt to Introduce the Experimental Method of Reasoning into Moral Subjects, 1738, book III, part I, section I.Google Scholar

  • [31] Pew Research Center’s Social and Demographic Trends Project, “On pay gap, millennial women near parity - for now: Despite gains, many see roadblocks ahead,” 2013.Google Scholar

  • [32] T. Z. Zarsky, “Understanding discrimination in the scored society,” Washington Law Review, vol. 89, pp. 1375-1412, 2014.Google Scholar

  • [33] R. S. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning fair representations,” in Proceedings of the 30th International Conference on Machine Learning, ser. JMLR: W&CP, vol. 28. JMLR.org, 2013, pp. 325-333.Google Scholar

  • [34] Adgooroo, “Adwords cost per click rises 26% between 2012 and 2014,” http://www.adgooroo.com/resources/blog/ adwords-cost-per-click-rises-26-between-2012-and-2014/, accessed Nov. 21, 2014.Google Scholar

  • [35] L. Olejnik, T. Minh-Dung, and C. Castelluccia, “Selling off privacy at auction,” in Network and Distributed System Security Symposium (NDSS). The Internet Society, 2013.Google Scholar

  • [36] C. J. Clopper and E. S. Pearson, “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika, vol. 26, no. 4, pp. 404-413, 1934. Google Scholar

About the article

Received: 2014-11-22

Revised: 2015-02-18

Accepted: 2015-02-18

Published Online: 2015-04-18

Published in Print: 2015-04-01

Citation Information: Proceedings on Privacy Enhancing Technologies, Volume 2015, Issue 1, Pages 92–112, ISSN (Online) 2299-0984, DOI: https://doi.org/10.1515/popets-2015-0007.

Export Citation

© 2015. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Lambèr Royakkers, Jelte Timmer, Linda Kool, and Rinie van Est
Ethics and Information Technology, 2018
Evaggelia Pitoura, Panayiotis Tsaparas, Giorgos Flouris, Irini Fundulaki, Panagiotis Papadakos, Serge Abiteboul, and Gerhard Weikum
ACM SIGMOD Record, 2018, Volume 46, Number 4, Page 16
Aws Albarghouthi, Loris D'Antoni, Samuel Drews, and Aditya V. Nori
Proceedings of the ACM on Programming Languages, 2017, Volume 1, Number OOPSLA, Page 1
Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck
Philosophy & Technology, 2017
Serge Abiteboul, Leonid Libkin, Wim Martens, Tova Milo, Filip Murlak, Frank Neven, Magdalena Ortiz, Thomas Schwentick, Julia Stoyanovich, Jianwen Su, Dan Suciu, Marcelo Arenas, Victor Vianu, Ke Yi, Pablo Barceló, Meghyn Bienvenu, Diego Calvanese, Claire David, Richard Hull, Eyke Hüllermeier, and Benny Kimelfeld
ACM SIGMOD Record, 2017, Volume 45, Number 4, Page 5
Javier Parra-Arnau, Jagdish Prasad Achara, and Claude Castelluccia
ACM Transactions on the Web, 2017, Volume 11, Number 1, Page 1
Sourya Joyee De and Daniel Le Métayer
Synthesis Lectures on Information Security, Privacy, and Trust, 2016, Volume 81, Number 3, Page 1
Kevin Brock and Dawn Shepherd
Computers and Composition, 2016, Volume 42, Page 17
Eileen Fischer
Journal of Marketing Management, 2015, Volume 31, Number 15-16, Page 1718

Comments (0)

Please log in or register to comment.
Log in