Beyond a Bag of Words: Using PULSAR to Extract Judgments on Specific Human Rights at Scale

Baekkwan Park 1 , Michael Colaresi 1 , and Kevin Greene 1
  • 1 University of Pittsburgh, Department of Political Science, Pittsburgh, United States of America
Baekkwan Park
  • University of Pittsburgh, Department of Political Science, Pittsburgh, United States of America
  • Search for other articles:
  • degruyter.comGoogle Scholar
, Michael Colaresi
  • Corresponding author
  • University of Pittsburgh, Department of Political Science, Pittsburgh, United States of America
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar
and Kevin Greene
  • University of Pittsburgh, Department of Political Science, Pittsburgh, United States of America
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

Sentiment, judgments and expressed positions are crucial concepts across international relations and the social sciences more generally. Yet, contemporary quantitative research has conventionally avoided the most direct and nuanced source of this information: political and social texts. In contrast, qualitative research has long relied on the patterns in texts to understand detailed trends in public opinion, social issues, the terms of international alliances, and the positions of politicians. Yet, qualitative human reading does not scale to the accelerating mass of digital information available currently. Researchers are in need of automated tools that can extract meaningful opinions and judgments from texts. Thus, there is an emerging opportunity to marry the model-based, inferential focus of quantitative methodology, as exemplified by ideal point models, with high resolution, qualitative interpretations of language and positions. We suggest that using alternatives to simple bag of words (BOW) representations and re-focusing on aspect-sentiment representations of text will aid researchers in systematically extracting people’s judgments and what is being judged at scale. The experimental results below show that our approach which automates the extraction of aspect and sentiment MWE pairs, outperforms BOW in classification tasks, while providing more interpretable parameters. By connecting expressed sentiment and the aspects being judged, PULSAR (Parsing Unstructured Language into Sentiment-Aspect Representations) also has deep implications for understanding the underlying dimensionality of issue positions and ideal points estimated with text. Our approach to parsing text into aspects-sentiment expressions recovers both expressive phrases (akin to categorical votes), as well as the aspects that are being judged (akin to bills). Thus, PULSAR or future systems like it, open up new avenues for the systematic analysis of high-dimensional opinions and judgments at scale within existing ideal point models.

  • Bespalov, D., Bai, B., Qi, Y., & Shokoufandeh, A. (2011). Sentiment classification based on supervised latent n-gram analysis. Proceedings of the 20th ACM international conference on Information and knowledge management – CIKM ’11. URL: http://dx.doi.org/10.1145/2063576.2063635

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

  • Brandt, P. T., Freeman, J. R., & Schrodt, P. A. (2014). Evaluating forecasts of political conflict dynamics. International Journal of Forecasting, 30(4), 944–962.

    • Crossref
    • Export Citation
  • Calzolari, N., Fillmore, C. J., Grishman, R., Ide, N., Lenci, A., MacLeod C., & Zampolli, A. (2002). Towards best practice for multiword expressions in computational lexicons. In LREC.

  • Cingranelli, D. L., Richards, D. L., & Clay, K. C. (2014). The CIRI human rights dataset. v.2014.04.14.

  • Colaresi, M., & Mahmood, Z. (2017). Do the robot: Lessons from machine learning to improve conflict forecasting. Journal of Peace Research, 54(2), 193–214.

    • Crossref
    • Export Citation
  • Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82–89. URL: http://dx.doi.org/10.1145/2436256.2436274

    • Crossref
    • Export Citation
  • Gibney, M., Cornett, L., Wood, R., Haschke, P., & Arnon, D. (2015). The political terror scale 1976–2015. Date Retrieved, from the Political Terror Scale website: http://www.politicalterrorscale.org.

  • Handler, A., Denny, M., Wallach, H., & O’Connor, B. (2016). Bag of what? Simple noun phrase extraction for text analysis. In Proceedings of the First Workshop on NLP and Computational Social Science. pp. 114–124.

    • Crossref
    • Export Citation
  • Ho, D. E., & Quinn, K. M. (2008). Measuring explicit political positions of media. Quarterly Journal of Political Science, 3(4), 353–377.

    • Crossref
    • Export Citation
  • Justeson, J. S., & Katz, S. M. (1995). Technical terminology: some linguistic properties and an algorithm for identification in text. Natural Language Engineering, 1(1), 9–27.

    • Crossref
    • Export Citation
  • Laver, M., Benoit, K., & Garry, J. (2003). Extracting policy positions from political texts using words as data. American Political Science Review, 97(1), 311–331.

  • Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sentiments and Emotions. New York: Cambridge University Press.

  • Liu, Q., Gao, Z., Liu, B., & Zhang, Y. (2013). A logic programming approach to aspect extraction in opinion mining. In Web Intelligence (WI) and Intelligent Agent Technologies (IAT), 2013 IEEE/WIC/ACM International Joint Conferences. Vol. 1 IEEE pp. 276–283.

  • Liu, Q., Gao, Z., Liu, B., & Zhang, Y. (2015). Automated rule selection for aspect extraction in opinion mining. In International Joint Conference on Artificial Intelligence (IJCAI).

  • Lowe, W. (2013). There’s (basically) only one way to do it. Available at SSRN.

  • Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pp. 55–60. URL: http://www.aclweb.org/anthology/P/P14/P14-5010

  • Monroe, B. L., & Maeda, K. (2004). Talk’s cheap: Text-based ideal point estimation. In presented to the Political Methodology Society. Palo Alto, CA.

  • Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). ‘Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4), 372–403.

    • Crossref
    • Export Citation
  • Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Informational Retrieval, 2(1-2), 1–135.

    • Crossref
    • Export Citation
  • Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Proceedings of the ACL-02 conference on Empirical methods in natural language processing – EMNLP ’02. URL: http://dx.doi.org/10.3115/1118693.1118704

  • Qiu, G., Liu, B., Bu, J., Chen, C. (2011). Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1), 9–27.

    • Crossref
    • Export Citation
  • Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M., & Radev, D. R. (2010). How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1), 209–228.

    • Crossref
    • Export Citation
  • Sag, I. A., Baldwin, T., Bond, F., Copestake, A., & Flickinger, D. (2002). Multiword expressions: a pain in the neck for NLP. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer pp. 1–15.

  • Schrodt, P. A., Beieler, J., & Idris, M. (2014). Three’sa charm?: Open event data coding with el: Diablo, Petrarch, and the open event data alliance. In ISA Annual Convention.

  • Slapin, J. B., & Proksch, S.-O. (2008). A scaling model for estimating time-series party positions from texts. American Journal of Political Science, 52(3), 705–722.

    • Crossref
    • Export Citation
  • Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1631–1642.

  • Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning. Association for Computational Linguistics pp. 1201–1211.

  • Socher, R., Lin, C. C., Manning, C., & Ng, A. Y. (2011). Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11). pp. 129–136.

  • Taylor, A., Marcus, M., & Santorini, B. (2003). The Penn treebank: an overview. In A. Abeillé (Ed.), Treebanks (pp. 5–22). Dordrecht: Springer.

  • Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics pp. 173–180.

  • Wallach, H. M. (2006). Topic modeling. Proceedings of the 23rd international conference on Machine learning – ICML ’06. URL: http://dx.doi.org/10.1145/1143844.1143967

  • Wu, Y., Zhang, Q., Huang, X., & Wu, L. (2009). Phrase dependency parsing for opinion mining. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Vol. 3 Association for Computational Linguistics pp. 1533–1541.

  • Yessenalina, A., & Cardie, C. (2011). Compositional matrix-space models for sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. EMNLP ’11 Stroudsburg, PA, USA: Association for Computational Linguistics pp. 172–182. URL: http://dl.acm.org/citation.cfm?id=2145432.2145452

Purchase article
Get instant unlimited access to the article.
$42.00
Log in
Already have access? Please log in.


or
Log in with your institution

Journal + Issues

Search