Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter October 20, 2018

Beyond a Bag of Words: Using PULSAR to Extract Judgments on Specific Human Rights at Scale

  • Baekkwan Park , Michael Colaresi EMAIL logo and Kevin Greene


Sentiment, judgments and expressed positions are crucial concepts across international relations and the social sciences more generally. Yet, contemporary quantitative research has conventionally avoided the most direct and nuanced source of this information: political and social texts. In contrast, qualitative research has long relied on the patterns in texts to understand detailed trends in public opinion, social issues, the terms of international alliances, and the positions of politicians. Yet, qualitative human reading does not scale to the accelerating mass of digital information available currently. Researchers are in need of automated tools that can extract meaningful opinions and judgments from texts. Thus, there is an emerging opportunity to marry the model-based, inferential focus of quantitative methodology, as exemplified by ideal point models, with high resolution, qualitative interpretations of language and positions. We suggest that using alternatives to simple bag of words (BOW) representations and re-focusing on aspect-sentiment representations of text will aid researchers in systematically extracting people’s judgments and what is being judged at scale. The experimental results below show that our approach which automates the extraction of aspect and sentiment MWE pairs, outperforms BOW in classification tasks, while providing more interpretable parameters. By connecting expressed sentiment and the aspects being judged, PULSAR (Parsing Unstructured Language into Sentiment-Aspect Representations) also has deep implications for understanding the underlying dimensionality of issue positions and ideal points estimated with text. Our approach to parsing text into aspects-sentiment expressions recovers both expressive phrases (akin to categorical votes), as well as the aspects that are being judged (akin to bills). Thus, PULSAR or future systems like it, open up new avenues for the systematic analysis of high-dimensional opinions and judgments at scale within existing ideal point models.


Bespalov, D., Bai, B., Qi, Y., & Shokoufandeh, A. (2011). Sentiment classification based on supervised latent n-gram analysis. Proceedings of the 20th ACM international conference on Information and knowledge management – CIKM ’11. URL: in Google Scholar

Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.Search in Google Scholar

Brandt, P. T., Freeman, J. R., & Schrodt, P. A. (2014). Evaluating forecasts of political conflict dynamics. International Journal of Forecasting, 30(4), 944–962.10.1016/j.ijforecast.2014.03.014Search in Google Scholar

Calzolari, N., Fillmore, C. J., Grishman, R., Ide, N., Lenci, A., MacLeod C., & Zampolli, A. (2002). Towards best practice for multiword expressions in computational lexicons. In LREC.Search in Google Scholar

Cingranelli, D. L., Richards, D. L., & Clay, K. C. (2014). The CIRI human rights dataset. v.2014.04.14.Search in Google Scholar

Colaresi, M., & Mahmood, Z. (2017). Do the robot: Lessons from machine learning to improve conflict forecasting. Journal of Peace Research, 54(2), 193–214.10.1177/0022343316682065Search in Google Scholar

Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82–89. URL: in Google Scholar

Gibney, M., Cornett, L., Wood, R., Haschke, P., & Arnon, D. (2015). The political terror scale 1976–2015. Date Retrieved, from the Political Terror Scale website: in Google Scholar

Handler, A., Denny, M., Wallach, H., & O’Connor, B. (2016). Bag of what? Simple noun phrase extraction for text analysis. In Proceedings of the First Workshop on NLP and Computational Social Science. pp. 114–124.10.18653/v1/W16-5615Search in Google Scholar

Ho, D. E., & Quinn, K. M. (2008). Measuring explicit political positions of media. Quarterly Journal of Political Science, 3(4), 353–377.10.1561/100.00008048Search in Google Scholar

Justeson, J. S., & Katz, S. M. (1995). Technical terminology: some linguistic properties and an algorithm for identification in text. Natural Language Engineering, 1(1), 9–27.10.1017/S1351324900000048Search in Google Scholar

Laver, M., Benoit, K., & Garry, J. (2003). Extracting policy positions from political texts using words as data. American Political Science Review, 97(1), 311–331.10.1017/S0003055403000698Search in Google Scholar

Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sentiments and Emotions. New York: Cambridge University Press.10.1017/CBO9781139084789Search in Google Scholar

Liu, Q., Gao, Z., Liu, B., & Zhang, Y. (2013). A logic programming approach to aspect extraction in opinion mining. In Web Intelligence (WI) and Intelligent Agent Technologies (IAT), 2013 IEEE/WIC/ACM International Joint Conferences. Vol. 1 IEEE pp. 276–283.Search in Google Scholar

Liu, Q., Gao, Z., Liu, B., & Zhang, Y. (2015). Automated rule selection for aspect extraction in opinion mining. In International Joint Conference on Artificial Intelligence (IJCAI).Search in Google Scholar

Lowe, W. (2013). There’s (basically) only one way to do it. Available at SSRN.10.2139/ssrn.2318543Search in Google Scholar

Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pp. 55–60. URL: in Google Scholar

Monroe, B. L., & Maeda, K. (2004). Talk’s cheap: Text-based ideal point estimation. In presented to the Political Methodology Society. Palo Alto, CA.Search in Google Scholar

Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). ‘Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4), 372–403.10.1093/pan/mpn018Search in Google Scholar

Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Informational Retrieval, 2(1-2), 1–135.10.1561/9781601981516Search in Google Scholar

Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Proceedings of the ACL-02 conference on Empirical methods in natural language processing – EMNLP ’02. URL: in Google Scholar

Qiu, G., Liu, B., Bu, J., Chen, C. (2011). Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1), 9–27.10.1162/coli_a_00034Search in Google Scholar

Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M., & Radev, D. R. (2010). How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1), 209–228.10.1111/j.1540-5907.2009.00427.xSearch in Google Scholar

Sag, I. A., Baldwin, T., Bond, F., Copestake, A., & Flickinger, D. (2002). Multiword expressions: a pain in the neck for NLP. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer pp. 1–15.10.1007/3-540-45715-1_1Search in Google Scholar

Schrodt, P. A., Beieler, J., & Idris, M. (2014). Three’sa charm?: Open event data coding with el: Diablo, Petrarch, and the open event data alliance. In ISA Annual Convention.Search in Google Scholar

Slapin, J. B., & Proksch, S.-O. (2008). A scaling model for estimating time-series party positions from texts. American Journal of Political Science, 52(3), 705–722.10.1111/j.1540-5907.2008.00338.xSearch in Google Scholar

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1631–1642.Search in Google Scholar

Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning. Association for Computational Linguistics pp. 1201–1211.Search in Google Scholar

Socher, R., Lin, C. C., Manning, C., & Ng, A. Y. (2011). Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11). pp. 129–136.Search in Google Scholar

Taylor, A., Marcus, M., & Santorini, B. (2003). The Penn treebank: an overview. In A. Abeillé (Ed.), Treebanks (pp. 5–22). Dordrecht: Springer.10.1007/978-94-010-0201-1_1Search in Google Scholar

Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics pp. 173–180.10.3115/1073445.1073478Search in Google Scholar

Wallach, H. M. (2006). Topic modeling. Proceedings of the 23rd international conference on Machine learning – ICML ’06. URL: in Google Scholar

Wu, Y., Zhang, Q., Huang, X., & Wu, L. (2009). Phrase dependency parsing for opinion mining. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Vol. 3 Association for Computational Linguistics pp. 1533–1541.10.3115/1699648.1699700Search in Google Scholar

Yessenalina, A., & Cardie, C. (2011). Compositional matrix-space models for sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. EMNLP ’11 Stroudsburg, PA, USA: Association for Computational Linguistics pp. 172–182. URL: in Google Scholar

Published Online: 2018-10-20

©2018 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 11.12.2023 from
Scroll to top button