Abstract
We demonstrate several ways to use morphological word analogies to examine the representation of complex words in semantic vector spaces. We present a set of morphological relations, each of which can be used to generate many word analogies. 1. We show that the difference-vectors for pairs which have the same relation to each other are similarly aligned. 2. We suggest that addition of difference-vectors is a useful phrase-building operator. 3. We propose that pairs in the same relation may have similar relative frequencies. 4. We suggest that homographs, which necessarily have the same semantic vectors, can sometimes be separated into different vectors for different senses, using frequency estimates and alignment constraints obtained from word analogies. 5. We observe that some of our analogies seem to be parallel, and might be combined. We use Arabic words as a case study, because Arabic orthography includes verb conjugations, object pronouns, definitive articles, possessive pronouns, and some prepositions in single word-forms. Therefore, a number of short phrases, built up of easily perceived constituents, are already present in stock semantic spaces for Arabic available on the web. Similar phrases in English would require including bigrams or trigrams as lemmas in the word embedding, although English derivational morphology allows for other relationships in standard semantic spaces which Arabic does not, for example negation. We make our corpus of morphological relations available to other researchers.
References
[1] Turney P. D., Pantel P., From frequency to meaning: Vector space models of semantics, Journal of Artificial Intelligence Research, 2010, 37, 141-18810.1613/jair.2934Search in Google Scholar
[2] Collobert R., Weston J., A uniffied architecture for natural language processing: Deep neural networks with multitask learning, In: Proceedings of the 25th International Conference on Ma chine Learning, 2008, 160-16710.1145/1390156.1390177Search in Google Scholar
[3] Lample G., Ballesteros M., Subramanian S., Kawakami K., Dyer C., Neural architectures for named entity recognition, In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2016), San Diego, California, Association for Computational Linguistics, June 2016, 260-27010.18653/v1/N16-1030Search in Google Scholar
[4] dos Santos C. N., Gatti M., Deep convolutional neural networks for sentiment analysis of short texts, In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, Dublin City University and Association for Computational Linguistics, August 2014, 69-78Search in Google Scholar
[5] Harris Z., Distributional structure, Word, 1954, 10(23), 146-16210.1080/00437956.1954.11659520Search in Google Scholar
[6] Deerwester S., Dumais S. T., Furnas G.W., Landauer T. K., Harshman R., Indexing by latent semantic analysis, Journal of the American Society for Information Science, 1990, 41(6), 391-40710.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9Search in Google Scholar
[7] Bengio Y., Ducharme R., Vincent P., Jauvin C., A neural probabilistic language model, Journal of Machine Learning Research, 2003, 3, 1137-1155Search in Google Scholar
[8] Mikolov T., Chen K., Corrado G., Dean J., Efficient estimation of word representations in vector space, CoRR, 2013, abs/1301.3781Search in Google Scholar
[9] Pennington J., Socher R., Manning C., Glove: Global vectors for word representation, In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, Association forComputational Linguistics, October 2014 , 1532-154310.3115/v1/D14-1162Search in Google Scholar
[10] Bojanowski P., Grave E., Joulin A., Mikolov T., Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics, 2017, 5, 135-14610.1162/tacl_a_00051Search in Google Scholar
[11] Soliman A. B., Eissa K., El-Beltagy S. R., AraVec: A set of Arabic word embedding models for use in Arabic NLP, Procedia Computer Science, 2017, 117(Supplement C), 256-26510.1016/j.procs.2017.10.117Search in Google Scholar
[12] Zahran M. A., Magooda A., Mahgoub A. Y., Raafat H., Rashwan M., Atyia A., Word representations in vector space and their applications for arabic, In: Gelbukh A. (Ed.), International Conference on Intelligent Text Processing and Computational Linguistics (CICLing 2015), Part I, LNCS 9041, Springer, 2015, 430-44310.1007/978-3-319-18111-0_32Search in Google Scholar
[13] Mikolov T., Yih W.-T., Zweig G., Linguistic regularities in continuous space word representations, In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), 2013, 746-751Search in Google Scholar
[14] Levy O., Goldberg Y., Linguistic regularities in sparse and explicit word representations, In: Proceedings of the Eighteenth Conference on Computational Language Learning, Association for Computational Linguistics, 2014, 171-18010.3115/v1/W14-1618Search in Google Scholar
[15] Le Q., Mikolov T., Distributed representations of sentences and documents, In: Proceedings of the 31st International Conference on Machine Learning, 2014Search in Google Scholar
[16] Kiros R., Zhu Y., Salakhutdinov R. R., Zemel R., Torralba A., Urtasun R., Fidler S., Skip-thought vectors, In: Advances in neural information processing systems, 2015, 3294-3302Search in Google Scholar
[17] Hill F., Cho K., Korhonen A., Learning distributed representations of sentences from unlabelled data, In: Proceedings of NAACL-HLT 2016, Association for Computational Linguistics, 2016, 1367-137710.18653/v1/N16-1162Search in Google Scholar
[18] Wieting J., Bansal M., Gimpel K., Livescu K., Charagram: Embedding words and sentences via character n-grams, In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2016, 1504-151510.18653/v1/D16-1157Search in Google Scholar
[19] Sutskever I., VinyalsO., Le Q. V., Sequence to sequence learning with neural networks, In: Proceedings of the 28th Conference on Neural Information Processing Systems (NIPS 2014), 2014Search in Google Scholar
[20] Conneau A., Kruszewski G., Barrault L., Lample G., Baroni M., What you can cram into a single vector: Probing sentence embeddings for linguistic properties, In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), 2018, 2126-213610.18653/v1/P18-1198Search in Google Scholar
[21] Mitchell J., Lapata M., Composition in distributional models of semantics, Cognitive Science, 2010, 34, 1388-142910.1111/j.1551-6709.2010.01106.xSearch in Google Scholar PubMed
[22] Coecke B., Sadrzadeh M., Clark S., Mathematical foundations for a compositional distributional model of meaning, Lambek Festschrit, special issue of Linguistic Analysis, 2010, 36, 345-384Search in Google Scholar
[23] Turney P. D., Littman M. L., BighamJ., Shnayder V.,Combining independent modules to solve multiple-choice synonymand analogy problems, In: Recent Advances in Natural Language Processing, 200310.1075/cilt.260.11turSearch in Google Scholar
[24] Turney P. D., A uniform approach to analogies, synonyms, antonyms, and associations, In: Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), 2008, 905-91210.3115/1599081.1599195Search in Google Scholar
[25] Jurgens D. A., Turney P. D., Mohammad S. M., Holyoak K. J., Semeval-2012 task 2: Measuring degrees of relational similarity, In: Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, SemEval ’12, Stroudsburg, PA, USA, Association for Computational Linguistics, 2012, 356-364Search in Google Scholar
[26] Linzen T., Issues in evaluating semantic spaces using word analogies, In: Proceedings of the 1st Workshop on Evaluating Vector Space Representations for NLP, 2016, 13-1810.18653/v1/W16-2503Search in Google Scholar
[27] Drozd A., Gladkova A., Matsuoka S., Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen, In: Proceedings of the 26th International Conference on Computational Linguistics (COLING 2016): Technical Papers, 2016, 3519-3530Search in Google Scholar
[28] Elrazzaz M., Elbassuoni S., Helwe C., Shaban K., Methodical evaluation of Arabic word embeddings, In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers), July 2017, 454-45810.18653/v1/P17-2072Search in Google Scholar
[29] Pasha A., Al-Badrashiny M., Diab M., Kholy A. E., Eskander R., Habash N., et al., MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic, In: Proceedings of the 9th International Conference on Language Resources and Evaluation, 2014, 1094-1099Search in Google Scholar
[30] Darwish K.,Mubarak H., Abdelali A., Arabic diacritization: Stats, rules, and hacks, In: Proceedings of The Third Arabic Natural Language Processing Workshop (WANLP), 2017, 9-17.10.18653/v1/W17-1302Search in Google Scholar
© by Stephen Taylor, Tomáš Brychcín, published by De Gruyter
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
