Abstract
Addressing Answer Selection (AS) tasks with complex neural networks typically requires a large amount of annotated data to increase the accuracy of the models. In this work, we are interested in simple models that can potentially give good performance on datasets with no or few annotations. First, we propose new unsupervised baselines that leverage distributed word and sentence representations. Second, we compare the ability of our neural architectures to learn from few annotated examples in a weakly supervised scheme and we demonstrate how these methods can benefit from a pre-training on an external dataset. With an emphasis on results reproducibility, we show that our simple methods can reach or approach state-of-the-art performances on four common AS datasets.
References
[1] Rajpurkar P., Zhang J., Lopyrev K., Liang P., SQuAD: 100,000+ Questions for Machine Comprehension of Text, 2016, arXiv: 1606.05250 [cs]10.18653/v1/D16-1264Search in Google Scholar
[2] Yin W., Schütze H., Xiang B., Zhou B., ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence, Pairs, 2015, arXiv: 1512.05193 [cs]10.1162/tacl_a_00097Search in Google Scholar
[3] Wang S., Jiang J., A Compare-Aggregate Model for Matching Text Sequences, 2016, arXiv: 1611.01747 [cs]Search in Google Scholar
[4] Tran N. K., Niedereée C., Multihop Attention Networks for Question Answer Matching, in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, ACM, New York, NY, USA, 2018, 325–334, 10.1145/3209978.321000910.1145/3209978.3210009Search in Google Scholar
[5] Shen D., Wang G., Wang W., Min M.R., Su Q., Zhang Y., Li C., Henao R., Carin L., Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms, arXiv:1805.09843 [cs], 2018, arXiv: 1805.09843 [cs]10.18653/v1/P18-1041Search in Google Scholar
[6] Bogdanova D., Foster J., This is how we do it: Answer Reranking for Open-domain How Questions with Paragraph Vectors and Minimal Feature Engineering, in HLT-NAACL, 201610.18653/v1/N16-1154Search in Google Scholar
[7] Yadav V., Sharp R., Surdeanu M., Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering, 2018, arXiv: 1807.01836 [cs]10.1145/3209978.3210142Search in Google Scholar
[8] Seo M., Kembhavi A., Farhadi A., Hajishirzi H., Bidirectional Attention Flow for Machine Comprehension, 2016, arXiv: 1611.01603 [cs]Search in Google Scholar
[9] Min S., Seo M., Hajishirzi H., Question Answering through Transfer Learning from Large Fine-grained Supervision Data, 2017, arXiv: 1702.02171 [cs]10.18653/v1/P17-2081Search in Google Scholar
[10] Zhang P., Hou Y., Su Z., Su Y., Two-Step Multi-factor Attention Neural Network for Answer Selection, in PRICAI 2018: Trends in Artificial Intelligence, Springer, Cham, 2018, 658–670, 10.1007/978-3-319-97304-3_5010.1007/978-3-319-97304-3_50Search in Google Scholar
[11] Kuhn H. W., The Hungarian method for the assignment problem, Naval Research Logistics Quarterly, 1955, 2(1-2), 83-97, 10.1002/nav.380002010910.1002/nav.3800020109Search in Google Scholar
[12] Mikolov T., Sutskever I., Chen K., Corrado G.S., Dean J., Distributed Representations of Words and Phrases and their Compositionality, in C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K.Q. Weinberger, eds., Advances in Neural Information Processing Systems 26, Curran Associates, Inc., 2013, 3111–3119Search in Google Scholar
[13] Cer D., Yang., Kong S. Y., Hua N., Limtiaco N., John R.S., Constant N., Guajardo-Cespedes M., Yuan S., Tar C., Sung Y.H., Strope B., Kurzweil R., Universal Sentence Encoder, 2018, arXiv: 1803.11175 [cs]10.18653/v1/D18-2029Search in Google Scholar
[14] Crane M., Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results, Transactions of the Association for Computational Linguistics, 2018, 6(0), 241–25210.1162/tacl_a_00018Search in Google Scholar
[15] Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser L., Polosukhin I., Attention Is All You Need, 2017, arXiv: 1706.03762 [cs]Search in Google Scholar
[16] Robertson S. E., Jones K. S., Relevance weighting of search terms, Journal of the American Society for Information Science, 1976, 27(3), 129–146, 10.1002/asi.463027030210.1002/asi.4630270302Search in Google Scholar
[17] Lai T. M., Bui T., Li S., A Review on Deep Learning Techniques Applied to Answer Selection, in Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, 2132–2144Search in Google Scholar
[18] Wang M., Smith N. A., Mitamura T., What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA, in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007Search in Google Scholar
[19] Wang D., Nyberg E., A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, 2015, 707–712, 10.3115/v1/P15-2116, event-place: Beijing, China10.3115/v1/P15-2116Search in Google Scholar
[20] Nakov P., Màrquez L., Moschitti A., Magdy W., Mubarak H., Freihat a.A., Glass J., Randeree B., SemEval-2016 Task 3: Community Question Answering, in Proceedings of the 10th International Workshop on Semantic Evaluation, Association for Computational Linguistics, San Diego, California, 2016, 525–545, 10.18653/v1/S16-108310.18653/v1/S16-1083Search in Google Scholar
[21] Jansen P., Surdeanu M., Clark P., Discourse Complements Lexical Semantics for Non-factoid Answer Reranking, in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Baltimore, Maryland, 2014, 977–986, 10.3115/v1/P14-109210.3115/v1/P14-1092Search in Google Scholar
[22] Kingma D.P., Ba J., Adam: A Method for Stochastic Optimization, 2014, arXiv: 1412.6980 [cs]Search in Google Scholar
[23] Liu Y., Rong W., Xiong Z., Improved Text Matching by Enhancing Mutual Information, in AAAI, 201810.1609/aaai.v32i1.11948Search in Google Scholar
[24] Tayyar Madabushi H., Lee M., Barnden J., Integrating Question Classification and Deep Learning for improved Answer Selection, in Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018, 3283–3294Search in Google Scholar
[25] Zhang H., Rao J., Lin J., Smucker M.D., Automatically Extracting High-Quality Negative Examples for Answer Selection in Question Answering, in Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, ACM, New York, NY, USA, 2017, 797–80010.1145/3077136.3080645Search in Google Scholar
[26] Bogdanova D., Foster J., Dzendzik D., Liu Q., If You Can’t Beat Them Join Them: Handcrafted Features Complement Neural Nets for Non-Factoid Answer Reranking, in HLT-EACL, Association for Computational Linguistics, Valencia, Spain, 2017, 121–13110.18653/v1/E17-1012Search in Google Scholar
[27] Cohen D., Mitra B., Hofmann K., Croft W. B., Cross Domain Regularization for Neural Ranking Models Using Adversarial Learning, 2018, arXiv: 1805.03403 [cs]10.1145/3209978.3210141Search in Google Scholar
[28] Seo M., Kwiatkowski T., Parikh A. P., Farhadi A., Hajishirzi H., Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension, 2018, arXiv: 1804.07726 [cs]10.18653/v1/D18-1052Search in Google Scholar
[29] Rücklé A., Gurevych I., Representation Learning for Answer Selection with LSTM-Based Importance Weighting, IWCS 2017 — 12th International Conference on Computational Semantics — Short papers, 2017Search in Google Scholar
© 2019 Emmanuel Vallee et al., published by De Gruyter Open
This work is licensed under the Creative Commons Attribution 4.0 Public License.
