Word2Vec: Optimal hyperparameters and their impact on natural language processing downstream tasks

Acronyms

BERT

Bidirectional Encoder Representations from Transformers

Bi-LSTM

bi-directional long short-term memory network

BW

billion word

BoW

bag-of-words

CBoW

continuous bag-of-words

CNN

convolutional neural network

GDC

Gothenburg dialog corpus

GMB

Groningen meaning bank

IMDB

internet movie dataset

LSTM

long short-term memory network

NER

named entity recognition

NLG

natural language generation

NLP

natural language processing

NN

neural network

SA

sentiment analysis

SoTA

state-of-the-art

SRT

department of computer science, electrical and space engineering

SW

simple wiki

1 Introduction

There are two architectures in the implementation of the Word2Vec model: continuous skipgram and continuous bag-of-words (CBoW) [1]. Similar distributed models of word or subword embeddings find usage in recent state-of-the-art (SoTA), deep neural networks like Bidirectional Encoder Representations from Transformers (BERT) and its successors [2,3,4]. BERT generates contextual representations of words after it has been trained for extended periods on large corpora, unsupervised, using the attention mechanisms [5]. Unsupervised learning provides feature representations using large unlabelled corpora [6].

It has been observed that various hyperparameter combinations have been used in different works involving Word2Vec, with the possibility of some of them being suboptimal [7,8,9]. Therefore, the authors seek to address the research question: what is the optimal combination of Word2Vec hyperparameters for intrinsic and extrinsic natural language processing (NLP) purposes, specifically for named entity recognition (NER) and sentiment analysis (SA)? The number of combinations of hyperparameters that is possible for a neural network (NN) is very high, even for an NN with only a few layers [10]. Hence, the scope of our extensive, empirical work over three English corpora is on the dimension sizes, training epochs, window sizes, architectures, and the training algorithms (hierarchical softmax and negative sampling) of both skipgram and CBoW, as given in Section 3.

The objective of this work is to determine the relative optimal combinations of Word2Vec hyperparameters for intrinsic evaluation (semantic and syntactic analogies) and a couple of extrinsic evaluation tasks [11,12]. It is not our objective in this work to set new SoTA results. Some of the main contributions of this work are the discovery of the behaviour of quality of word-vectors vis-a-vis increasing dimensions for a given data size, the empirical establishment of more optimal combinations of Word2Vec hyperparameters for the identified NLP tasks, and the confirmation of the performance of embeddings being task-specific. The rest of this article is organized as follows: related work, materials and methods used, experimental that describes experiments performed, results and discussion that present final results, and conclusion.

2 Related work

2.1 Background

Breaking away from the non-distributed (high-dimensional, sparse) representations of words, typical of traditional bag-of-words (BoW) or one-hot-encoding [1,13] created Word2Vec. Word2Vec consists of two shallow NN architectures: continuous skipgram and CBoW, as shown in Figure 1. It uses distributed (low-dimensional, dense) representations of words that group similar words. This model traded the complexity of deep NN architectures for more efficient training over large corpora. It has two training algorithms: negative sampling and hierarchical softmax [14]. The publicly released model was trained on Google news dataset of 100 billion words (BWs). Implementations of the model have been undertaken in the programming languages: Python and C++, though the original was done in C [15]. The Python implementation is slower to train, being an interpreted language [16,17].

Figure 1

The CBoW and continuous skipgram model architectures [1].

Continuous skipgram predicts (by maximizing classification of) words before and after the centre word, for a given range. Since distant words are less connected to a centre word in a sentence, less weight is assigned to such distant words in training. CBoW, on the other hand, uses words from the history and future in a sequence, with the objective of correctly classifying the target word in the middle. It works by projecting all history or future words within a chosen window into the same position, averaging their vectors. Hence, the order of words in the history or future does not influence the averaged vector. This is similar to the traditional BoW. A log-linear classifier is used in both architectures [1]. In further work, Mikolov et al. [14] extended the model to be able to do phrase representations and subsample frequent words. Earlier models like latent Dirichlet allocation and latent semantic analysis effectively achieve low-dimensional vectors by matrix factorization [10,18].

3 Materials and methods

3.3 Downstream architectures

The downstream experiments were run on a Tesla GPU on a shared DGX cluster running Ubuntu 18. Pytorch deep learning framework was used. A long short-term memory network (LSTM) was trained on the GMB dataset for NER. A Bi-LSTM network was trained on the IMDB dataset for SA. The Bi-LSTM includes an additional hidden linear layer before the output layer (Figures 2 and 3). Hyperparameter details of the two networks for the downstream tasks are given in Table 2. The metrics for extrinsic evaluation include F1, precision, recall and accuracy (for SA).

Table 2

Downstream network hyperparameters

Archi	Epochs	Hidden Dim	LR	Loss
LSTM	40	128	0.01	Cross entropy
BiLSTM	20	128 ∗ 2	0.0001	BCELoss

Figure 2

Network architecture for NER.

Figure 3

Network architecture for SA.

4 Experimental

In our work, we extended research to 3,000 dimensions and epochs of 5 and 10. To form the vocabulary for the embeddings, words occurring less than 5 times in the corpora were dropped, stop words removed using the natural language toolkit [34] and additional data pre-processing carried out. Table 1 describes the hyperparameters explored for each dataset and notations used. In all, 80 runs per data were conducted for both the Wiki News Abstract and the SW. Experiments for all combinations for 300 dimensions were conducted on the BW corpus and additional runs for the window size 8 + skipgram (s1) + hierarchical softmax (h1) combination to verify the trend of quality of word vectors as dimensions are increased.

Preferably, more than one training instance would have been run per combination for each embedding and an average taken; however, the long hours involved made this prohibitive. Despite this, we randomly ran a few combinations more than once and confirmed the difference in intrinsic scores was negligible. For both downstream tasks, we evaluated the default Pytorch embedding, the pre-trained embedding by Mikolov et al. [1] (based on GoogleNews of 100B tokens) and ours. In each case, the dataset was shuffled before training and split in the ratio 70:15:15 for training, dev, and test sets. Batch size of 64 was used, and Adam was used as an optimizer. For each task, each experiment was conducted four times and an average value calculated.

5 Results and discussion

We obtain comparable accuracy in SA to ref. [32], though we used less than half of the dataset for training and also evaluated on a smaller different set. The WordSim output file from the Gensim library always has more than one value reported, including the Spearman correlation. The first value is reported as WordSim score1 in the relevant table. It is a cosine similarity variant. Table 3 summarizes key results from the intrinsic evaluations for 300 dimensions.^[2] Table 4 reveals the training time and average embedding loading time (in seconds), representative of the various models used. Tables 5 and 6 summarize key results for the extrinsic evaluations. Figures 4 and 5 present the line graph of SW for the eight hyperparameter combinations for different dimension sizes and the score trend for SW and BW corpora over several dimension sizes, respectively. Results for the smallest dataset (Wiki Abstract) are so poor, because of the tiny file size (15M), and there is no reason for reporting them here. Hence, we have focused on results from the SW and BW corpora.

Figure 4

SW: Analogy scores for ten Epochs (w: window size, s1: skipgram, s0: CBoW, h1: hierarchical softmax, h0: negative sampling).

Figure 5

Analogy scores for w4s1h1 of SW for five Epochs and w8s1h1 of BW for ten epochs (not drawn to scale from 400).

From Table 3, we observe that there is a general better performance of the hyperparameter combination: skipgram-negative sampling (s1h0). Indeed, the embedding with the highest analogy score is the one by Mikolov et al. [1]; however, the highest WordSim score1 and Spearman correlation values are obtained by the skipgram-negative sampling embedding of window size 8 of the SW. The analogy score of the GoogleNews embedding, which we tested, is confirmed in the original article [1]. It is based on a vocabulary size of 3,000,000 words while SW has a vocabulary size of 367,811. Furthermore, it can be observed from Table 4 that the training and the loading times for the embedding of the SW are the lowest. Information on the training time for the released GoogleNews model [1] is not readily available. The other two embeddings take multiple times longer to train or load. The marginal gain of the GoogleNews or BW embedding in analogy score over that of the SW is not directly proportional to the length of time required to train or load them. This may put the embeddings based on the larger corpora at a disadvantage in cases where factors like the training time and the energy spent on training are critical. Energy-saving and time-saving methods are crucial for the environment and humanity as a whole.

We observe from Figure 4 that analogy scores for all the embeddings start to drop after 300 dimensions for the SW dataset. The rate of decrease, however, differs for each embedding. Also, from Figure 5, we note that from 100 dimensions, scores improve but start to drop after over 300 dimensions for SW and after over 400 dimensions for BW, confirming a similar observation by Mikolov et al. [1]. This trend is true for all the combinations. Polynomial interpolation may be used to determine the optimal dimension in both corpora.

For the NER downstream task, our BW skipgram-negative sampling of window size 4 (w4s1h0) embedding performs best in F1 score. This embedding is also the best in analogy score among our models. Most pretrained embeddings outperform the default Pytorch embedding in F1 scores. With regards to SA, the default Pytorch embedding outperforms the pretrained embeddings in accuracy and F1 scores but is closely followed by our CBoW-negative sampling embedding (w8s0h0) of the SW. The combination of CBoW-negative sampling for both window sizes 4 and 8 (w8s0h0 and w4s0h0) of the SW performs relatively well in both the extrinsic tasks.

For alpha of 0.05 and power of 1, significance tests of the differences of two means for the F1 scores give p -values of 0 in both cases: for the 100B and the skipgram-negative sampling (w4s1h0) of the BW embedding for NER, and the CBoW-negative sampling (w8s0h0) for the SW for SA. Since both p -values are 0, the differences are unlikely due to chance, so we reject the null hypothesis in both cases.

6 Conclusion

This work analyses, empirically, relatively optimal combinations of hyperparameters for embeddings, specifically for Word2Vec. It further shows that for downstream tasks, such as NER and SA, there is no silver bullet! However, some combinations show strong performance across tasks. Performance of embeddings appears to be task-specific and high analogy scores do not necessarily correlate positively with performance on downstream tasks. This point on correlation is somewhat similar to results by Chiu et al. [35] and Wang et al. [12]. It was discovered that increasing embedding dimension size depreciates performance after a point. If strong considerations of saving time, energy, and the environment are made, then relatively smaller corpora may suffice or even be better in some cases. Indeed, hyperparameter choices are very important in NNs [10].

Future work that may be investigated are the performance of other architectures of word or subword embeddings in SoTA networks like BERT (based on a matrix of hyper-parameters), the performance and comparison of embeddings applied to other less-explored languages, and how these embeddings perform in other downstream tasks.