Unit Under Test Identification Using Natural Language Processing Techniques

2.1 Latent Semantic Analysis

LSA is an indexing and IR method that uses Singular Value Decomposition (SVD) to identify relationships between words in an unstructured text. The model is based on the assumption that words used in the same context have a similar meaning [10]. By extracting terms from the document's body it seeks to create relationships between individual documents. It is important to choose a right number of topics to generate because if too many topics are requested for a short document the algorithm also returns words that should not determine the resulting topic of the document and vice versa.

2.2 Latent Dirichlet Allocation

The model is a generative probabilistic model of a corpus and considers each document as a set of topics which characterize it. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words [11]. Each topic consists of a set of words in a certain proportion. Based on the number of topics required the model attempts to rearrange the topics distribution within the documents to achieve the best composition. It is also very important to determine the right number of topics that the algorithm returns. Cvitanic et al. [12] discuss LSA and LDA model differences in more detail.

2.3 Term Frequency-Inverse Document Frequency

A slightly more sophisticated model is tf-idf which tries to encode two different kinds of information - term frequency and inverse document frequency [13]. It expresses a statistical measure used to evaluate how important a word is to a document in a collection or corpus. Term frequency (tf) is the number of times the word appears in a document. Since document length can differ a term would appear much more often in long documents than shorter ones. Therefore the term frequency is often divided by the document length as a form of normalization.

Inverse Document Frequency (idf) measures how important a term is. E.g. the word class can occur in documents representing a java file very often, so when evaluating similarity it is not so important as a word that occurred only in a small number of documents. Thus it is important to weigh down the frequent terms while scale up the rare ones. It is possible to extend this model, for example using topic models (like LSA or LDA).

2.4 Random Projections

RP model tries to reduce vector space dimensionality [14]. Using a fine randomness the approach approximates tf-idf distances between documents and thanks to reduced vector space dimensions it is a very memory- and CPU-friendly model. The model is similar to LSA but it can reduce the vector space model to lower dimensions and in this way reduce the cost of computing resources with similar accuracy. More about random indexing of text samples can be seen in the article [15] written by Kanerva et al.

2.5 Hierarchical Dirichlet Process

The HDP model is a non-parametric Bayesian method using the Dirichlet distribution in the same way as the LDA. For LDA it is necessary to correctly estimate the number of topics by document size which can greatly affect the results. In HDP during a document collection, posterior inference is used to determine the number of topics needed and to characterize their distributions. It was proven by Wang et al. that HDP improved performance over LDA topic model [16].

3.1 Programming language and library

For projects’ source code analysis we chose Python language which is great for processing computationally difficult tasks. Also the availability of gensim [17] library for this language was crucial. The library is very popular in the NLP field and according to its author Řehůřek [18] gensim is the most robust, efficient and hassle-free piece of software to realize unsupervised semantic modeling from plain text.

3.2 Documents preparation

All analyzed projects are built on the Android platform and implemented in Java and/or Kotlin. For each project we recursively searched for files having the .java or .kt extension. Kotlin is designed to interoperate fully with Java so they use similar programming conventions and it is also suitable for our analysis. Of course, only the files of the project were included in the analysis, i.e. without dependencies and platform software development kit files. We already knew the names of the test classes from [6] so we divided the particular files into test and production set. The content of production classes served for model training and content of test classes for searching similarity and UUT detection.

File preprocessing was the same for both test and production classes. From the content of each file new line characters have been removed and the result was saved in a project-depended training file. One line in this file represented one document for further processing.

Since the result is highly dependent on the quality of the input data and is governed by the idiom ”garbage in, garbage out”, it is very difficult to assume in a non-natural text how it should be properly preprocessed. To find out how to prepare input documents derived from the source code in the best way (RQ3), we incrementally created 5 versions of document preprocessing:

1. Full version - original file version, removed only \n chars.

2. Word split - all camelCase or snake_case words has been split. Words out of base conventions, such as ORMLite, remained unsplit.

3. Removed Java keywords - all Java keywords have been discarded.

4. Removed comments - multi- and one-line comments discarded.

5. Removed imports - all Java imports removed.

Incremental preprocessing means that, for example in the 4^th iteration, both Word split and Removed Java keywords have been included. Another preprocessing of documents that applied to all iterations was the removal of frequently occurring English words using nltk library, such as and, a, the, etc. At the same time stemming over the documents has been executed, where inflected or sometimes derived words have been reduced to their word stem, e.g. cars to car. The last step was to remove words that occurred only once in the corpus of training documents to eliminate their negative impact on results.

3.3 Model training

To train the LSA model only the necessary parameters were supplied - number of topics, dictionary (BoW) and corpus (of vectors). For the LDA model training we also set the alpha = auto parameter which means the model learns asymmetric prior from provided corpus. From the alpha attribute LDA model computes theta that decides how the topic distribution is drawn. The last special parameter set for the LDA model was passes = 20 which expresses the number of passes through the corpus during training. In terms of statistics, more training means statistically more accurate results.

For tf-idf, RP and HDP models corpus and BoW were provided. It was also possible to define the topics number for the RP model, but the default value was retained. For tf-idf the parameter normalize = True has been set to obtain better results from the model (explained in Section 2.3). Creating a bag-of-words representation for all models in the form of a dictionary (id + word pair) and creating a corpus of sparse vectors was relatively easy using the functions of gensim.

3.4 Estimating number of topics

Evaluation of the quality of LSA and LDA models, which is significant for results of both considered models, can be determined by topic coherence that is a measure used to evaluate topic models. The topic coherence is applied to the top N words from the topic and it is defined as the average/median of the pairwise word-similarity scores of the words in the topic [19]. A good model will generate coherent topics with high coherence scores. Good topics are those that can be described by a short label. Since the number of topics is dependent on the nature of the documents, we searched for the highest coherence value for each iteration and project.

Training a large number of models is a very time consuming task so in the early stages of the experiment we tried to obtain an approximate range to try in. After multiple tests we decided to calculate coherence values from 7 to 50 for each model, project and iteration, and the model with the highest value has been chosen for the analysis. Estimating the best number of topics was executed only for LSA and LDA models.

3.5 Evaluation of document similarity

When using different NLP models we focused on a single aspect of possible similarities, i.e. on apparent semantic relatedness of their texts (words), just a semantic extension over the boolean keyword match. Modern search engines also take into account random-walk static ranks, hyperlinks, etc. Gensim basically uses cosine similarity [20] to determine the similarity of two vectors and it is a standard measure in Vector Space Modeling.

Every single search query was made up of the content of a test class. We created bag-of-words for each document (test body) and converted it to the corresponding NLP model space. Subsequently, an index has been created from the trained model against which the query was evaluated. Similarities to all production classes were calculated and we obtained the result as (document_no, similarity_value) pairs where similarity_value ∈ 〈−1, 1〉. The greater similarity_value the more similar document. Every document_no has been paired with the value stored in a relation database created during training and document preparation to identify a particular production class.

4.1 Search speed

The time of searching/comparing a document similarity in the index is very important, especially when a realtime response is expected. NLP methods could be part of an integrated development environment (IDE), e.g. by displaying tooltips to navigate from test to related parts of the code or UUT, it is desirable that the search is fast enough. In the Table 2 statistics of mean search time in the prepared index are presented to help answer the RQ1.

As can be seen, the number of documents in the index has a big influence on the search speed, as the size of the matrix increases proportionally with the number of documents and thus the search response. It is interesting to observe an increase in search time for the tf-idf, RP and HDP models. For example, tf-idf search was 6 times and RP up to 185 times slower in average than LDA/LSA one. From the experiment the most powerful search model is LDA with an average time of 31.82 ms. For real-life usage the responses of the LSA and tf-idf models are also acceptable and could be used in the IDE where further research following this experiment should be directed. According to Tolia et al. [21] response times below 150 milliseconds do not impact user productivity and experience, but due to the complexity of this issue, which is related to thousand of source code lines, times are sufficient (compared to full project search on indexed files in an IDE).

4.2 Optimal topic number

Model training lasted the longest time especially because of searching for the optimal topic number. The tests were performed on a 12-core Intel Core i7-5820K CPU with 12GB RAM and Debian GNU/Linux 9 installed. Nevertheless, analysis was performed in one thread only to simulate the use in a real environment, e.g. in the background of an IDE. The average search times for the highest coherence value can be seen in Table 3.

To answer RQ2 the Table 4 was created. As can be seen the mean value for LSA is relatively stable. Although the maximum deviation of the LSA value is 40, this situation occurred in only one case, i.e. it was just an exception that could be ignored. For the source code analyses using the LSA model it is therefore possible to use a relatively stable value of topic numbers in the range 7–10.

When using LDA the selected topic numbers were more diverse as the model is less stable. This can be obtained in multiple model training with the same data when the results vary slightly. That's why topic numbers are more diverse than in LSA. Most often the topic number for a project was in similar values and in the difference range of 10 units. However, the differences between the particular projects were large and based on this data it is not possible to determine the recommended topic number for LDA.

4.3 Training times

The differences between the models’ training times for the different projects were minimal, i.e. a few milliseconds. Larger differences (in seconds) were noted for LDA and HDP models, indicating the complexity of using dirichlet allocation. The fastest average training time of 8ms was achieved by the tf-idf model. The greatest decrease of training time in a particular NLP model was recorded in the 4^th iteration when comments were removed. By removing them a lot of the training data have been lost so finding the best coherence value in the 4^th iteration was faster: 2.17 times for LSA, 2.83 times for LDA and 2.5 times for other models. Despite the increased speed a large amount of potentially natural text in the source code has been lost and as pointed out later (Section 4.5) the loss of this data negatively affects the results.

4.4 Accuracy of UUT identification

Since we assume that manually identified UUTs are correct it is possible to determine the accuracy of a particular model based on the order of production class in the search result. The Figure 1 shows the frequencies in the search queries for manually labeled production classes as UUT and the presented frequencies are generalized for all projects.

Figure 1

Position frequency of manually identified UUT for all models.

First let's discuss the results of LDA and LSA models which are similar from the viewpoint of finding the best topic number for the gensim library. As can be seen, LSA performed much better than LDA. The LSA is based on the frequency of words in the documents and as was found in [6], the words between the test and UUT are very similar which positively influenced the result. The accuracy of the LDA model was very low, in the first five results the correct UUT appeared only 2 times. Although the LSA achieved 82 correct UUTs in the first five results for all iterations it is still only 13% success rate which is considerably inadequate! and only 5% of UUTs were marked absolutely correctly in all iterations and solely by the LSA method.

The if-idf model was the most accurate because it marked 46% of UUTs classes on the top five positions and 22% on the first position in average, independently of preprocessing iteration. This result can be considered as excellent compared to the original results presented in [8]. Taking into account also the preprocessing of the input data, it can be seen that in 3^rd iteration up to 40 UUTs have been accurately labeled, i.e. 33% success rate of rightly marked UUTs. Similarly good results were achieved by the RP model with a success rate of 36% for the first 5 positions and exact identification of 16%, but it is relatively inaccurate compared to tf-idf. The HDP model achieved only 8% detection in the first 5 places and 4% in the exact identification of UUT, but this result is also influenced by the fact that HDP is still rough around its academic edges.

In response to RQ3, our results indicate that the most accurate model for use on the source code is tf-idf one. As mentioned in the previous section, the search speed in the index is sufficient, so this model seems to be the most suitable for real use. In the results it is necessary to take into account the fact that for the 6 test classes, which tested multiple production classes at once, the most tested class was chosen as the correct UUT (discussed in Section 3).

4.5 Data preparation

A more detailed look at the best search results in each iteration is needed to respond to the RQ4 (see Table 6). As can be seen, word split and removal of java keywords (I2 + I3) has the greatest impact on the accuracy of the results through all used NLP models. Our expectation was that when comments are removed the results will get worse because there is a potential for sole natural language in the comments. In Table 6 it can be seen that removing comments (I4) and imports (I5) had a significant impact on accuracy only for the tf-idf model.

It also shows that the meaning in the code is most often expressed directly in the names of the identifiers, i.e. class, methods and variables names. On the other hand, the removal of imports (I5) did not have a big influence on the accuracy of the results, except in some rare cases. The assumption was that the test and production class would also use similar imports and this similarity could help identify UUT, but this was not shown to be crucial in this experiment. Using word splitting (I3) generally gave rise to the biggest increase in precision of methods and removing the java keywords (I4) didn’t have so much impact on accuracy, compared to I3. At the same time 3^rd iteration was one of the fastest ones in terms of finding the best coherence value and searching in the index.

RQ5 deals with a very important aspect for real IDE usage and that is the duration of document preprocessing (see Table 7). As can be seen, there is a significant difference in preprocessing duration between I2 and I3. For the most accurate model tf-idf we have already achieved very good results during I2 and preprocessing time was remarkably short (note that the times in the table are for all 5 projects). In terms of the accuracy, training times, speed of preprocessing and searching the index the 2^nd iteration with the tf-idf model is the best combination.

4.6 Detected errors of manual identification

In the experiment, we unexpectedly found that 8 classes manually labeled as UUT were not in the corpus. Manual testing was done in Android Studio IDE where we used references created directly by IDE. After a more detailed analysis we found that production classes that were not in the corpus were incorrectly labeled. Incorrect identification was due to references to generated source code that were not present in the file system of the project without run. The method can be therefore used to prevent such errors.