Aspect-Oriented Sentiment Analysis: A Topic Modeling-Powered Approach

Abstract Because of exponential growth in the number of people who purchase products online, e-commerce organizations are vying for each other to offer innovative and improved services to its customers. Current platforms give its customers innovative services such as product recommendations based on their purchase histories and location, product comparison, and most importantly, a platform for expressing their experience and feedback. It is important for any e-commerce organization to analyze this feedback and to find out the sentiment of the customers for giving them better products and services. As large reviews may contain feedback in a mixed manner where a customer gives his opinion on different product features in the same review, finding out the exact sentiment is tedious. This work proposes aspect-specific sentiment analysis of product reviews using a well-sophisticated topic modeling algorithm called latent Dirichlet allocation (LDA). The topic words, thus, extracted are mapped with various aspects of an entity to perform the aspect-specific sentiment analysis on product reviews. Experiments with synthetic and real dataset show promising results compared to existing methods of sentiment analysis.


Introduction
Platforms such as social networks, micro-blogs, online reviews, and discussion forums are growing very fast, and thus, the need for analyzing the sentiments of the users are also increasing. Sentiment analysis proves to be very effective in businesses and social domains because opinions matter and is critical for all human activities, and thus, they have become the key influence of human behavior. There are two broadly categorized types of the textual representation: one is fact; another is opinion. While facts are objective expressions about entities, events, and their properties, opinions are subjective expressions that describe people's sentiments toward entities or events. Product reviews in e-commerce platforms help the customer to analyze the product and take better judgments based on its previous usage history.
Sentiment analysis is the process of extracting subjective information from natural language text, and it also expresses the opinion or view of a user toward a topic. Sentiment means feelings, and it includes attitudes, emotions, and opinions, and sentiment is a subjective impression, not a fact. Sentiment analysis can be done by applying natural language processing, statistical methods, or machine learning techniques to identify, extract, or characterize the sentiment content of a text unit. The insights and applications from sentiment analysis were useful in other areas, like politics, law, policy making, sociology, and psychology. In general, humans are subjective creatures, and opinions are important, and being able to interact with people on that level has many advantages for the information system.
The main challenges in sentiment analysis are that people express their opinions in complex ways and the relevance of lexical content in text. There are many possibilities for classification such as users, texts, words, sentences, etc. The words or short phrases are the building blocks of the sentiment expressed. The next step is to find the polarity keywords to identify the sentiment behind them. For example, there will be some relation between the words and positive reviews, and such keywords can be found out by data-driven methods for better accuracy. In the case of social networks like Twitter and Facebook, sentiment is most succinctly represented with emotions or smileys, and in such scenario, emotion recognition is performed to capture the sentiment expressed in them [35]. Therefore, customers are not only interested in the whole quality of the product but also some specific aspects of the product also. For example, considering a laptop as an example, customers are not only interested in the whole product but also into some other aspects such as "screen quality", "processor generation", or "random-access memory (RAM) speed". This leads to the extraction of various aspects of a product [31] to perform sentiment analysis of product reviews. For each class of products, there is a set of very general categorical aspects that are common to all products in that class. Aspects can be identified either manually or through an appropriate automated method. This work aims to identify aspects and then perform the sentiment analysis of product reviews using topic modeling. The main contributions of this paper are summarized as follows: 1. Outlines a brief explanation of aspect-oriented sentiment analysis and topic modeling, which are wellstudied and addressed areas in text mining. 2. Proposes a topic modeling-powered approach for aspect extraction and aspect-based sentiment analysis from unstructured product reviews expressed by users of various products. 3. Verify experimentally the effectiveness and usefulness of proposed algorithms in extracting aspectoriented sentiments from real-world and dynamic datasets.
Organization of this paper: The remainder of this paper is organized as follows. Section 2 describes the state-of-the-art in sentiment analysis; Section 3 throws light on some background of latent Dirichlet allocation (LDA), which is a widely used topic modeling algorithm. The experimental setup used in this work is explained in Section 5, and Section 6 presents the results and a detailed evaluation of the same. The author concludes the paper, and some interesting future dimensions are given in Section 7.

Related Work
In this section, we discuss some of the very recent works on sentiment analysis from unstructured text specifically focusing on aspect-specific sentiment analysis. Sentiment analysis is an ever-evolving research area, and this caused rapid generation of a vast array of research publications. Because of space constraints, we, here, mention some of the prominent works that are closely related to our proposed method. Shelke et al. [30] proposed an approach for domain-independent aspect-oriented sentiment analysis for product reviews. This unsupervised approach uses SentiWordNet lexical resource to determine the polarity of identified features. A client perception-based product feature mining algorithm was introduced by Li et al. [14] that makes use of sentiment orientation. The feature and sentiment orientation of products are sorted by the weights of user interest that help customers access review orientation more effectively. Another notable approach in aspect-oriented sentiment analysis was introduced in the literature by Akhtar et al. [1] that is composed of a two-step approach, viz., aspect term extraction and sentiment classification. Their pruned, compact set of features performed better compared to the chosen baselines that uses a complete set of features for sentiment classification. An ensemble-based particle swarm optimization they construct in the second phase using maximum entropy, conditional random field and support vector machine outperformed the baseline. A linguistic rule-based approach for aspect-level sentiment analysis for movie reviews was reported in the literature, which was devised by Piryani et al. [25]. Their method used a linguistic rule-based approach, which identifies the aspects from movie reviews, locates opinion about that aspect, and computes the sentiment polarity of that opinion using linguistic approaches. The system then generates an aspect-level opinion summary, and the authors claim that the system shows good accuracy, and this can be deployed in an integrated opinion profiling system. Another interesting work that finds out aspect-level sentiment analysis on tweets about diabetes was introduced by Salas-Zárate et al. [28]. The authors calculate the sentiment of aspects by considering the words around the aspects using n-gram-based approach. They claim that this approach showed a precision of 81.93 % and an f-measure of 81.24 %.
A very recent comprehensive survey on text sentiment analysis from opinion to emotion mining by Yadollahi et al. [34] was reported in the literature very recently. The authors present the state-of-the-art methods in this area and proposes a taxonomy of sentiment analysis, a survey on polarity classification methods, emotion theories, and emotion mining research. Schouten et al. [29] proposed a novel approach for supervised and unsupervised aspect category detection for sentiment analysis with co-occurrence data. Their proposed approach applies association rule mining on co-occurrence frequency data to find aspect categories. An approach for aspect-based rating prediction on reviews using sentiment strength analysis was introduced by Wang et al. [32]. Previous works on sentiment analysis used reviewer's sentiment orientation, and in this work, the authors focus on the sentiment strength users expressed in the reviews.
A feature selection approach using multi-objective optimization for aspect-based sentiment analysis was reported by Akhtar et al. [2] in which the authors incorporated the concepts of multi-objective optimization, distributional thesaurus, and unsupervised lexical induction. This work uses support vector machines for sentiment classification and conditional random fields for aspect term and opinion target expression extraction tasks. Their experiments on benchmark setups of SemEval-2014 and SemEval-2016 shared tasks show that the proposed method achieved the state-of-the-art on aspect-based sentiment analysis for several languages. Another comprehensive and systematic literature review on opinion types and sentiment analysis techniques was contributed in the literature by Qazi et al. [27] in which the authors identified nine practices of review types, eight standard machine learning classification techniques, and seven practices of concept learning Sentic computing techniques. Their review offers insights on promising concept-based approaches to SA, which leverage commonsense knowledge and linguistics for tasks such as polarity detection. In addition to this, the practical implications of the methods are also explained in this review.
Ontology-based approach for enhancing aspect extraction process by identifying features pertaining to implicit entities was reported recently, which was proposed by Marstawi et al. [19]. Their Ontology-Based Product Sentiment Summarization (OBPSS) method outperformed other existing summarization systems in terms of aspect extraction and sentiment scoring. Our proposed method devise algorithms for aspect-oriented sentiment analysis, which are guided by topic modeling algorithm. This method model topics from product reviews and the obtained "topics" are mapped to corresponding aspects. Then, the polarity of these aspects are computed to summarize the overall sentiment of customers toward a product.
Very recently, Hazarika et al. proposed an approach for modeling inter-aspect dependencies for aspectbased sentiment analysis [9]. Majority of the present neural-based approaches largely ignore the inter-aspect dependencies and only capture aspect and its contextual information. In their proposed work, authors incorporated the inter-aspect pattern by simultaneous classification of all aspects in a sentence along with temporal dependency processing of their corresponding sentence representations using recurrent networks [9]. Another recent work that proposed a general two-stage approach for target-based sentiment analysis was introduced by Wang et al. [33]. In this approach, the first stage extract and group target-related words, and the second stage separates aspect and opinions from these identified groups. The proposed method showed better performance when there were no sufficient labeled data was available.

Background: Latent Dirichlet Allocation (LDA)
Topic modeling is a method for analyzing large quantities of unlabeled text data. A topic is a probability distribution over a collection of words, and a topic model is a formal statistical relationship between a group of observed and latent random variables that specifies a probabilistic procedure to generate the topics -a generative model [5]. LDA is the most popular and simplest topic model. It is a generative for text and other collections of discrete data that generalizes or improves on several previous models including PLSI (probabilistic latent semantic indexing). LDA is a generative probabilistic model for collections of discrete data such as text corpora. It is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities, and the topic probabilities provide an explicit representation of a document in the context of topic modeling [6]. In LDA, we represent a word as the basic unit of discrete data, defined to be an item from a vocabulary indexed by {1. . .V}. A document is a sequence of N words denoted by w = (w 1 , . . . w n ), where w n is the nth word in the sequence, and a corpus is a collection of "M" documents denoted by D = W 1 . . . W m . The generative process for each document "w" in a corpus "D" is represented as follows [5]: 3. For each of the N words w n : (a) Choose a topic z n ∼ Multinomial (θ) (b) (Choose a word w n from p(w n | z n , β), a multinomial probability conditioned on the topic z n .
A k-dimensional Dirichlet random variable θ can take values in the (k − 1) simplex (a k-vector θ lies in the and has the following probability density on this simplex: where the parameter α is a k-vector with components α i > 0, and where Γ(x) is the γ function. Given the parameters α and β, the joint distribution of a topic mixture θ, a set of N topics z, and a set of N words w, is given by: where P(z n |θ) is simply θ i for the unique i such z n = 1. Integrating over θ and summing over z, we obtain the marginal distribution of a document as follows: Finally, taking the product of the marginal probabilities of single documents, we obtain the probability of a corpus as:

Aspect-Oriented Sentiment Analysis: A Topic Modeling-Powered Approach
In this section, we describe our proposed framework for finding aspect-oriented sentiment of product reviews expressed by customers. Those reviews mostly contain mixed opinions so that individual word-level polarity computation may not be useful. Our topic modeling-powered approach first models topics from customer reviews, which are a collection of word uni-grams and then extracts aspects from it. Based on the probability, we map these topics to corresponding aspects, and then aspect-level polarity is computed, and using Naive Bayes classifier, we classify sentiments. The overall work flow of the proposed approach is shown in Figure 1.
In this work, we used Amazon product review dataset, which consists of a large collection of review texts on mobile phones. Pre-processing is performed for converting raw text file into a well-defined sequence of linguistically meaningful units. This is an essential part of this system, as the characters, words, and sentences identified at this stage are the basic elements passed to all other processing stages. The words such as "and", "the", "but", and other punctuation and hyperlinks are removed from the reviews. Then, the LDA algorithm is executed on top of this cleansed dataset to generate topics. In the case of hundreds of review documents from this dataset, topic modeling will be a good approach to understand something of what the archive contains without reading every document and this will be helpful for extracting aspects from the review texts. Table 1 represents some topics that are extracted by topic modeling. Algorithm 1 takes the product review corpus as the input, and the output will be aspect-specific sentences. First of all, the review corpus is pre-processed by removing stop-words and symbols. Then, LDA is performed

Topic Topic keywords
Topic 1 Software treo palm keyboard blackberry device pda data email pc sync computer windows web internet mobile version support contacts Topic 2 Charger car cable work usb works plug charging power fine cord adapter charge worked device port doesn plugged connection Topic 3 Buy money worth love cheap thing blue bought quot buying don people dont color waste paid cost cool wanted Topic 4 Phone phones cell motorola nokia razr samsung lg reception features flip dropped calls drop basic sanyo vx razor krzr Topic 5 Small size pocket design makes bit feel find big large nice hand easily carry fact easier smaller style buttons on the pre-processed corpus to do topic modeling, and thus, topics are extracted. Each topic is then mapped to a particular aspect, and for each aspect, the algorithm scans the pre-processed corpus to find a match with topic words to extract aspect-specific sentences. Thus, for each aspect, the aspect-specific sentences will be written to a text file. Algorithm 2 takes the file containing aspect-specific sentences as the input, and it calculates the polarity score for each such sentence. The polarity score determines the sentiment of the sentence. If the polarity score of a sentence is greater than zero, the sentiment is "Positive", and if the polarity score is less than zero, then the sentiment is "Negative". If the polarity score is equal to zero, then, the sentiment toward the particular sentence is said to be "Neutral". Algorithm 3 takes the total polarity count of aspect-specific sentences as the input, and it expresses the sentiment on that particular aspect. If the total positive polarity count is higher than the negative and neutral polarity counts, then the sentiment toward the particular aspect is said to be "Positive". Similarly, if the total negative polarity count is higher than the positive and neutral polarity counts, then the sentiment toward a particular aspect is said to be "Negative". If the total neutral count is higher than the positive and negative polarity counts, then the sentiment toward the particular aspect is said to be "Neutral".

Experimental Setup
This section details the experimental setup we have used for implementing proposed algorithms on Amazon product reviews available on public web. A brief description of datasets used in this experiment and the experimental testbed details are discussed below.

Dataset Description
We use Amazon product review dataset available at the url http://jmcauley.ucsd.edu/data/amazon/links. html. This dataset contains product reviews and associated metadata from Amazon, including 142.8 million reviews spanning for the period May 1996 to July 2014. This dataset includes reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features), and links (also viewed/also bought). It contains reviews for a variety of product categories such as books, electronics, beauty, automotive, etc. For this current experiment, we used "Cell Phones and Accessories" category and used 194,439 reviews for our experiment.

Experimental Testbed
All algorithms discussed in this paper were implemented on a server configured with AMD Opteron 6376 @ 2.3 GHz having 16 core processor and 16 GB of main memory. These algorithms were programmed using Python 2.7 and removal of stopwords and other pre-processing were done using Python NLTK (Natural Language Tool Kit) [4], and stopword list for English language was used. We also removed punctuation, URLs, and other special characters to make the experimental ready copy of our dataset. We observed that the user reviews contain emojis, which plays an important role in expressing the user sentiment, but those emojis cannot be handled by our topic modeling algorithm. We also removed emojis from the dataset and considered only text as our input to the LDA algorithm. For topic modeling, we used MALLET (MAchine Learning for LanguagE Toolkit), which is freely available at the url http://mallet.cs.umass.edu/. For modeling topics, we chose 300 as the number of iterations in Gibbs sampling as by trail and error method we found that Gibbs sampling approaches the target distribution after 300 rounds of iterations. The α and β parameters of LDA was set to 50/Z and 0.01, respectively.

Baselines
We compare our proposed algorithms with two state-of-the-art methods available in the literature for aspectoriented sentiment analysis. The first baseline [19] is an approach that uses ontology for extraction of aspects and eliminates lexicon-based sentiment scoring issues. Thus, they claim that this will improve the accuracy of sentiment analysis. The second baseline [25] is a linguistic rule-based approach for aspect-level sentiment analysis of movie reviews. The authors devised a linguistic rule-based approach, which identifies the aspects from movie reviews, locates opinion about that aspect, and computes the sentiment polarity of that opinion using linguistic approaches [25]. We chose these two baselines as we found these methods closely associated with our proposed algorithms. A comparison result in terms of Precision, Recall, and Accuracy is given in the experimental results section.

Results and Evaluation
In this section, we present the results obtained from the experimental setup discussed in Section 5. The topics generated by the LDA algorithm is shown in Table 1. Because of space constraints, we show only the first five topics generated by the LDA, which are word unigrams. Each of these topics are then mapped to aspects identified from the product descriptions such as Network, Battery, Price, etc., and the results of such mappings are shown in Table 2. In Table 3, we show the polarity score and sentiment of the review text for the "Battery" aspect of the product. Various polarity counts for such identified aspects are given in Table 4. In Table 5, we outline the result of our aspect-oriented sentiment analysis algorithm. For further analysis and interpretation, we plot the true positive rate (TPR) and false positive rate (FPR) using the receiver operating characteristics (ROC) plot, and such graphs for the aspects "Battery and Body", "Display and Network", "Price and Service" are shown in Figures 2-4, respectively. Figures 5 and  6 shows the charts depicting the precision, recall, and f-measure comparisons of our proposed method Based on the number of words that are related to an aspect, we map the entire topic to the corresponding aspect.    with the chosen baselines for the aspects "Battery and Body", "Display and Network", "Price and Service". We compare our proposed algorithms with two state-of-the-art methods for establishing the usefulness and better accuracy of our proposed algorithms. The results in terms of precision, recall, and f-measure are given in Table 6, and the data clearly shows that our proposed method outperforms our chosen baselines for all the aspects we have considered. Figure 7 shows the screenshot of a web application we have developed to visualize the results of our aspect-oriented sentiment analysis framework implemented on real-time data. The current screenshot shows the result of aspect-based sentiment analysis for the mobile Xiaomi Redmi Note 3, and the corresponding user reviews are collected from the famous customer review platform called MouthShut. Our proposed algorithms could clearly show the user sentiments on different aspects of the mobile, and the algorithms were found to be highly scalable. This system has already been implemented for some e-commerce vendors, and they are using it as a pilot version.   This system was extended to production level, and a number of vendors are already using a pilot version of the same.