Context-based sentiment analysis on customer reviews using machine learning linear models

Methodologies and process flow

Recently, researchers are used deep learning and neural network models for SA problems, however neural network approach cost more compare to traditional baseline methods for both supervised and unsupervised learning. The combination of right methodologies and text classification algorithms are contributed to overcome SA models accuracy and performance problems. The following process flow diagrams in Fig. 1 show the steps followed for this experiment. In general, the data pre-processing step, data obtain from publicly available customers review data is very often incomplete, inconsistent and filled with a lot of noise and it is likely contained errors not suitable for training and testing the machine learning models.

The following minimal syntactical data preprocessing steps of lowercasing all words, removing new lines, punctuation, special characters and stripping recurring headers are needed for neural networks and embedding models. To improve data quality, introduced additional steps which includes stop words removal, text standardization, spelling correction, correcting the negation words, tokenization, stemming, and Exploratory Data Analysis (EDA). Figure 2 shows the three proposed models input and output data flow. These models are customized to fit the dataset. In this experiment, added additional layers SA-BLSTM to handle large volume of customer reviews and speech transcript data. fastText and SVM are linear models. All these pre-trained models and developed algorithms can be used for production purpose on real-time SA business applications.

Proposed models

LSVM, fastText and SAB-LSTM models are used for this experiment, before building the models, reviewed algorithms towards solving the short and long text classification and SA problems. In these following sections, explained all these three custom model architectures.

Support Vector Machine (SVM)

SVM is used for text classification problems, this algorithm is viewed as a kernel machine, and the kernel functions can be changes based on the problem. Definition of Support Vector Machines (SVM): It performs classification by finding the hyperplane that maximize the margin between the two classes. The support vector is the vectors that define the hyperplane. For instance: given pictures of apples and oranges, state whether the object in question is an apple or an orange. Equally well, it can predict whether a customer is satisfied or not satisfied given customers positive and negative sentiment data. SVM performs classification by finding the hyperplane that maximize the margin between the two classes. In other words, SVM is that the partition which segregate the classes. Figure 3 shows an example and definition of a point and vector, plotted a point A(4, 2) and any point, (1)${\displaystyle x=\left(x_{1,}{x}_2\right),x\ne 0}$

A vector is an object that has both a magnitude and a direction. In geometrical term, a hyperplane is a subspace whose dimension is one less than that of its ambient space. If a space is in 3-dimensional then it’s a plane, if a space is in two dimensions, then it’s a line, if the space in one dimension, then it’s a point. Figure. 4 shows the hyperplane definition.

The linear and non-linear classifier (Kowsari et al., 2019) data separation shown in the Fig. 5 for the 2-dimensional dataset. If the dataset is separable then linear kernel works well for classification. Because of the following reasons, the linear kernel is used for the text classification. The most text categorization problems are linearly separable and the linear kernel works well with lot of features and also less parameters to Joachims (1998) optimize for training the model.

Figure. 6 shows the linear kernel (Crone, Lessmann & Stahlbock) SVM model. There many kernel functions have been developed over the years; a kernel is a function, that returns the result of a dot product performed (Alexandre Kowalczyk, 2017) in another space.

The linear kernel is simplest kernel function k(x, y), it is given by the inner product <x, y >, and an optional constant c. (2)${\displaystyle k\left(x,y\right)=x^T+c.}$

Linear kernel is used in this experiment for text classification. The Polynomial kernel, RBF kernel and String kernel functions can be used for other classifications problems.

fastText

Facebook AI research (FAIR) lab release an open-source free library called fastText for text representation and classification. It’s a lightweight method and work on standard generic hardware with multicore CPU. fastText approach evaluated for tag prediction and sentiment analysis by FAIR. fastText experiments (Joulin et al., 2016) show that it is often on par with recently proposed DL methods in terms of accuracy, performance, faster for training and evaluation. Facebook allows research community to build the models on top of the fastText open-source code. fastText introduce a new word embedding approach an extension of the continuous skipgram and Continuous Bag of Words (CBOW) model like word2vec, where each word is represented as a bag of character n-grams. The original version of fastText is trained on (Nitsche & Halbritter 2019) Wikipedia and its available for 294 languages. The main difference between word2vec and fastText is that fastText sees words as the sum of their character n-grams and it treats a vector representation is associated to each character n-gram and words being represented (Bojanowski et al., 2017) as the sum of these representations. This new approach has clear advantages, as it can calculate embeddings even for out-of-vocabulary (OOV) words. However, word2vec embedding approach treat word as the minimal entity and try to learn their respective embedding vector, in case if the word does not appear in the training corpus, then it fails to get word vector representation. Figure 7 shows the CBOW and Skip-gram model architecture, The CBOW is the distributed representation of context model, it predict the words in middle of a sentence based on surrounding words. However, the Skip-gram predicts context within a sentence (Mikolov, Le & Sutskever).

The Skip-gram maximize the average log probability for input of training words w, (3)${\displaystyle w_{1,}{w}_2,w_3\dots ..,w_T.}$

Eq. (4) is used for computing probability. (4)${\displaystyle \frac{1}{T}\sum _{t=1}^T\sum _{j=-k}^{T}logp\left(w_{t+j}|w_t\right).}$

Here k represents the size of the training window and function for w_t word in the middle.

−k to k is the representation inner summation and it computes the log probability of the word w_t+j prediction for word in the middle w_t. The outer summation words are based on the corpus used for model training.

The skip-gram model consists of input u_w, and output v_w, vectors associated with each word w. The following probability (Eq. 5) is used to predict the word u_i, from w_j. (5)${\displaystyle p\left(w_i|w_j\right)=\frac{exp\left(u_{w_i}\top v_{w_j}\right)1}{\sum _{l=1}^{V}exp\left(u_{w_l}\top v_{w_j}\right)}.}$

Here, the total number V of words in the given vocabulary.

CBOW and skip-gram models are obtaining the semantic information during the large datasets used for training. The words which are closely related has the similar vector representation the words. For example, school, college, university, education words are having similar context, similarly orange and apple are having similar context representations.

The Fig. 8 shows the architecture (Joulin et al., 2016) of a simple linear model of fastText with N gram features. (6)${\displaystyle x_{1,}{x}_2,\dots ..x_N.}$

The features are embedded and averaged to form the hidden variable. This model is a simple neural network with only one layer. The bag-of-words representation of the text is first fed into a lookup layer, where the embedding is fetched for every single word. Then, those word embedding are averaged, so as to obtain a single averaged embedding for the whole text. At the hidden layer we end up with n words x dim number of parameters, where dimension is the size of the embedding and n words is the vocabulary size. After the averaging, we only have a single vector which is then fed to a linear classifier: we apply the softmax over a linear transformation of the output of the input layer. The linear transformation is a matrix with dimension1 x_N output, where N output is the number output classes (Mestre, 2018). The following Eq. (7) is the negative log likelihood function of fastText model. (7)${\displaystyle -\frac{1}{N}=\sum _{n=1}^N{y}_{n}log\left(f\left(BA{x}_n\right)\right).}$

Here, x_n represents the n-gram feature of the word,

A represents the lookup matrix of the word embedding,

B represents the linear output of the model transformation,

f represents the softmax function.

The softmax function calculates the probabilities distribution of the event over n different events. The softmax takes a class of values and converts them to probabilities with sum 1. So, it is effectively squashing a k-dimensional vector of arbitrary real values to k-dimensional vector of real values within the range 0 to 1. Eq. (8) is the softmax function f of fastText. (8)${\displaystyle \text{softmax}\left(z\right)=\frac{\mathbf{exp}\left(\mathit{z}\right)}{\sum _{\mathit{k}=1}^{\mathit{K}}\mathit{exp}\left(\mathit{z}\right)}.}$

The fastText has the following tuning parameters: Epoch: By default, the model is trained on each example for 5 epochs, to increase this parameter for better training specify the number of epoch argument. Learning rate (lr): The learning rate controls how “fast” the model updates during training. This parameter controls the size of the update that is applied to the parameters of the models. Changing learning rate implies changing the learning speed of our model is to increase (or decrease) the learning rate of the algorithm. This corresponds to how much the model changes after processing each example. A learning rate of 0 would means that the model does not change at all, and thus, does not learn anything (Mestre, 2018). Note that this calculation of the best model is going to be quite expensive. There is no magic formula to find the hyperparameters for the best model. Just taking one hyperparameter, the learning rate, would make the calculation impractical. This is a continuous variable and it would need to feed in each specific value, compute the model, and check the performance. Loss function: In this we are using softmax as loss function. The most popular methods for learning parameters of a model are using gradient descent. Gradient descent is basically an optimization algorithm that is meant for minimizing a function, based on which way the negative gradient points toward. In machine learning, the input function that gradient descent acts on is a loss function that is decided for the model. The idea is that if we move towards minimizing the loss function, the actual model will “learn” the ideal parameters and will ideally generalize to out-of-sample or new data to a large extent as well. In practice, it has been seen this is generally the case and stochastic gradient, which is a variant of gradient descent, has a fast-training time as well. Since it needs to obtain the posterior distribution of words, the problem statement is more of a multinomial distribution instead of a binary.

Figure 9 shows the text documents process flow using fastText linear model. In this model text classification pipe-line, raw text documents are processed using data pre-processing steps described in Fig. 1 and processed text data tested using fastText model, the model output classified the output into two classes (Satisfied and Not-Satisfied). fastText is a classification algorithm and C++ used to compile fastText mode. It provides high accuracy as well as good performance (Zolotov & Kung, 2017) during training and testing the model.

SA-BLSTM model

SA-BLSTM is a sequence processing model (Kumar & Chinnalagu, 2020). The Bidirectional LSTM is used in this model. The extended LSTM architecture is shown in Fig. 10 with Input, Output, multiplicative Forget gates and all the gates are using (Mikolov et al., 2013) activation function f sigmoid.

The Constant Error Carousels (CEC) is the central feature of LSTM (Kumar & Chinnalagu, 2020) and it solves the vanishing problem. CECs back flow is constant when there are no input or error signals to the cell. Input and output gates protect CECs error flow from forward and backward activation. If the gates are closed or the gates activation is around 0 then the irrelevant input will not enter the cell. The Forward pass LSTM computes the output of the network given the input data, the Backward pass LSTM computes the output error with respect to the expected output and then go backward into the network and update the weights using gradient descent. To compute (Tomas et al., 2013) the network weight for a single input to output network, the back propagation (BP) uses the loss function to compute the gradient. The following are the equations for gates and activation functions. LSTM Gates are the activation of sigmoid function, between 0 and 1 is output value of the sigmoid. When the gates are blocked the value is 0 and when the value is 1 then gates allow the input to pass through. (9)${\displaystyle \mathrm{sig}\left(t\right)=\frac{1}{1+e^{-t}}.}$

Here are the equations for all three gates.

Input Gate (10)${\displaystyle i_t=\sigma \left({\omega }_i\left[h_{t-1},x_t\right]+b_i\right).}$

Output Gate (11)${\displaystyle o_t=\sigma \left({\omega }_f\left[h_{t-1},x_t\right]+b_o\right).}$

Forget Gate (12)${\displaystyle f_t=\sigma \left({\omega }_o\left[h_{t-1},x_t\right]+b_f\right).}$

σ Represents sigmoid function, x_t Represents input at current timestamp. h_t−1 Represents LSTM block output of previous state at timestamp t −1. ω_i, ω_f, andω_o are represents weight for the input, forget and output gates. b_i, b_o and b_f are represents bias for input, output, forget gates. The following are equations for cell state for gates (Kumar & Chinnalagu, 2020).

Cell State of Input gate (13)${\displaystyle {\tilde{c}}_t=tanh\left(c\omega \left[h_{t-1},x_t\right]+b_c\right).}$

Cell Sate of Output gate (14)${\displaystyle c_t=f_t\ast c_{t-1}+i_t\ast {\tilde{c}}_t.}$

Cell Sate of Forget gate (15)${\displaystyle h_t=o_t\ast tanh\left(c^t\right).}$

${\tilde{c}}_t$ Represents input gate cell state at timestamp (t). c_t Represents memory cell state at timestamp (t). h_t Represents cell state of final output at timestamp (t).

This model (Kumar & Chinnalagu, 2020) consists of Input, Language detection and translation, embedded, Bi-Directional LSTM neutral network layer, dropout, dense and output layers. The input layers process the multilingual mixed customer reviews and speech transcript dataset and vectorizing the data using word embedding technique, each post has one or more sentences, and each sentence is composed with n number of words sequence. Here x representing input during the language detection, language translation process. (16)${\displaystyle x=x_{1,}{x}_{2,}{x}_{3,}\dots ..x_T.}$

In the detection layer, input text processed to detect the non-English text, here d represents input to detection layer. (17)${\displaystyle d=d_{1,}{d}_{2,}{d}_{3,}\dots ..d_T.}$

If the input text identified as non-English text, then the language translation layer converts the text to English, here t represents the input of translation layer. (18)${\displaystyle t=t_{1,}{t}_{2,}{t}_{3,}\dots ..t_T.}$

The output of the translation layer processed by embedding layer, each input word converted to vector, here S represents vector value. (19)${\displaystyle S=w_{1,}{w}_{2,}{w}_{3,}\dots ..w_n.}$

The Bi-Directional LSTM (BLSTM) is a sequence processing model, it consists of two LSTM units, (Gopalakrishnan & Salem, 2020) one unit taking the input in a forward direction and other unit taking the input in a backward direction. It effectively processes the input and context available to the network. Figure 11 shows the mixed-language data processing flow, the language detection and translation layers convert the non-English to English and then it’s embedding the words.

Input layer fed the embedded dataset to Bi-LSTM model and it processes vector output of the embedded layer. The following Fig. 12 shows the SA-BLSTM model architecture. This model used for both binary and multiclass classifications and SA applications.