Which data representation of text as input for NLP Deep Learning models?

Question

I have been given a data set with 30.000 text documents (each text file is rather small with respect to its length and consists in most cases of around 20 sentences), which are labelled with 0 or 1. Using this data set, I want to train machine learning and deep learning models in order to be able to classify new text files.

On the one hand, I want to use classical machine learning models (such as logistic regression, random forest, SVM, etc.) with the Bag of Words (BoW)/TF-IDF approach. In this context, the text data are represented by a matrix with 30.000 rows (and a number of columns that correspond to the unique words in the overall data set), where each row stands for one observation (i.e., a text document) and each column stands for a unique word. The entries of this matrix are then (kind of) frequencies of the unique words in a text document. However, these approaches do not take the sequence of words, negations, etc. into account.

On the other hand, I want to use new deep learning models (such as RNN, LSTM, BERT, XLNET, etc.), which take the sequence of words, etc. in a text document into account. Obviously, the data set of text files cannot be represented with the BoW or TF-IDF approach in this case as this would neglect the order of words, etc. Which data representation technique can be used to input a data set with labelled text files into a deep learning model (such as RNN, LSTM, BERT, XLNET, etc.)? Is there something similar like the BoW or TF-IDF approach that also pays attention to the sequence of words, etc.?

Answer 1

For deep learning models, embedding vectors have become the standard way of encoding text features almost immediately after their introduction.

The reason for this is that neural networks work with data encoded with continuous values ranging from 0 to 1 (or sometimes from -1 to 1). Bag of Words and TF-IDF can be modified to produce values in this range, but it wouldn't make much sense conceptually, especially for Bag of Words techniques where each number is simply a code and it doesn't represent any inner property of the encoded text. Embedding vectors instead are designed to be dense representation, i.e. they possess all properties of a vector space, allowing us to perform math with them, e.g. computing the "distance" between two words, so different numbers in this case do represent different properties for each word.

Embedding vectors also contain information about sequences. Mathematically they are derived from factorization of frequency (or co-occurrence) matrices, similar to TF-IDF, which are then compressed (with various dimensionality reduction techniques) to a set of vectors of fixed size. Of course the initial frequency matrix can be designed to encode a single word-pair probability per element, but nothing prevent us to make frequency matrices containing co-occurrence frequencies of triples or quadruplets (or more) of words. Some embedding are also trained using neural networks, most famous example being Word2Vec, more specifically the CBOW (Continuous Bag of Words) variation, a simple multi layer perceptron trained to project a word to a dense vector given a sequence of one hot encoded context words (see image below).

Given you rather large set of documents, you could train your own domain specific CBOW embeddings using a context widow of your choice. There are though many available pretrained embeddings that can be used directly, trained on massive datasets (billions of documents) like Wikipedia, Amazon reviews or Twitter. These embeddings usually outperform embeddings trained on small datasets due to the huge amount of common sense relationships encoded in them. I would therefore suggest you to try these sources before digging into training your own vectors:

• Glove: Global Vectors for Words Represetation
• FastText
• Google News vectors