I have been given a data set with 30.000 text documents (each text file is rather small with respect to its length and consists in most cases of around 20 sentences), which are labelled with 0 or 1. Using this data set, I want to train machine learning and deep learning models in order to be able to classify new text files.
On the one hand, I want to use classical machine learning models (such as logistic regression, random forest, SVM, etc.) with the Bag of Words (BoW)/TF-IDF approach. In this context, the text data are represented by a matrix with 30.000 rows (and a number of columns that correspond to the unique words in the overall data set), where each row stands for one observation (i.e., a text document) and each column stands for a unique word. The entries of this matrix are then (kind of) frequencies of the unique words in a text document. However, these approaches do not take the sequence of words, negations, etc. into account.
On the other hand, I want to use new deep learning models (such as RNN, LSTM, BERT, XLNET, etc.), which take the sequence of words, etc. in a text document into account. Obviously, the data set of text files cannot be represented with the BoW or TF-IDF approach in this case as this would neglect the order of words, etc. Which data representation technique can be used to input a data set with labelled text files into a deep learning model (such as RNN, LSTM, BERT, XLNET, etc.)? Is there something similar like the BoW or TF-IDF approach that also pays attention to the sequence of words, etc.?