Which pre-processing steps are necessary for Deep Learning models to solve a document classification problem?

Question

I have created a data set with 30.000 text documents (each text file is rather small with respect to its length), which are labelled with 0 and 1. Using this data set, I want to train machine learning and deep learning models in order to be able to classify new text files.

On the one hand, I want to use classical machine learning models (such as logistic regression, random forest, SVM, etc.) with the Bag of Words/TF-IDF approach. This requires extensive text pre-processing, such as tokenization, stemming, converting to lower case, removing of stopwords and punctuation, lemmatization, etc.

On the other hand, I want to use new deep learning models (such as RNN, LSTM, BERT, XLNET, etc.).

Which pre-processing steps are necessary/advantageous for these deep learning models? Should I also use tokenization, stemming, converting to lower case, removing of stopwords and punctuation, lemmatization, etc. or can I omit most of these steps?

Answer 1

It depends on the type of model you use and the task, you are attempting to solve.

Almost all the preprocessing steps that you mention remove some information from the text. If you think the removed information is irrelevant (stopwords, suffixes, sentence boundaries), you can happily do it and use your favorite architecture. If you think, the linguistic information in the text is relevant (e.g., you need to care about negations), it is better not to remove any words and only tokenize the text (perhaps into subwords).

In general, lower-casing and stemming can help with data sparsity. However, if you work with a language for which a large pre-trained models exist (and you do not care much about efficiency), it is always better to use a pre-trained BERT-like model. The pre-trained models come with their own preprocessing and tokenization (e.g., WordPiece in case of BERT), and using a different pre-processing would just break them.