As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual effectiveness of doing such a step; in particular, if we are learning a neural network for a downstream task, it seems like modern state of the art (BERT, GPT-2) just take essentially raw input.

For instance, this ACL paper seems to show that the result of text preprocessing is mixed, to say the least.

So is text preprocessing really all that necessary for NLP? In particular, I want to contrast/compare against vision and tabular data, where I have empirically found that standardization usually actually does help. Feel free to share your personal experiences/what use cases where text preprocessing helps!


It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link , if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be spectacular.

In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". Also, you should remember that BERT or GPT-2 it's deep learning algorithms so they not need too much processing. Using preprocessing in machine learning is more that needed(prediction of sentiment for example).

Shortly, preprocessing is optional, but highly advisable.

  • $\begingroup$ Depends, I agree in some tasks it can help. But I don't think in general we should think about raw data as "garbage' necessarily. $\endgroup$ – information_interchange Apr 12 '20 at 1:00

It depends on the dataset we have and algorithm we use, usually text preprocessing can help your model perform better. But, some preprocessing method can have no significant impact on accuracy. We need to choose which preprocessing method that can help us make better quality dataset to give to the model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.