0
$\begingroup$

I really wonder about data preprocessing is really necessary and important in deep learning.

It's really hard to say clearly about difference between Machine learning and deep learning. In definetion, The difference is whether or not humans intervene in the process of finding features. Machine learning need humans intervene to do feature extraction, so must need data preprocessing. So we do PCA, denoising, data balancing, and so on.

Machine learning is important to pre-processing. But, In deep learning, Is really important to do preprocessing at data? I'm not asking about such as embedding in NLP. text embedding is necessary for transform text to computer recognizable.

I'm really confused while studying about deep-learning. Someone says that it's important to do pre-processing, and someone says that isn't important to do pre-processing. For example, I'll show some situation about computer vision. Let's think about Denoising, erase such as pepper and salt noise in image. Is it important to do denoise for input image? someone who agree about to do preprocessing says that noise is unexpected pixel value, so if we denoised noise, than we can get more clear image, so It brings more good performance. someone who disagree about to do preprocessing says 2 reasons.

First, If we can get 100% clear original image without any noise, then compair denoised image and clear original image. It's not totally same image. Also, we can't get without any noise image because of physical reasons.

Second, Deep learning shows better performance when the amount of input data is large. If human intervene to feature extraction, It's not efficient to do with human power. Deep learning can do feature extraction itself, don't need human to feature extraction.

I apolized that I think it's not good example about my question. so, I give another example. Think about Iris dataset. In Machine learning, We do PCA(or else) and select efficient data(ex: sepal and petal width and length data) and input data to model. But, deep learning don't need select efficient data. If we have nice deep learning model, then we can input whole data.

last example, It is estimated that GPT-3 has been learned with more than 300 billion sentences. I don't think training dataset did pre-processing. Because, sentence might have typo, wrong grammer, slang, abbreviation, etc because human writing. If do preprocessing about typo, wrong grammer, etc then It would have consumed a lot of human labor and costs.

So... Is deep learning really need pre-processing?

Improve this question
$\endgroup$
2
  • $\begingroup$ About your salt & pepper noise, actually in most situation (like classification) it might be good to add it/keep it since it allows the model to be more robust. It's a particular case of data augmentation... Most of the time, preprocessing means making the optimization process easier by e.g scaling the numerics. Do not confuse it with data augmentation (e.g adding rotation, translation, chaning brightness, contrast, adding noise etc). $\endgroup$
    – Lelouch
    Jul 13 at 15:11
  • $\begingroup$ @Lelouch I agree salt and pepper example is not a good example. But, examples of Iris dataset and gpt isn't correct in preprocessing part? $\endgroup$
    – Yang
    Jul 14 at 5:59

1 Answer 1

Reset to default
1
$\begingroup$

Building off of @Lelouch's comment -- adding augmentations like noise and performing preprocessing can help make a model robust, especially if your dataset is limited. However, if your dataset is huge to begin with (e.g., containing 300 billion sentences), then that's not really necessary as there's enough edge cases in your training data.

For your comparison to PCA/feature selection, deep models do that automatically to some extent. Early layers can be thought of as "feature-extractors" for later layers. This means that your model can be more prone to overfitting, but that can be fixed with data augmentation (or just by adding more data).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for answering. I know that deep learnings are doing feature-extractors(In transformer, encoders do that). So I think that deep learning isn't need pre-processing. But, data-miners(also do deep learning) ask me why I don't using preprocessing dealing with data. I can't answering about that. Data-miners who I met belives that preprocessing bring more better result. If It is true, but why many researchers are not included preprocessing tasks on their paper? $\endgroup$
    – Yang
    Jul 17 at 16:06
  • $\begingroup$ Of course, many researchers doing preprocessing tasks and included preprocessing tasks on their paper. As far as the papers I read, there were more that did not have a preprocessing process. So I really want to know with reference if data preprocessing really don't need preprocessing(And vice versa). I couldn't find, and searching continuously about reference that deep learning really needed preprocessing. $\endgroup$
    – Yang
    Jul 17 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .