2
$\begingroup$

I'd like to ask you if feature engineering is an important step for a deep learning approach.

By feature engineering I mean some advanced preprocessing steps, such as looking at histogram distributions and try to make it look like a normal distribution or, in the case of time series, make it stationary first (not filling missing values or normalizing the data).

I feel like with enough regularization, the deep learning models don't need feature engineering compared to some machine learning models (SVMs, random forests, etc.), but I'm not sure.

Improve this question
$\endgroup$
1
  • $\begingroup$ Here is an older related (if not duplicate) question. $\endgroup$
    – nbro
    Mar 19, 2021 at 11:37

2 Answers 2

Reset to default
0
$\begingroup$

No, feature engineering is not an important step for deep learning (EDIT: compared to other techniques) provided that you have enough data. If your dataset is big enough (which varies from task to task), you can perform what is called an end-to-end learning.

To further clarify, according to this article, deep neural nets trained with backpropagation algorithm are basically doing an automated feature engineering.

I feel like with enough regularization, the deep learning models don't need feature engineering compared to some machine learning models (SVMs, random forests, etc.)

That is basically correct. Beware, you need a large dataset. When a large dataset is not available, you will do some manual work (feature engineering).

Nevertheless, it is always a good idea to look at your data first!

EDIT

I would also like to quote Rich Sutton here:

We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.

Perhaps this statement is more true with Deep Learning than with previous techniques, but we are not quite there yet. And as user nbro rightfully pointed out in the comments below, you may still need to normalise your data, pre-process it, remove outliers, etc. Thus in practice, you may still need to transform your data to a certain degree, depending on many factors.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for the two articles and for your answer. $\endgroup$
    – Daviiid
    Mar 13, 2021 at 0:50
  • $\begingroup$ End-to-end learning is somehow orthogonal to what the OP means by "feature engineering". If e.g. your neural network trained "end-to-end" (which simply means that you have only one net, so you don't have intermediate steps, or training procedures, etc.) is not learning anything, you may need to analyse your data, look at the histograms, find anomalies, no matter how much data you have. So, I think that this answer is misleading, and saying that "feature engineering" is not important is actually wrong. The more interesting question would be: when is feature engineering required? $\endgroup$
    – nbro
    Mar 18, 2021 at 13:50
  • 1
    $\begingroup$ @nbro I think we don't have a common language here. "Feature engineering is the process of using domain knowledge to extract features from raw data" (Wikipedia). End-to-end in contrast will not require using domain knowledge. For example, in computer vision people would traditionally do Gabor filters (domain knowledge). Today, people tend to avoid that as the filters are learned automatically by a convnet. Moreover, the filters learned automatically tend to be more efficient compared to handcrafted ones. Please also check this out incompleteideas.net/IncIdeas/BitterLesson.html $\endgroup$
    – penkovsky
    Mar 19, 2021 at 10:29
  • 1
    $\begingroup$ @penkovsky Even if you use a ConvNet, you may still need to normalise your data, pre-process it, remove outliers, etc. So, yeah, ConvNets learn features, but that doesn't mean that you cannot have a feature engineering step where you actually improve your (raw) data so that learning can happen faster. $\endgroup$
    – nbro
    Mar 19, 2021 at 10:32
  • $\begingroup$ "> you may need to analyse your data, look at the histograms, find anomalies, no matter how much data you have. <" I also specifically wrote "Nevertheless, it is always a good idea to look at your data first!" However, I don't really see any reason to elaborate on that point since data analysis and visualisation have been discussed exhaustively. $\endgroup$
    – penkovsky
    Mar 19, 2021 at 10:34
1
$\begingroup$

From what I believe, feature engineering is important, it's a part of the job of ML network designer.

Network designing involves

  • Feature engineering: What should be in the input to the network, as processed from similar or totally different data
  • Deciding network shape, layer shapes, types of neurons in layers, etc.
  • Feature engineering again (but labels), in the output, what should the output be, either regression values or classes

And possibly also tasks rather simple as mentioned in the question: filling missing values, normalising data, create pre-feeding normalisation steps in code, etc.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank your for your answer and sorry for my late reply. I see. I think I didn't provide enough details in my question, but for example with images, I think it's better to feed them directly to the neural network, provided they work well with this type of data like CNNs, than to feed some other features like HOG descriptors. $\endgroup$
    – Daviiid
    Mar 13, 2021 at 0:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .