Claim: Deep Neural Networks "Automatically" Perform Feature Selection and Feature Engineering?

Question

I have often heard that Deep Learning Models (i.e. Deep Neural Networks) are automatically performing feature engineering within the network themselves. This is contrasted with traditional statistical and Machine Learning models where the feature engineering is typically done prior to training the model:

Apparently, the operations that Neural Networks perform during the training phase are equivalent to "searching for meaningful combinations of variables that produce better performance results on the training data". I have heard that this is in some way similar to Principal Component Analysis (PCA) and Kernel Methods, seeing as these methods combine many different existing features into new features that have "more meaningful representation".

For instance, in the "Deep Learning Book" (https://www.deeplearningbook.org/contents/intro.html), I read about this ability of Deep Neural Networks to "automatically perform feature engineering" :

"Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts."

However, in the end - I still do not understand how Neural Networks during the training phase (i.e. gradient descent, weight update, backpropagation) are "automatically performing (some degree of) feature engineering".

How exactly are Neural Networks doing this "automatically"?

Thanks!

Note: I have heard that NN's are said to be universal function approximators (i.e. universal approximation theorem). Therefore, for a given data set - there should exist a specific neural network architecture (i.e. specific number/value of weights, layers and choice of activation functions) that will result in your target function being approximated sufficiently well - however, the combinatorial search space is massive and recovering that network can be very difficult for large data because of the time required to train the network using gradient descent. So "automatic feature engineering" is something indirect that might or might not happen because of the universal approximation theorem.

Answer 1

Probably the best and most suggestive example of automatic feature extraction comes from Restricted Boltzmann Machines (RBMs). Because of their architecture, composed in its simplest form of one visible layer and one hidden layer of same size, they allow us to visualise the weights learnt during training really easily, i.e. by simply reshaping them in the same size as the input images fed during training.

Below you can see an example of such weight visualisation, learned on the classic MNIST (hand written digits) dataset. An important thing to understand is that dark and bright pixels are associated with respectively small and big weights. Again, I want to stress out that every image is a set of weights that connect each pixel of an input image to a single hidden node (as shown in the second picture).

So each image effectively is showing us what the RBM consider important in an image. When we feed an image to the RBM, the first feature map is applied, and we can see that this feature map recall something like a 9 (in black) and a 2 (in white), so the RBM will respond with high activations to images similar to a 2 and very low activations for images similar to a 9.

How did the RBM decided that these features are important? The answer lies in two main factors:

• the training algorithm
• the data distribution

In this case the training algorithm is contrastive divergence, an unsupervised energy based algorithm that minimise the energy of the weights when similar training inputs are presented while trying to maximise their energy when different ones are presented. This algorithmic design explain why we see feature maps like the first one. Contrastive divergence is trying to minimise the energy for all inputs images containing 2 while trying also to keep 2 and 9 separated from each other (in terms of nodes activations). Clearly 2 and 9 have similarities like the top curve, so the model need to learn features maps that activate a lot nodes connected to areas belonging to features specific to only one class, like the bottom straight line of 2. Unfortunately, this is almost as far as we can go into the analysis of "why a model is learning something", without falling into models interpretability discussions (for which there are already good threads in this community), and surely we'll never know "how exactly", or if that day will come, then we won't need machine learning anymore.

Last but not least, a well known bias of the MNIST dataset is the centred position of each digit. Nowhere in the data you will find lines on the edges of an image, and indeed the RBM also never learn to pay attention to the edges. So, regardless of the choice of the training algorithm, the training data distribution will also always impact what a model learn (or in this case not learn).