I have often heard that Deep Learning Models (i.e. Deep Neural Networks) are automatically performing feature engineering within the network themselves. This is contrasted with traditional statistical and Machine Learning models where the feature engineering is typically done prior to training the model:
Apparently, the operations that Neural Networks perform during the training phase are equivalent to "searching for meaningful combinations of variables that produce better performance results on the training data". I have heard that this is in some way similar to Principal Component Analysis (PCA) and Kernel Methods, seeing as these methods combine many different existing features into new features that have "more meaningful representation".
For instance, in the "Deep Learning Book" (https://www.deeplearningbook.org/contents/intro.html), I read about this ability of Deep Neural Networks to "automatically perform feature engineering" :
"Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts."
However, in the end - I still do not understand how Neural Networks during the training phase (i.e. gradient descent, weight update, backpropagation) are "automatically performing (some degree of) feature engineering".
How exactly are Neural Networks doing this "automatically"?
Note: I have heard that NN's are said to be universal function approximators (i.e. universal approximation theorem). Therefore, for a given data set - there should exist a specific neural network architecture (i.e. specific number/value of weights, layers and choice of activation functions) that will result in your target function being approximated sufficiently well - however, the combinatorial search space is massive and recovering that network can be very difficult for large data because of the time required to train the network using gradient descent. So "automatic feature engineering" is something indirect that might or might not happen because of the universal approximation theorem.