The order in which features are learned by DNNs during training is typically random & depends on a number of factors:
- Including the specific architecture of the DNN
- The type of data being used for training
- & the optimization algorithm being used
However, it is known that features are learned in a hierarchical manner, with lower-level(simpler) features being learned first, followed by higher-level(complex) features.
There is some evidence that certain features are learned early on in the training process, while others are learned later. For example, a study by Karpathy et al. (2014) found that a convolutional neural network (CNN) trained on the MNIST dataset tends to learn to distinguish between the digits 0 and 1 before it learns to distinguish between the digits 0 and 8. However, it is not clear how generalizable these findings are to other tasks and datasets.
A good general resource on how different features are learned at different stages in neural networks is the
Deep Learning book by Goodfellow, Bengio, and Courville. In particular, Chapter-6 (Feature Representations) and Chapter-7 (Neural Networks) cover this topic in detail.
Artificial Intelligence: A Modern Approach by Russell and Norvig (2010) covers this topic in Chapter 18 on Learning Complexity. The chapter starts with a discussion of the concept of concept learning and the ways in which different concepts can be learned. It then covers the question of how well different concepts can be learned, and how this affects the overall learning process.
One paper that explores this question is "Visualizing and Understanding Convolutional Networks" by Matthew D. Zeiler and Rob Fergus.
In this paper, the authors visualize the features learned by a convolutional neural network as it is trained on different datasets. They find that the network gradually learns increasingly complex features, starting with simple edge detectors and then moving on to more complex shapes and patterns.
Another paper Why does deep and cheap learning work so well? by Henry W. Lin, Max Tegmark, and David Rolnick
In this paper, the author tries to answer the question of why deep learning works so well, despite being "cheap" (in terms of the amount of data and computation required). He argues that deep learning is able to learn a wide range of features, from low-level features (e.g. edge detectors) to high-level features (e.g. object detectors), and that this enables it to achieve better performance than shallower methods.