This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012.
What is the advantage of training all the layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt? And why would there ever be an advantage for an autoencoder like Le et al. 2012 where there is no backpropagation to communicate from the later layers to the earlier layers?
I think the conventional answer is that the lower layers can actually learn to provide low-level features that support the layers above. An example of this is learning to detect a horizontal yellow-blue feature to detect the water line in a beach scene.
But couldn't the yellow-blue feature be found just as easily by training the lower layers first? This would be especially true of an autoencoder such as Le et al. 2012, which picks up on patterns in the training set without having ground truth-labels to group them.
Citations to experiments or theoretical work that directly answers this question would be appreciated!
This is a follow-on to an earlier question.