Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).
Their network is built with three autoregressive layers, and six pooling and normalization layers.
In the paper they state,
Optimization: All parameters in our model were trained jointly with the objective being the sum of the objectives of the three layers.
Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?
I think by asking this question, I have answered it for myself. Please see my answer below.
What is the advantage of training all three layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt?
I do not have a very good answer to the follow-on question. Perhaps the lower layers can actually learn to provide low-level features that support the layers above?
This training of all layers simultaneously is done in all deep neural networks today -- AlexNet, VGG, etc. -- but why?
Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng, Building high-level features using large scale unsupervised learning (2012) arXiv:1112.6209 [cs.LG]