Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).
Their network is built with three autoregressive layers, and six pooling and normalization layers.
In the paper they state,
Optimization: All parameters in our model were trained jointly with the objective being the sum of the objectives of the three layers.
Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?
I asked a follow-on question on the advantage of training all layer simultaneously.