This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012.

What is the advantage of training all the layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt? And why would there ever be an advantage for an autoencoder like Le et al. 2012 where there is no backpropagation to communicate from the later layers to the earlier layers?

I think the conventional answer is that the lower layers can actually learn to provide low-level features that support the layers above. An example of this is learning to detect a horizontal yellow-blue feature to detect the water line in a beach scene.

But couldn't the yellow-blue feature be found just as easily by training the lower layers first? This would be especially true of an autoencoder such as Le et al. 2012, which picks up on patterns in the training set without having ground truth-labels to group them.

Citations to experiments or theoretical work that directly answers this question would be appreciated!

This is a follow-on to an earlier question.


1 Answer 1


I suspect there are two principles at play here, but I don't know which principle is more important in this case.

The first principle is the conventional wisdom: The lower layers not only learn to summarize the visual features found in the training set, but also learn those features which allow discrimination between categories. For example, in a beach image, there are likely many subtle yellow vs yellow or blue vs blue features that are technically part of the beach image. But where we sea the line in the sky may be a stronger indication of what this picture is of. All the features would be important for reconstructing an image of the beach, but the horizon line may be more important for distinguishing the beach from a picture of the sun in the sky for example. In a CNN like AlexNet, the horizon line would receive more weight early on in training.

But I don't see how this first principle would apply in the case of an autoencoding network like Le et al. 2012

The second principle is that random weights actually have some value from the very beginning of training, and those features which are useful early in training for a particular high-level decision will become even more useful later in training as they pick up patterns from the images to which they relate.

But I again don't see how this applies to an autoencoder like Le et al.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .