Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).

Their network is built with three autoregressive layers, and six pooling and normalization layers.

In the paper they state,

Optimization: All parameters in our model were trained jointly with the objective being the sum of the objectives of the three layers.

Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?

I think by asking this question, I have answered it for myself. Please see my answer below.

Follow-on Question

What is the advantage of training all three layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt?

I do not have a very good answer to the follow-on question. Perhaps the lower layers can actually learn to provide low-level features that support the layers above?

This training of all layers simultaneously is done in all deep neural networks today -- AlexNet, VGG, etc. -- but why?


Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng, Building high-level features using large scale unsupervised learning (2012) arXiv:1112.6209 [cs.LG]

  • $\begingroup$ Welcome to ai.se...i think you should add your answer to the question itself or in a comment as community guidelines dictate you should not have a question or speculation in the answer...keep visiting :) $\endgroup$
    – user9947
    May 7 '18 at 17:04
  • 1
    $\begingroup$ Done for speculation. Solid answers of my own in the answer section are acceptable, right? $\endgroup$ May 7 '18 at 17:05
  • 1
    $\begingroup$ Yes..but no follow up questions in the answer itself.. $\endgroup$
    – user9947
    May 7 '18 at 17:06
  • $\begingroup$ This looks like a good answer to my follow-on question. $\endgroup$ Jul 10 '18 at 17:58

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.

This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use uniform weights.

I do not have a good answer for the follow-on question yet... would love to hear what others think!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.