2
$\begingroup$

Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).

Their network is built with three autoregressive layers, and six pooling and normalization layers.

In the paper they state,

Optimization: All parameters in our model were trained jointly with the objective being the sum of the objectives of the three layers.

Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?

I asked a follow-on question on the advantage of training all layer simultaneously.

$\endgroup$
0

1 Answer 1

2
$\begingroup$

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.

This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use uniform weights.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .