I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.

Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.

However, reading HERE, I read:

Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.


Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.

My question is then: if I'm building a network already using "modern" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?


1 Answer 1


It depends. It could give you a boost or it could not.

Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).

Ideal World: We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder

Reality years ago: Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).

  • solution: Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step
    • Cons: Each step is looking for a greedy solution, rather than a deep one.

Reality today: Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)

  • solution: We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.

I hope this helped give some form of intuition of the matter.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .