I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.
Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.
However, reading HERE, I read:
Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.
Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.
My question is then: if I'm building a network already using "modern" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?