2
$\begingroup$

I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.

Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.

However, reading HERE, I read:

Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.

and

Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.

My question is then: if I'm building a network already using "modern" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?

$\endgroup$
2
$\begingroup$

It depends. It could give you a boost or it could not.

Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).

Ideal World: We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder

Reality years ago: Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).

  • solution: Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step
    • Cons: Each step is looking for a greedy solution, rather than a deep one.

Reality today: Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)

  • solution: We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.

I hope this helped give some form of intuition of the matter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.