0
$\begingroup$

To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?

Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.

So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?

$\endgroup$
1
  • $\begingroup$ What do you mean by "tying weights" or "tie the weights"? $\endgroup$ – nbro Feb 24 at 10:32
1
$\begingroup$

We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.

If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.