0
$\begingroup$

It struck me that regular auto-encoders with batch-norm and dropout have quite similar properties to VAEs which made me wonder whether VAEs where really much better than this simpler alternative. Let me explain:

  • BatchNorm: encourages activations to follow N(0,1), just like KL-divergence does to the output layer of the encoder
  • Dropout: during training time creates random (Gaussian due to CLT) encoder output distribution, mean and variance are learnable indirectly via network weights and mean/variance override params of batch norm layer

You might argue that KL-divergence adds more flexibility by allowing it to not exactly follow N(0,1) but batch norm allows learning to override this by default anyways.

All that considered are there really any practical benefits to VAEs which cannot be provided by this simpler setup?

Improve this question
$\endgroup$
4
  • $\begingroup$ Have you seen any work that uses autoencoders with batch norm and dropout and claimed that this would be (somehow) equivalent to VAEs? If yes, it might be a good idea to provide that reference. I know that, Monte Carlo Dropout with neural networks can be viewed as Bayesian neural networks. This is the closest thing to what you're describing, but for regular neural networks, not autoencoders. $\endgroup$
    – nbro
    Mar 22, 2022 at 15:20
  • $\begingroup$ No I'm just reasoning from first principles here, I could be wrong. But if I am I'm curious why. $\endgroup$
    – profPlum
    Mar 22, 2022 at 16:06
  • $\begingroup$ @nbro I take that back I did just find a reference trying to do something similar to what I mentioned: vitalab.github.io/article/2021/08/20/… They say they can use deterministic auto-encoders + injected noise for regularization as an alternative to VAEs. That is precisely what dropout is (though they don't use dropout explicitly). $\endgroup$
    – profPlum
    Mar 22, 2022 at 16:21
  • $\begingroup$ De-noising auto-encoders also are essentially auto-encoders with dropout. But I don't know if anyone has directly compared them to VAEs. Still I sense they are similar... proceedings.neurips.cc/paper/2013/file/… $\endgroup$
    – profPlum
    Mar 22, 2022 at 16:41

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .