2
$\begingroup$

I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space dimension.

My VAE structure is as follows : Input : 2048 -> 1024 -> 512 -> Latent space dimension -> 512 -> 1024 -> Output : 2048

Here is what I get in terms of relative error when the latent space dimension goes from 2 to 100 : enter image description here

Everything works as expected at the beginning, but the error starts rising up at around 50 and I have no idea why. With a large latent space dimension, the output is orders of magnitude smaller than the input, which explains the relative error of value 1.

Here is the same figure when I run the exact same experiment but with a normal autoencoder this time.enter image description here

This time the results are consistent.

What's wrong with my VAE ?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ This is a really interesting phenomena, my guess would be the network is unable to learn a continuous representation in the number of epochs you provided for learning. Try upping the number of epochs in the VAE for learning and see if the error drops. $\endgroup$
    – Recessive
    Jan 12, 2021 at 4:54
  • 1
    $\begingroup$ @Recessive I already tried that. Went from 50 epochs to 100. Nothing changed, had the same exact behavior. $\endgroup$
    – tyassine
    Jan 12, 2021 at 7:20
  • $\begingroup$ The question of how the latent space size influences the performance is answered in this & this Cross Validated posts. In short: the amount of training data upper bounds the latent dimension, and a too large latent space leads to overfitting. This however seems opposite to what you report, maybe your model becomes too large and is thus underfitting? $\endgroup$
    – Mariusmarten
    Aug 18 at 18:32

1 Answer 1

Reset to default
0
$\begingroup$

Old question but hopefully OP is still active.

I also agree with recessive that perhaps it is struggling to create a continuous representation of your data in the latent space. Have you looked at your KL loss and your reconstruction loss independantly?

I was wondering: have you tried changing your input? If you reduce it from 2048 dimensions to 1024, does the dip in your curve also shift proportionally?

Have you also looked at what the latent space and the reconstructions looks like? what sort of errors is it making?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jul 19 at 16:10
  • $\begingroup$ It's been some time now so I don't know what I did of the code so I won't be able to test your suggestions. However, if I remember correctly, I solved the problem by using tanh activation functions instead of relu that I was using originally. I still have no idea what the origin of this behavior is or why changing the activation function fixes it tho. $\endgroup$
    – tyassine
    Jul 28 at 9:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .