2
$\begingroup$

Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?

For example, we may want to learn how to reconstruct a relatively low-dimensional input (let's say $20$ dimensions), then could I define my encoder and decoder to have $64,256,512...$ hidden neurons before bringing back the reconstruction?

EDIT: Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.

Improve this question
$\endgroup$
3
  • $\begingroup$ I recommend that you put your specific question in the title. Here, your question is very specific and it's not just "Dimension of the latent space in VAE". Anyway, one thing is not clear to me. It's clear that you're suggesting to improve the dimensionality of the layers of the encoder, but what about the decoder? Let's say the encoder produces a latent vector of dimension 512, which is bigger than the input, then the decoder should do what? Should it do like the usual role of the encoder and convert this latent vector to a lower-dimensional reconstructed input? Is this what you mean? $\endgroup$
    – nbro
    Commented Mar 15, 2022 at 14:05
  • 1
    $\begingroup$ Apologizes, I've edited the question title with a more specific one. Anyway yes, shortly speaking we would invert the encoder/decoder role in such a situation. Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose. $\endgroup$
    – James Arten
    Commented Mar 15, 2022 at 15:14
  • $\begingroup$ I would recommend that you add this useful info in your comment to your post directly (because comments are temporary so they might get deleted later)! $\endgroup$
    – nbro
    Commented Mar 15, 2022 at 16:46

1 Answer 1

Reset to default
1
$\begingroup$

I am not sure about the VAE in particular, but the convnext presented here uses the "inverse bottleneck" (i.e. internal representations being higher dimensional than inputs) as one of the core changes that leads to an increase in its performance when compared to prior convolutional networks. This and some other changes make the convnet be more competitive against the visual transformer (that is pretrained in a large dataset).

But for really low-dimensional data (as in your example) and for VAEs, I am not sure if the same idea would prove useful.

P.S. I wanted to add this as a comment but I do not have enough reputation to do so.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .