0
$\begingroup$

I am reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems and came across the following paragraph -

You can recognize the basic structure of all autoencoders, with an encoder followed by a decoder (in this example, they both have two hidden layers), but there is a twist: instead of directly producing a coding for a given input, the encoder produces a mean coding μ and a standard deviation σ. The actual coding is then sampled randomly from a Gaussian distribution with mean μ and standard deviation σ. After that the decoder decodes the sampled coding normally.

I do not understand how sampling is conducted over here and have the following question -

I understand it is trivial to sample from a univariate Gaussian distribution. However, for a dataset with n features, we won't be able to use a univariate Gaussian. How does that work?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

The number of features $n$ of an input $\mathbf{x} \in \mathbb{R}^n$ to the VAE may be related to the dimension $m$ of the latent vector $\mathbf{z} \in \mathbb{R}^m \sim \mathcal{N}(\mu, \sigma^2)$, but the formulation of the VAE does not go far enough to tell us how these parameters are related. The only assumption in the VAE is that $\mathbf{x}$ depends on $\mathbf{z}$. Here's the diagram (from the VAE paper) that illustrates the graphical model they consider

enter image description here

In practice, we may not use a univariate Gaussian, but most likely we use a multivariate Gaussian, i.e. $m > 1$, but $m$ is really a hyper-parameter, so it could also be $1$. Moreover, note that we don't sample $\mathbf{z}$ directly from $\mathcal{N}$, if we use the re-parametrization trick. See this implementation, where $\epsilon$ is actually sampled. $\epsilon$ is just a sample from a standard normal (usually), which we use (along with the learnable mean and variance) to construct $\mathbf{z}$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .