0
$\begingroup$

Is there any result on the distribution of autoencoder embeddings?

For example, the following image (taken from this article) visualizes the latent space with t-SNE. As you can see, images from the same class (in this case MNIST digits) roughly form a cluster. Is there any theory explaining why that is the case?

enter image description here

Note that I am talking about autoencoders, not variational autoencoders, whose embedding by definition has a multi-variate Gaussian distribution.

$\endgroup$

1 Answer 1

1
$\begingroup$

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize the change in distribution under the application of an autoencoder.

Let $g:\mathcal{X}\rightarrow\mathcal{Z}$ be the encoder, and $f:\mathcal{Z}\rightarrow\mathcal{X}$ be the decoder. $\mathcal{X}$ corresponds to the input space, whereas $\mathcal{Z}$ corresponds to the latent space. Let $P$ be the true distribution of your data, from which you have an i.i.d. sample $S = \{\mathbf{x}_{i}\}_{i=1}^{n}$. You can approximate $P$ using your sample,

$$\hat{P}(\mathbf{x}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{x}-\mathbf{x}_{i})$$

this is called an empirical approximation of $P$. Now, if you want to study the distribution of $\mathbf{z}_{i} = g(\mathbf{x}_{i})$, you can write a new distribution,

$$\hat{Q}(\mathbf{z}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{z}-\mathbf{z}_{i})$$

which is an approximation of $Q = g_{\sharp}P$, where $g_{\sharp}$ is the pushforward of $P$ by $g$. Note that this does not tell us much about the properties of $Q$, since we did not make enough assumptions. In some frameworks (e.g. Optimal Transport), this formulation is very useful. If you want to read more on an probabilistic/transport analysis of the distributions in an autoencoder you can take a look on [1].

References

[1] Sonoda, S., & Murata, N. (2019). Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1), 31-82.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .