# What is the distribution of autoencoder embeddings?

Is there any result on the distribution of autoencoder embeddings?

For example, the following image (taken from this article) visualizes the latent space with t-SNE. As you can see, images from the same class (in this case MNIST digits) roughly form a cluster. Is there any theory explaining why that is the case?

Note that I am talking about autoencoders, not variational autoencoders, whose embedding by definition has a multi-variate Gaussian distribution.

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize the change in distribution under the application of an autoencoder.

Let $$g:\mathcal{X}\rightarrow\mathcal{Z}$$ be the encoder, and $$f:\mathcal{Z}\rightarrow\mathcal{X}$$ be the decoder. $$\mathcal{X}$$ corresponds to the input space, whereas $$\mathcal{Z}$$ corresponds to the latent space. Let $$P$$ be the true distribution of your data, from which you have an i.i.d. sample $$S = \{\mathbf{x}_{i}\}_{i=1}^{n}$$. You can approximate $$P$$ using your sample,

$$\hat{P}(\mathbf{x}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{x}-\mathbf{x}_{i})$$

this is called an empirical approximation of $$P$$. Now, if you want to study the distribution of $$\mathbf{z}_{i} = g(\mathbf{x}_{i})$$, you can write a new distribution,

$$\hat{Q}(\mathbf{z}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{z}-\mathbf{z}_{i})$$

which is an approximation of $$Q = g_{\sharp}P$$, where $$g_{\sharp}$$ is the pushforward of $$P$$ by $$g$$. Note that this does not tell us much about the properties of $$Q$$, since we did not make enough assumptions. In some frameworks (e.g. Optimal Transport), this formulation is very useful. If you want to read more on an probabilistic/transport analysis of the distributions in an autoencoder you can take a look on [1].

## References

[1] Sonoda, S., & Murata, N. (2019). Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1), 31-82.