# How to quantify the amount of information lost by the decoder NN in an AE?

Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom?

I was thinking maybe to use somehow the mutual information either in the image or frequency domain.

$$\mathrm{I}(X ; Y)=\sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}} p_{(X, Y)}(x, y) \log \left(\frac{p_{(X, Y)}(x, y)}{p_{X}(x) p_{Y}(y)}\right)$$

where $$p_{(X,Y)}$$ is the joint probability density function which is deduced somehow empirically from a set of $$N$$ input and output to the network.

Maybe it's not even an interesting question since the loss function evaluates exactly that?

• Johnson Lindenstrauss lemma and PCA might be of help. – user9947 Sep 24 '20 at 23:50