Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom?

I was thinking maybe to use somehow the mutual information either in the image or frequency domain.

$$ \mathrm{I}(X ; Y)=\sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}} p_{(X, Y)}(x, y) \log \left(\frac{p_{(X, Y)}(x, y)}{p_{X}(x) p_{Y}(y)}\right) $$

where $p_{(X,Y)}$ is the joint probability density function which is deduced somehow empirically from a set of $N$ input and output to the network.

Maybe it's not even an interesting question since the loss function evaluates exactly that?

enter image description here

  • 1
    $\begingroup$ Johnson Lindenstrauss lemma and PCA might be of help. $\endgroup$
    – user9947
    Sep 24 '20 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.