In the famous Information bottleneck paper by Tishby(https://arxiv.org/abs/1703.00810), the author proposed a framework that the neural network can compress information. And they computed the mutual information between different layers.

Now I have a question. We all know that the neural network for each layer is a deterministic function, which means given a specific X. We can get a new Z=f(X). So, we can get H(Z|Y) is 0. Because given any X, we can compute corresponding Z from layer function.

But actually, in this article, H(Z|Y) not equals 0, and we can compute H(Z|Y), or I(X, Z). Seems that we omit the deterministic relationship. Why?

  • $\begingroup$ Hello. Please, edit your post to format your math symbols and equations with latex/mathjax. If you don't know how to use them, please, take a look at this guide: math.meta.stackexchange.com/questions/5020/…. $\endgroup$
    – nbro
    Jul 15 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.