Recently, I read some interesting papers on mutual information (MI) estimation in high dimensional variables using neural networks [Belghazi et al., 2018][Poole et al., 2019]. These methods besides providing a way to estimate MI also enable the application of the infomax principle.

I am wondering if it'd be possible to control the relative amount of MI in different hidden units/latent variables within a layer.

Let's say we are trying to predict a quantity y from an input x.

$$z = f_{encoder}(x)$$ $$y = f_{model}(z)$$ where $z = (z_1, z_2, ..., z_n)$ is the latent feature vector.

Will it be possible to use some heuristics along with the above methods to control how much information about y is captured in different latent variables? i.e., Can any MI regularization techniques be used to achieve the following constraint? $$I(y;z_1,z_2) > I(y;z_3,z_4) > .... > I(y;z_{n-1},z_n)$$


  • Belghazi et al., Mutual Information Neural Estimation. In International Conference on Machine Learning,2018.
  • Poole et al., On variational lower bounds of mutual information. In International Conference on Machine Learning, 2018.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.