0
$\begingroup$

Recently, I read some interesting papers on mutual information (MI) estimation in high dimensional variables using neural networks [Belghazi et al., 2018][Poole et al., 2019]. These methods besides providing a way to estimate MI also enable the application of the infomax principle.

I am wondering if it'd be possible to control the relative amount of MI in different hidden units/latent variables within a layer.

Let's say we are trying to predict a quantity y from an input x.

$$z = f_{encoder}(x)$$ $$y = f_{model}(z)$$ where $z = (z_1, z_2, ..., z_n)$ is the latent feature vector.

Will it be possible to use some heuristics along with the above methods to control how much information about y is captured in different latent variables? i.e., Can any MI regularization techniques be used to achieve the following constraint? $$I(y;z_1,z_2) > I(y;z_3,z_4) > .... > I(y;z_{n-1},z_n)$$

References:

  • Belghazi et al., Mutual Information Neural Estimation. In International Conference on Machine Learning,2018.
  • Poole et al., On variational lower bounds of mutual information. In International Conference on Machine Learning, 2018.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.