# Controlling mutual information in latent variables

Recently, I read some interesting papers on mutual information (MI) estimation in high dimensional variables using neural networks [Belghazi et al., 2018][Poole et al., 2019]. These methods besides providing a way to estimate MI also enable the application of the infomax principle.

I am wondering if it'd be possible to control the relative amount of MI in different hidden units/latent variables within a layer.

Let's say we are trying to predict a quantity y from an input x.

$$z = f_{encoder}(x)$$ $$y = f_{model}(z)$$ where $$z = (z_1, z_2, ..., z_n)$$ is the latent feature vector.

Will it be possible to use some heuristics along with the above methods to control how much information about y is captured in different latent variables? i.e., Can any MI regularization techniques be used to achieve the following constraint? $$I(y;z_1,z_2) > I(y;z_3,z_4) > .... > I(y;z_{n-1},z_n)$$

References:

• Belghazi et al., Mutual Information Neural Estimation. In International Conference on Machine Learning,2018.
• Poole et al., On variational lower bounds of mutual information. In International Conference on Machine Learning, 2018.