Is there a way to make a certain output dimension of a neural network independent of a particular feature dimension? For example, I have a function $f_{\theta} : \mathcal{R}^{10} \rightarrow \mathcal{R}^2$, I want to make $f_{\theta}(\mathbf{x})_2$ independent of $\mathbf{x}_6$. How can this condition be imposed on a neural network?

I am thinking of penalizing the gradient of $f_{\theta}(\mathbf{x})$ w.r.t $\mathbf{x}_6$ for a considerable range of $\mathbf{x}_6 \in [-1, 1]$. Will this give me the similar effect? If so, how can this be coded in Pytorch or any other deep learning framework?

    def call(self, x):
        x_no_x6 = tf.concat([x[:,:5], x[:,5+1:]], axis=1)
        f2 = ... model architecture goes here ... (function of x_no_x6)
        f1 = ... model architecture... (function of x)
        return tf.concat([f1, f2], axis=1)

You could have two models, one of which uses $x$ (including $x_6$) and the other which removes $x_6$ from the input. The overall model is just the concatenation of the two outputs. (Pseudo-code is assuming model is implemented by subclassing tensorflow.keras.Model link)

Not sure about the gradient penalization idea. Seems nontrivial to implement and it's not clear to me if it would work.

  • $\begingroup$ Not sure if it is doing what I am looking for. Your output is still dependent on x_6 through f_1. In my question, I am limited to just one function $f$. In your solution, $f_1$ and $f_2$ are different functions with different parameters. $\endgroup$
    – pg2455
    Aug 31 at 16:02
  • $\begingroup$ No. There is a single function $f \in \mathcal{R}^2$, and the second component is independent of $x_6$. I think you might be confused because you are not familiar with model subclassing in keras. When training/doing inference etc. you just have the single function (model.call) which goes from $x$ to R2. The parameters are model.trainable_variables, a single set of parameters. $\endgroup$
    – Taw
    Aug 31 at 18:53
  • $\begingroup$ Yeah. I wasn't familiar with subclassing as I mainly use PyTorch. However, I am limited to just one forward computation of $f$ because of time constraints. The suggested solution seems to be doing forward passes, is that correct? $\endgroup$
    – pg2455
    Sep 1 at 11:40

You could use Mutual Information between the model's prediction, and that particular feature as a regularization term. This will minimize the dependence of the output to that particular feature. Note that simply removing the feature from the dataset might not work if other features are associated with the feature which you don't want your model to depend on.

  • $\begingroup$ That sounds reasonable. After searching through Mutual information loss, I didn't find easy to read references. Can you please point me to some of the references that explain/implement mutual information loss? $\endgroup$
    – pg2455
    Oct 5 at 14:56
  • $\begingroup$ @pg2455 I'm afraid I don't have reference material regarding implementation. You might wanna try PyTorch forums in case you're using that framework. $\endgroup$
    – SpiderRico
    Oct 7 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.