# How to make an output independent of input feature in neural networks?

Is there a way to make a certain output dimension of a neural network independent of a particular feature dimension? For example, I have a function $$f_{\theta} : \mathcal{R}^{10} \rightarrow \mathcal{R}^2$$, I want to make $$f_{\theta}(\mathbf{x})_2$$ independent of $$\mathbf{x}_6$$. How can this condition be imposed on a neural network?

I am thinking of penalizing the gradient of $$f_{\theta}(\mathbf{x})$$ w.r.t $$\mathbf{x}_6$$ for a considerable range of $$\mathbf{x}_6 \in [-1, 1]$$. Will this give me the similar effect? If so, how can this be coded in Pytorch or any other deep learning framework?

    def call(self, x):
x_no_x6 = tf.concat([x[:,:5], x[:,5+1:]], axis=1)
f2 = ... model architecture goes here ... (function of x_no_x6)
f1 = ... model architecture... (function of x)
return tf.concat([f1, f2], axis=1)


You could have two models, one of which uses $$x$$ (including $$x_6$$) and the other which removes $$x_6$$ from the input. The overall model is just the concatenation of the two outputs. (Pseudo-code is assuming model is implemented by subclassing tensorflow.keras.Model link)

Not sure about the gradient penalization idea. Seems nontrivial to implement and it's not clear to me if it would work.

• Not sure if it is doing what I am looking for. Your output is still dependent on x_6 through f_1. In my question, I am limited to just one function $f$. In your solution, $f_1$ and $f_2$ are different functions with different parameters. Aug 31 at 16:02
• No. There is a single function $f \in \mathcal{R}^2$, and the second component is independent of $x_6$. I think you might be confused because you are not familiar with model subclassing in keras. When training/doing inference etc. you just have the single function (model.call) which goes from $x$ to R2. The parameters are model.trainable_variables, a single set of parameters.
– Taw
Aug 31 at 18:53
• Yeah. I wasn't familiar with subclassing as I mainly use PyTorch. However, I am limited to just one forward computation of $f$ because of time constraints. The suggested solution seems to be doing forward passes, is that correct? Sep 1 at 11:40

You could use Mutual Information between the model's prediction, and that particular feature as a regularization term. This will minimize the dependence of the output to that particular feature. Note that simply removing the feature from the dataset might not work if other features are associated with the feature which you don't want your model to depend on.

• That sounds reasonable. After searching through Mutual information loss, I didn't find easy to read references. Can you please point me to some of the references that explain/implement mutual information loss? Oct 5 at 14:56
• @pg2455 I'm afraid I don't have reference material regarding implementation. You might wanna try PyTorch forums in case you're using that framework. Oct 7 at 9:27