# Compute Jacobian matrix of Deep learning model?

I am trying to implement this paper. In this paper, the author uses the forward derivative to compute the Jacobian matrix dF/dx using chain rule where F is the probability got from the last layer and X is input image. My model is given below. Kindly let me know how to go about doing that?

class LeNet5(nn.Module):

def __init__(self):

self.derivative= None # store derivative

super(LeNet5, self).__init__()
self.conv1= nn.Conv2d(1,6,5)
self.relu1= nn.ReLU()
self.maxpool1= nn.MaxPool2d(2,2)

self.conv2= nn.Conv2d(6,16,5)
self.relu2= nn.ReLU()
self.maxpool2= nn.MaxPool2d(2,2)

self.conv3= nn.Conv2d(16,120,5)
self.relu3= nn.ReLU()

self.fc1= nn.Linear(120,84)
self.relu4= nn.ReLU()

self.fc2= nn.Linear(84,10)
self.softmax= nn.Softmax(dim= -1)

def forward(self,img, forward_derivative= False):
output= self.conv1(img)
output= self.relu1(output)
output= self.maxpool1(output)

output= self.conv2(output)
output= self.relu2(output)
output= self.maxpool2(output)

output= self.conv3(output)
output= self.relu3(output)

output= output.view(-1,120)
output= self.fc1(output)
output= self.relu4(output)

output= self.fc2(output)
F= self.softmax(output)

# want to comput the jacobian dF/dimg
jacobian= computeJacobian(F,img)#how to write this function

return F, jacobian

• do you want to implement this for training purpose ? because a lot of framework already do that. – Jérémy Blain Sep 20 '18 at 8:54
• No, not for training as I can do that using backward() methods. I want it during evaluation. I need to craft adversarial examples and for that, I need to modify a few pixels in the original image based on the Jacobian matrix. – Akhilesh Pandey Sep 20 '18 at 8:58
• When I say "training"" I was talking about you, train to write something in python. Because there are existing module in python, that does exactly what you want. Why reinventing the wheel ? – Jérémy Blain Sep 20 '18 at 9:00
• I want to compute the Jacobian matrix of the output layer F w.r.t each pixel in the input image. How to get that using modules? Also, the existing module does that using backward derivative whereas I want it using forward derivative. – Akhilesh Pandey Sep 20 '18 at 9:03
• Here is a module in python which implement a lot of attacks : foolbox.readthedocs.io/en/latest/user/installation.html You can find code which implements the JSMA attack (JacobianAttack). Myabe you can find something useful ? also worth to see : raghakot.github.io/keras-vis a module for Keras which implement saliency map creation (I know there are also other module for other framework like Tf, pytorch, but I haven't the links). Maybe you can use it, or look at the code to find ideas. – Jérémy Blain Sep 20 '18 at 9:09

In the paper The Limitations of Deep Learning in Adversarial Settings, Papernot et. al., 2016, IEEE the chain rule is used, "To express $$\nabla F(X∗)$$ in terms of $$X$$ and constant values only."

Earlier is stated, "Our understanding of how changes made to inputs affect a DNN’s output stems from the evaluation of the forward derivative: a matrix we introduce and define as the Jacobian of the function learned by the DNN. The forward derivative is used to construct adversarial saliency maps indicating input features to include in perturbation $$\partial X$$ in order to produce adversarial samples inducing a certain behavior from the DNN;"

And later, "We define the forward derivative as the Jacobian matrix of the function F learned by the neural network during training. For this example, the output of F is one dimensional, the matrix is therefore reduced to a vector (below). Both components of this vector are computable using the adversary’s knowledge, and later we show how to compute this term efficiently."

$$\nabla F(X) = \Big[ \frac {\partial F(X)} {\partial X_1}, \frac {\partial F(X)} {\partial X_2} \Big] \quad\quad (2)$$

Between equation (2) and equation (6) your questeion is answered, resulting in an equation where the chain rule has already been applied.

$$\frac {\partial F_j (X)} {\partial x_i} = \Big(W_{n+1,j} \, . \, \frac {\partial H_n} {\partial x_i} \Big) \times \frac {\partial f_{n+1,j}} {\partial x_i} \;(W_{n+1,j} \, . \, H_n + b_{n+1,j}) \quad\quad(6)$$

It is (6) that you must implement in code, but that is done after the DNN is converged as the paper states above.

Applying the Chain Rule

If $$z = f(y)$$ and $$y = g(x)$$, then $$\frac {dz} {dx} = f'(g(x)) g'(x)$$

Applied twice, if $$w = e(z)$$, $$z = f(y)$$, and $$y = g(x)$$, then $$\frac {dw} {dx} = e'(f(g(x)) f'(g(x)) g'(x)$$

Applied indefinitely we multiply all the first order derivatives of intermediate values, which has the following first order predicate logic expression.

$$\forall \; 0 \le n \lt N \land v_{n + 1} = f_{n + 1}(v_n) \land F_{n+1}(x) = f_{n+1}(F_n(x)) \land g(a) = \dfrac {d} {d a} \, f(a)$$ $$\exists \; g_n(v_0) = \prod_{\,n = 1}^{\,n < N} {g_n} (F_n(v_0))$$

Derivatives of convolution kernel values are the values themselves because they represent attenuation of the form $$y = ax + b$$. This is true of first degree linear activation functions too. The derivative normally used for ReLU (although the x = 0 case is actually undefined) is as follows:

$$\dfrac {d} {dx} R(x) = {\begin{cases}x < 0 & 0\\x = 0 & 1\\x > 0 & 1 \end{cases}}$$

The derivative of a max function used in max pool (although the u = v case is actually undefined) is as follows:

$$\dfrac {d} {dt} M(u, v) = {\begin{cases}u < v & \frac {dv} {dt}\\u = v & \frac {dv} {dt}\\u > v & \frac {du} {dt}\end{cases}}$$

• Thanks for the answer. Once my model is trained i.e the DNN has converged, I want to compute equation (6) but I don't know how to do that when there are different types of layers like conv2d, linear, relu, maxpool etc. Could you please provide me with some simple example (in PyTorch) if possible? Thanks for your help once again. – Akhilesh Pandey Sep 21 '18 at 4:52
• Thanks for the answer. It was really very helpful. However, as of now, I am able to compute the Jacobian w.r.t single X. Is it possible to extend, the same technique in the case where I am working in batches? – Akhilesh Pandey Sep 26 '18 at 5:04
• Yes, I am referring to mini-batch. – Akhilesh Pandey Sep 27 '18 at 3:32