3
$\begingroup$

I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at least on high level of understanding.

But what I don't know is how are partial derivatives itself computed or better said, how are they implemented in code.

I have checked few resources like this lecture slides from CS231N, this blog post or this blog post on TowardsDatascience. They explain graphs and how are expressions calculated in graph, but they don't explain how are partial derivatives derived (or I didn't understand from those explanations). For example, blog post from TowardsDatascience says:

Next, we need to calculate the partial derivatives of each connection between operations, represented by the edges. These are the calculations of the partials of each edge:

And then they show image with values of partial derivatives but they newer explain how are these equations actually calculated in implementation of graph.

Yes, okay, I know how to calculate these partial derivatives on paper and then hardcode them in my code, but I don't know how are they actually automatically computed and implemented in code of libraries like Torch or Theano.

Do they have some basic rules implemented in code, like, for example:

$$ \frac{\partial (a + b)}{\partial a} = \frac{\partial a}{\partial a} + \frac{\partial b}{\partial a} = 1 $$

and then decompose expressions until they reach basic elements/rules or is there another way that libraries like Torch, Theano or TF do it?

Or to put it in another way, if I have this code in Torch:

from torch import Tensor
from torch.autograd import Variable

def element(val):
    return Variable(Tensor([val]), requires_grad=True)

# Input nodes
i = element(2)
j = element(3)
k = element(5)
l = element(7)

# Middle and output layers
m = i*j
n = m+k
y = n*l

# Calculate the partial derivative
y.backward()
dj = j.grad
print(dj)

how does Torch know, that is, how does it compute internally that $ \frac{\partial y}{\partial j} = l \cdot 1 \cdot i = l \cdot i $?

$\endgroup$
1
  • $\begingroup$ I'm not sure if that's the actual implementation, but you just need to find all the paths from the input node to the output node in the computational graph (which is directed), compute the gradients along them by multiplying the edge gradients along each path, then sum them to the get the total gradient. The hard part is finding all the paths, but there's well-known algorithms for that. $\endgroup$
    – David Cian
    Commented Nov 12, 2021 at 20:28

0

You must log in to answer this question.

Browse other questions tagged .