Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a mathematical expression to work with, how can its derivatives be calculated?

  • $\begingroup$ What do you mean Neural Networks does not have a mathmatical expression to work with? Neural Networks is purely math in its nature? $\endgroup$ – SandMan Sep 12 '19 at 11:20

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?

Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation.

First, take the output of all the neurons (except the one you're about to differentiate by) as a function:

enter image description here

The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.

You then differentiate NN by a single neuron (n) as shown:

$$\frac{\partial NN}{\partial n} = \lim_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$

In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):

$$\frac{\partial A}{\partial n}$$

So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.

I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.

  • $\begingroup$ So, when we talk about the derivative of a NN w.r.t a weight, we are actually refering to the change in the output of the NN caused by the change of one neuron( to which that weight is associated). Can you explain a bit more why "In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n)"? Thanks. $\endgroup$ – qiang li Sep 12 '19 at 17:16
  • 1
    $\begingroup$ @qiangli Sure- differentiating the activation function is a mathematical shortcut to computing the weight of each neuron. Instead of plugging everything into the neural net to iterate over again and again, we can activate the result and get a gradient per-neuron. This is possible through some matrix calculus and algebraic rearranging: it's all available online, but learning the rearrangement probably won't deepen your understanding. $\endgroup$ – mcRobusta Sep 12 '19 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.