3
$\begingroup$

I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question -

enter image description here

While transitioning from the first line to the second, the gradient with respect to policy parameters $\theta$ was pushed into the summation. What bothers me is how it skipped over $\mu (s)$, the distribution of states - which (the way I understand it), is induced by the policy $\pi_\theta$ itself! Why then does it not depend on $\theta$?

Let me know what's going wrong! Thank you!

$\endgroup$
1
  • $\begingroup$ Form a math view, If the expression of expected value of reward is correct then that is the correct thing to do. $\mu$ is a function of state $s$ so a derivative w.r.t $\theta$ will not affect it. So if what you are claiming is true then $\mu$ would be a function of $\theta$. So you just need to check whether the expression of expected reward is correct or not. $\endgroup$
    – user9947
    Aug 27 '20 at 11:20
2
$\begingroup$

The proof you are given in the above post is not wrong. It's just they skip some of the steps and directly written the final answer. Let me go through those steps:

I will simplify some of the things to avoid complication but the generosity remains the same. Like I will think of the reward as only dependent on the current state, $s$, and current action, $a$. So, $r = r(s,a)$

First, we will define the average reward as: $$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)\sum_{s^{\prime}} P_{ss'}^{a} r $$ We can further simplify average reward as: $$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)r(s,a) $$ My notation may be slightly different than the aforementioned slides since I'm only following Sutton's book on RL. Our objective function is: $$ J(\theta) = r(\pi) $$ We want to prove that: $$ \nabla_{\theta} J(\theta) = \nabla_{\theta}r(\pi) = \sum_s \mu(s) \sum_a \nabla_{\theta}\pi(a|s) Q(s,a)$$

Now let's start the proof: $$\nabla_{\theta}V(s) = \nabla_{\theta} \sum_{a} \pi(a|s) Q(s,a)$$ $$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s) + \pi(a|s) \nabla_{\theta}Q(s,a)]$$ $$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s) + \pi(a|s) \nabla_{\theta}[R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})]]$$ $$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s) + \pi(a|s) [- \nabla_{\theta}r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})]]$$ $$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s) + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}r(\pi)\sum_{a}\pi(a|s)$$ Now we will rearrange this: $$\nabla_{\theta}r(\pi) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s) + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}V(s)$$ Multiplying both sides by $\mu(s)$ and summing over $s$: $$\nabla_{\theta}r(\pi) \sum_{s}\mu(s)= \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s) + \sum_{s}\mu(s) \sum_a \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$ $$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s) + \sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$ Now we are there: $$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)$$ This is the policy gradient theoram for average reward formulation (ref. Policy gradient).

$\endgroup$
15
  • $\begingroup$ Do you mean $\pi (a|s)$ instead of $\pi (s|a)$? Correct me if I'm misinterpreting! $\endgroup$ Aug 29 '20 at 6:13
  • 1
    $\begingroup$ Sorry, it should be $\pi(a|s)$. It was my mistake. $\endgroup$ Aug 29 '20 at 6:33
  • 1
    $\begingroup$ If you open the summation you can see clearly this is exactly telling us what is the probability of the agent to go to the state $s'$ from the state $s$. $\endgroup$ Aug 29 '20 at 6:39
  • 1
    $\begingroup$ Okay, I watched their lecture. At the time 40.55 they, replace $\sum r(s,a)$ with $Q(s,a)$. $\endgroup$ Aug 29 '20 at 7:11
  • 1
    $\begingroup$ $Q(s,a)$ is an unbiased estimate of $R(s,a)$ — ie they are the same in expectation. @cogito_ai $\endgroup$ Aug 29 '20 at 8:58
3
$\begingroup$

The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that $\mu(s)$ depends on the policy $\pi$ which in turn depends on the policy parameters $\theta$, and so there should be a derivative of $\mu$ wrt $\theta$, however the Policy Gradient Theorem doesn't require you to take this derivative.

In fact, the great thing about the Policy Gradient Theorem is that the final result does not require you to take a derivative of the state distribution with respect to the policy parameters. I would encourage you to read and go through the derivation of the Policy Gradient Theorem from e.g. Sutton and Barto to see why you don't need to take the derivative.

enter image description here

Above is an image of the Policy Gradient Theorem proof from the Sutton and Barto book. If you carefully go through this line by line you will see that you are not required to take a derivative of the state distribution anywhere in the proof.

$\endgroup$
6
  • $\begingroup$ Could you please elaborate and possibly provide mathematical details? I've gone through the derivation in Sutton and Barto and I'm still unclear why the derivation above works. $\endgroup$ Aug 28 '20 at 3:22
  • 1
    $\begingroup$ There’s not much to elaborate on. If you followed the Sutton and Barto proof you would see that you don’t need to take a derivative of the state distribution. I can’t really explain any clearer than just writing out the full proof which I am not going to do since that is not the point of this site. If you have a particular question about the proof that you find unclear then you should post that and I’d be happy to answer it. $\endgroup$ Aug 28 '20 at 8:53
  • $\begingroup$ Also, the above isn't a derivation. It is an extremely poor example, especially if they have not previously gone through the policy gradient theorem proof. What they have written (first two lines) gets the right answer but goes about it the wrong way. $\endgroup$ Aug 28 '20 at 9:21
  • $\begingroup$ I understand the proof written in Sutton and Barto's text. My question was - why does the wrong proof (in my post) work? I apologize if you've already explained this, but I still don't see why the wrong proof works. Sutton and Barto's proof doesn't require us to take the derivative of the state distribution anywhere, but that is no reason for me to overlook the same in the proof I've attached in my post. Please explain. Thanks! $\endgroup$ Aug 28 '20 at 13:37
  • $\begingroup$ What you have posted is not a proof. They have started with the final result of the Policy Gradient Theorem, and then they have derived the update that is used in REINFORCE. As I say, the source you are using does not look like something I would recommend. The equality on the first line should be replaced directly with the equality on the second line to avoid confusion. Regardless, if you want good source material for Reinforcement Learning then I would recommend you read Sutton and Barto to prepare you for more advanced topics you will find in papers. $\endgroup$ Aug 28 '20 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.