In the Berkeley RL class they mention the gradient would be 0 if the policy is deterministic. Why is that?
https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071
3
$\begingroup$
$\endgroup$
9
$\begingroup$
$\endgroup$
First, a small clarification; they do not say that "the gradient of a deterministic policy is 0" (I'm not sure what, if anything, "the gradient of a policy") would mean. They are talking about how "this gradient" (referring to the gradient on the slide visible in the video) is 0 in the case where the policy ($\pi$) is deterministic.
Now, here is the gradient that they are discussing in the video:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right)$$
In this equation, $\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t})$ denotes the probability of our policy $\pi_{\theta}$ selecting the actions $\mathbf{a}_{i, t}$ that it actually ended up selecting in practice, given the states $\mathbf{s}_{i, t}$ that it encountered during the episode that we're looking at.
In the case of a deterministic policy $\pi_{\theta}$, we know for sure that the probability of it selecting the actions that it did select must be $1$ (and the probability of it selecting any other actions would be $0$, but such a term does not show up in the equation). So, we have $\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) = 1$ for every instance of that term in the above equation. Because $\log 1 = 0$, this leads to:
\begin{aligned} \nabla_{\theta} J(\theta) &\approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\ % &= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log 1 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\ % &= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} 0 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\ % &= \frac{1}{N} \sum_{i=1}^N 0 \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\ % &= 0 \\ \end{aligned}
(i.e. you end up with a sum of terms that are all multiplied by $0$).
answered Sep 6 '18 at 14:44
-
$\begingroup$ Edited the question based on your clarification. Thank you very much for taking the time to explain this. $\endgroup$ – jonperl Sep 6 '18 at 22:54
-
$\begingroup$ Just thinking this through, for the case when the policy is stochastic. If you take a step in the direction of $\nabla_{\theta} \log \pi_{\theta}$, this will increase the probability the policy takes an action -- since the log from (0 -> 1] is decreasingly negative. Since the gradient is multiplied by the reward, by increasing the probability of an action you increase the expected value of that state-action reward (when the reward is positive, decrease the expected value if it is negative). Is this correct? $\endgroup$ – jonperl Sep 7 '18 at 1:41
-
$\begingroup$ Ok, yes that is basically what he says at youtu.be/XGmd3wcyDg8?t=1295. Thank you again so much -- I couldn't get past that part of the video until I understood the formula. $\endgroup$ – jonperl Sep 7 '18 at 2:00
-
1
2
$\begingroup$
$\endgroup$
Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.
First, having a deterministic policy inside the log would do create trivial terms.
Secondly, for me, in Policy Gradient methods, it's a non sense to have a deterministic policy during the optimization, because you want to explore the space of weights. In my experience, you only set the policy to deterministic (in PG method) when you're done with the optimization, and you want to test your network.
