# How to interpret the policy gradient expression in reinforcement learning?

I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return with respect to the policy weights:

$$\nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) R(\tau)\right]$$

Then they give the following explanation:

Taking a step with this gradient pushes up the log-probabilities of each action in proportion to $$R(\tau$$).

My question is: How does this expression mathematically reflect the fact that this gradient will push up the log probabilities of the actions?

• The phrase you quote is literally an English reading of the notation in your image. Would an answer that explained what $\nabla_{\theta}$, $\sum$, $\pi_{\theta}$ etc all mean help you in any way? Or is that part clear to you, and you are looking for something more intuitive? Jul 9, 2021 at 12:58
• I am already familiar with the notations. What I'm struggling to understand is how does this mathematical expression translate into the english explanation given. As I understand it, this gradient tell us how to update the weights so that we increase the return J and we need the grad log prob of the actions to compute it. But why does this also pushes up the log probabilities at the same time? Maybe something more intuitive could help. Thanks for your time. Jul 9, 2021 at 14:47

## 1 Answer

The value of the objective depends on policy (probabilities of taking an action). Intuitively speaking, better actions lead to better returns and by "pushing up" the probabilities (log or not same thing since log is monotonically increasing function) of those actions you're making sure you're getting better returns and increasing the value of your objective.