I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return with respect to the policy weights:

$$\nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) R(\tau)\right]$$

Then they give the following explanation:

Taking a step with this gradient pushes up the log-probabilities of each action in proportion to $R(\tau$).

My question is: How does this expression mathematically reflect the fact that this gradient will push up the log probabilities of the actions?

  • 1
    $\begingroup$ The phrase you quote is literally an English reading of the notation in your image. Would an answer that explained what $\nabla_{\theta}$, $\sum$, $\pi_{\theta}$ etc all mean help you in any way? Or is that part clear to you, and you are looking for something more intuitive? $\endgroup$ Jul 9 at 12:58
  • $\begingroup$ I am already familiar with the notations. What I'm struggling to understand is how does this mathematical expression translate into the english explanation given. As I understand it, this gradient tell us how to update the weights so that we increase the return J and we need the grad log prob of the actions to compute it. But why does this also pushes up the log probabilities at the same time? Maybe something more intuitive could help. Thanks for your time. $\endgroup$ Jul 9 at 14:47

The value of the objective depends on policy (probabilities of taking an action). Intuitively speaking, better actions lead to better returns and by "pushing up" the probabilities (log or not same thing since log is monotonically increasing function) of those actions you're making sure you're getting better returns and increasing the value of your objective.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.