3
$\begingroup$

I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return with respect to the policy weights:

$$\nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) R(\tau)\right]$$

Then they give the following explanation:

Taking a step with this gradient pushes up the log-probabilities of each action in proportion to $R(\tau$).

My question is: How does this expression mathematically reflect the fact that this gradient will push up the log probabilities of the actions?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The phrase you quote is literally an English reading of the notation in your image. Would an answer that explained what $\nabla_{\theta}$, $\sum$, $\pi_{\theta}$ etc all mean help you in any way? Or is that part clear to you, and you are looking for something more intuitive? $\endgroup$
    – Neil Slater
    Commented Jul 9, 2021 at 12:58
  • 1
    $\begingroup$ I am already familiar with the notations. What I'm struggling to understand is how does this mathematical expression translate into the english explanation given. As I understand it, this gradient tell us how to update the weights so that we increase the return J and we need the grad log prob of the actions to compute it. But why does this also pushes up the log probabilities at the same time? Maybe something more intuitive could help. Thanks for your time. $\endgroup$
    – Samuel Beaussant
    Commented Jul 9, 2021 at 14:47
  • $\begingroup$ Logarithm of Pi has the same monotonicity of policy Pi and its peaks and valleys are at the same loci as Pi's. Given some rewards for a specific trajectory, we assign the heavier weights to the actions as peaks in policy Pi corresponding to the highest rewards in order to maximize the total rewards. This is done by Gradient Ascent Algorithm for policy Pi(Remember Back Propagation). PS., we introduce Logarithm so that we can apply Sampling Method to estimate the reward for a trajectory. $\endgroup$
    – Kuo
    Commented Jun 21, 2023 at 10:08

1 Answer 1

Reset to default
1
$\begingroup$

The value of the objective depends on policy (probabilities of taking an action). Intuitively speaking, better actions lead to better returns and by "pushing up" the probabilities (log or not same thing since log is monotonically increasing function) of those actions you're making sure you're getting better returns and increasing the value of your objective.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .