1
$\begingroup$

I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial.

My first question is about the end result of this gradient derivation,

\begin{aligned} \nabla \mathbb{E}_{\pi}[r(\tau)] &=\nabla \int \pi(\tau) r(\tau) d \tau \\ &=\int \nabla \pi(\tau) r(\tau) d \tau \\ &=\int \pi(\tau) \nabla \log \pi(\tau) r(\tau) d \tau \\ \nabla \mathbb{E}_{\pi}[r(\tau)] &=\mathbb{E}_{\pi}[r(\tau) \nabla \log \pi(\tau)] \end{aligned}

Mainly in this equation

$$\nabla \mathop{\mathbb{E}_\pi }[r(\tau )] = \mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$$

Does expectation follow a distributive or associative property?

I know that expectations of a function can be written as below

$$\mathop{\mathbb{E}}[f(x)] =\sum p(x)f(x)$$

Then can we rewrite the first equations as

$$\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\nabla log \pi (\tau )] \\= \sum p(\tau)r(\tau ) \,\, \sum p(\tau)\nabla log \pi (\tau ) \\ = p(\tau) \sum r(\tau ) \nabla log \pi (\tau )$$

The problem is when I compare this to PyTorch implementation (line 71-74)

for log_prob, R in zip(policy.saved_log_probs, returns):
    policy_loss.append(-log_prob * R)
optimizer.zero_grad()
policy_loss = torch.cat(policy_loss).sum()

The pytorch implementation simply multiplied log probability and reward -log_prob * R and then summed the vector torch.cat(policy_loss).sum() there is no $p(\tau)$. What is really happening here?

The second question is the multiplication of log probability and reward in PyTorch implementation -log_prob * R, PyTorch implementation has a negative log probability and derived equation has a positive one $\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$. What is the need for multiplying log probability with a negative value in PyTorch implementation?

I have only a basic understanding of maths and that's why I am asking this question here.


Edit: found a better derivation of above equation https://youtu.be/Ys3YY7sSmIA?t=3622

$\endgroup$
1
$\begingroup$

You cannot do this:

$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$

That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation was over simple equiprobable discrete distribution where $\tau$ could be any integer in range $[1,10]$, then $\mathop{\mathbb{E} }[\tau^2] = 38.5$ whilst $\mathop{\mathbb{E} }[\tau]\mathop{\mathbb{E} }[\tau] = 30.25$

The pytorch implementation simply multiplied log probability and reward -log_prob * R and then summed the vector torch.cat(policy_loss).sum() there is no $p(\tau)$. What is really happening here?

The purpose of transforming the gradient into an expectation $\mathbb{E}$ for the policy gradient theorem, is so that you can estimate it using samples taken from the distribution. Typically, you don't know $p(\tau)$, but you do know that if you follow the same process where $p(\tau)$ applies (i.e. measure the return from the environment whilst following the policy represented by the policy function) that you will get an unbiased sample from that distribution.

So what is going on here is that you throw away the outer expectation $\mathbb{E}_{\pi}[]$ and replace it with a stochastic estimate for the same value based on taking samples. The samples are naturally obtained with distribution described by $p(\tau)$, if you follow the policy function when making action choices.

The second question is the multiplication of log probability and reward in pytorch implementation -log_prob * R, pytorch implementation has a negative log probability and derived equation has a positive one $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )]$. What is the need for multipling log probability with negative value in pytorch implementaion?

I don't know the code, but this very likely because of a sign change brought on by considering how to respond to the gradient estimate.

There is a clue in the use of the name "loss". To maximise return in policy gradient methods, you can perform gradient ascent based on the estimated gradient as the goal is to find higher values. However, it is more usual in NN libraries to perform gradient descent in order to minimise a loss function. That is a likely cause of the sign reversal here.

| improve this answer | |
$\endgroup$
  • $\begingroup$ you throw away the outer expectation Eπ[]Eπ[] and replace it with a stochastic estimate for the same value based on taking samples. When I see an expectation equation such as this $\mathop{\mathbb{E}}[f(x)]$ I always consider it as an "average" of that function. When replacing it with stochastic estimate are we now calculating the "total" instead of the "average"? Like we see in the code policy_loss = torch.cat(policy_loss).sum() $\endgroup$ – Eka Nov 8 '19 at 14:19
  • 1
    $\begingroup$ @eka: It will still be an average, but averaged over the samples you took instead of over a distribution function that you (probably) don't know. Although the code you show does indeed caclulate a total. If the batch size is always the same, this is just a scale difference though, and might be corrected for in later code, or in the learning rate etc. $\endgroup$ – Neil Slater Nov 8 '19 at 14:21
  • $\begingroup$ I am really weak in the area of math and I really appreciate that you took your time to explain my queries. Thank you $\endgroup$ – Eka Nov 8 '19 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.