I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial.
My first question is about the end result of this gradient derivation,
\begin{aligned} \nabla \mathbb{E}_{\pi}[r(\tau)] &=\nabla \int \pi(\tau) r(\tau) d \tau \\ &=\int \nabla \pi(\tau) r(\tau) d \tau \\ &=\int \pi(\tau) \nabla \log \pi(\tau) r(\tau) d \tau \\ \nabla \mathbb{E}_{\pi}[r(\tau)] &=\mathbb{E}_{\pi}[r(\tau) \nabla \log \pi(\tau)] \end{aligned}
Mainly in this equation
$$\nabla \mathop{\mathbb{E}_\pi }[r(\tau )] = \mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$$
Does expectation follow a distributive or associative property?
I know that expectations of a function can be written as below
$$\mathop{\mathbb{E}}[f(x)] =\sum p(x)f(x)$$
Then can we rewrite the first equations as
$$\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\nabla log \pi (\tau )] \\= \sum p(\tau)r(\tau ) \,\, \sum p(\tau)\nabla log \pi (\tau ) \\ = p(\tau) \sum r(\tau ) \nabla log \pi (\tau )$$
The problem is when I compare this to PyTorch implementation (line 71-74)
for log_prob, R in zip(policy.saved_log_probs, returns):
policy_loss.append(-log_prob * R)
optimizer.zero_grad()
policy_loss = torch.cat(policy_loss).sum()
The pytorch implementation simply multiplied log probability and reward -log_prob * R
and then summed the vector torch.cat(policy_loss).sum()
there is no $p(\tau)$. What is really happening here?
The second question is the multiplication of log probability and reward in PyTorch implementation -log_prob * R
, PyTorch implementation has a negative log probability and derived equation has a positive one $\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$. What is the need for multiplying log probability with a negative value in PyTorch implementation?
I have only a basic understanding of maths and that's why I am asking this question here.
Edit: found a better derivation of above equation https://youtu.be/Ys3YY7sSmIA?t=3622