Timeline for Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 21, 2022 at 13:52 | vote | accept | Peter | ||
| Jul 9, 2022 at 18:08 | comment | added | matorbi | We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar). What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it. | |
| Jul 8, 2022 at 16:25 | comment | added | Peter | Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}_{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}_{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$. | |
| Jul 6, 2022 at 21:37 | history | edited | matorbi | CC BY-SA 4.0 |
[Edit removed during grace period]
|
| Jul 6, 2022 at 19:21 | history | edited | matorbi | CC BY-SA 4.0 |
added 67 characters in body
|
| Jul 6, 2022 at 19:09 | history | edited | matorbi | CC BY-SA 4.0 |
added 12 characters in body
|
| Jul 6, 2022 at 18:58 | history | answered | matorbi | CC BY-SA 4.0 |