Timeline for In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 14, 2023 at 9:30 | vote | accept | CommunityBot | ||
| Nov 13, 2023 at 18:52 | comment | added | user77931 | I know that it's intractable to compute a full expectation, that was not my question. Btw I see the point, ty. | |
| Nov 13, 2023 at 13:55 | comment | added | Luca Anzalone | @LorenzoMancini yes, but you need to consider that, in practice, for deep RL the gradient is always approximated because you can't compute a full expectation over all possible trajectories: it's just intractable. What you can do is to sum over a finite num of trajectories, and MC assumes one trajectory is enough. | |
| Nov 13, 2023 at 10:45 | comment | added | user77931 | Hi, yes I know the N-step bootstrapping to set up a trade off between bias and variance but as I've seen it often refers to the computation of the returns (and then the advantages), as it is explained here arxiv.org/pdf/1506.02438.pdf. Indeed, there it turns out that the empirical returns are good estimators of the Q function and one can obtain them with a bootstrapping procedure. However, in the same paper the gradients is still approximated through an average over many trajectories (eq. 9). I cannot find a similar analysis for the whole gradient in the Sutton's book. | |
| Nov 12, 2023 at 16:13 | history | answered | Luca Anzalone | CC BY-SA 4.0 |