Timeline for PPO when does the update happen?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 2, 2021 at 7:56 | history | edited | user42664 | CC BY-SA 4.0 |
made answer more useful
|
| Sep 1, 2021 at 20:30 | comment | added | user42664 | Yes, and an estimation of the value function from the final step via the value network. | |
| Sep 1, 2021 at 20:24 | comment | added | hridayns | Okay, thanks! So my final takeaway is that it uses the rewards observed up until that point to create a 'value target' at every step from which we can obtain value function loss by calculating difference between value function estimate and that 'value target'. To calculate this partial value target in between the episode, it uses only the discounted sum of rewards seen until that step, correct? | |
| Sep 1, 2021 at 20:21 | vote | accept | hridayns | ||
| Sep 1, 2021 at 19:53 | comment | added | hridayns | I understand. But in order to get the gradients that will be used to nudge the weights, it needs to calculate a loss. One of the terms in the PPO loss is the value function loss. How does it calculate that without the value target - which can only be estimated at the end of the episode from the rewards? Does it use the discounted sum of rewards only until that step as the value target? | |
| Sep 1, 2021 at 18:41 | comment | added | hridayns | I mean...yes, there are immediate rewards. But my question is actually what update does the PPO do at every step? What is changed? I don't even need the actual formula. Just need to know if it changes the weights of either the value function (critic) or policy network (actor) at any step before the end of the episode. And if yes, then what does it use to make the update? is value target = advantage? | |
| Sep 1, 2021 at 18:36 | history | answered | user42664 | CC BY-SA 4.0 |