Timeline for Zero reward in policy gradient
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 6, 2023 at 13:08 | comment | added | Jason L | Thanks for your input @Neil. I don't think the quote from the link is incorrect. I've looked at a number of example implementations for vanilla policy gradient and −Gt log π(At|St,θt) is indeed the loss function being minimized. I think my question mainly originates from observing the loss value as training progresses - that I noticed sometimes the agent plays very badly achieving a low score, the loss decreases. Then I realised it could be term Gt in the loss that contributes to this. So it would appear that the agent is being 'rewarded' for bad play. | |
| Jun 5, 2023 at 14:02 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 1321 characters in body
|
| Jun 5, 2023 at 13:54 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 1321 characters in body
|
| Jun 5, 2023 at 13:49 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 1321 characters in body
|
| Jun 5, 2023 at 13:35 | comment | added | Neil Slater | @JasonL That doesn't make much difference - a cost function is conventially the sum of some loss functions. $J(\theta)$ is the cost function, and components of it weighted by state distribution are the loss functions. The quote from the link is incorrect. I will answer your comment in the body of the question | |
| Jun 5, 2023 at 13:17 | comment | added | Jason L | Sorry, I meant loss, not cost function. I didn't remove ∇θ, it was stated in the link that the function we want to minimise is −Gt ln π(At|St,θt). Example from other sources such as github.com/Finspire13/pytorch-policy-gradient-example/blob/… also seems to indicate this is indeed the function we want to minimize (correct me if I am wrong). Anyway, my question still remains - if the policy stumbles upon a set of parameters which makes Gt=0, doesn't that make the estimated gradient zero hence causing the policy to stop improving? | |
| Jun 5, 2023 at 11:45 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 135 characters in body
|
| Jun 5, 2023 at 11:39 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 135 characters in body
|
| Jun 5, 2023 at 10:53 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 69 characters in body
|
| Jun 5, 2023 at 10:45 | history | answered | Neil Slater | CC BY-SA 4.0 |