Timeline for Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 19, 2023 at 18:45 | vote | accept | Luca Anzalone | ||
| Apr 28, 2023 at 17:49 | comment | added | Luca Anzalone | Well, I can think of an env in which there are three paths that lead to a terminal state, the paths on the left and right are highly rewarding while the middle path is poor. Suppose the actions are steer left/right (from -1 to 1), the "mode" policy will always pick something close to either -1 or 1, while the "mean" policy just average to 0 (or so) leading to the non-rewarding path. In this case the best to do would either to be deterministic or taking one or the modes randomly at each trial. | |
| Apr 24, 2023 at 21:44 | history | edited | nbro | CC BY-SA 4.0 |
deleted 51 characters in body
|
| Apr 23, 2023 at 17:52 | comment | added | nbro | @LucaAnzalone That's actually a good question. I shouldn't have used "only if" but "if" in my last sentence. In other words, I just wanted to point out that, if you have a finite MDP, you're sure that the optimal policy is deterministic. In other cases, it might not be. So, I did not want to say that in continuous actions MDPs the optimal policy is always stochastic, because I don't know if that's the case (I think not - one can probably think of a very simple environment with 2 continuous actions where the optimal policy is deterministic, but I didn't think much about this). | |
| Apr 23, 2023 at 17:46 | history | edited | nbro | CC BY-SA 4.0 |
deleted 5 characters in body
|
| Apr 23, 2023 at 13:06 | comment | added | Luca Anzalone | Keep as an answer it's interesting. I'm aware of cases like rock-paper-scissor in which the best policy is always stochastic to avoid the opponent to exploit the determinism. But excluding that (I may clarify this in the question), I'd like to understand what is the best to do at inference time with the policy. Actually, could you elaborate a bit more about the difference of finite MDPs and continuous MDPs, in particular why in the continuous case one don't want to have deterministic actions? | |
| Apr 23, 2023 at 10:26 | comment | added | nbro | Maybe I should convert this to 1 or more comments, if this doesn't even partially answer your questions. | |
| Apr 23, 2023 at 10:20 | history | answered | nbro | CC BY-SA 4.0 |