Timeline for What are the conditions for the convergence of SARSA to the optimal value function?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 27, 2020 at 14:43 | comment | added | nbro | @KaneM This is an interesting question. Maybe ask on the site! I wouldn't currently be able to answer it, but maybe there's already some research work that goes in this direction. In all convergence results I've seen so far, the RM conditions are assumed (essentially, because these RL theorems and proofs follow from the original results by Robbins and Monro, AFAIK). | |
| Feb 27, 2020 at 14:32 | comment | added | KaneM | I wonder if the Robbins-Monro conditions are not present, does there exist some range of $\alpha$ such that the policy converges in some notion of expectation to the optimal policy. | |
| Feb 27, 2020 at 14:03 | comment | added | nbro | @KaneM These conditions are very related. I'm linking to a paper that seems to relate the two in a more precise way, but I haven't yet fully read this paper. | |
| Feb 27, 2020 at 13:59 | comment | added | KaneM | I believe they are separate conditions. From my understanding, the RM conditions ensure that the Q function eventually converges to a value while the condition that every state-action pair is visited infinitely often ensure that it converges to the correct value. | |
| Feb 27, 2020 at 13:48 | comment | added | nbro | @KaneM No problem! I have a look at source code of my and your answer (now after my edits) to understand better how markdown works :) Anyway, I am not completely sure that the Robbins-Monro conditions ensure that each state is visited infinitely often. Maybe it's the other way around: if each state-action pair is visited infinitely often, then the Robbins-Monro conditions are satisfied. | |
| Feb 27, 2020 at 13:45 | comment | added | KaneM | Thanks for the answer. Sorry about my poor formatting of posts here. Just learning about markdown. | |
| Feb 27, 2020 at 13:42 | vote | accept | KaneM | ||
| Feb 27, 2020 at 13:40 | comment | added | nbro | I was already writing this answer before the other answer was published, but these answers are equivalent. I am only citing the paper that originally proved this. | |
| Feb 27, 2020 at 13:40 | history | answered | nbro | CC BY-SA 4.0 |