I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers
- Learning rates for Q-learning, by Eyal Even-Dar et al.;
- The asymptotic convergence rate of Q-learning, by Cs. Szepesvari;
- Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms, by Kearns and Singh.
In the latter, they introduce a rather simple approach that seems appealing to me; however, they only sketch it for phased Q-learning.
I would be interested in knowing about any source where I could find the same approach modified for standard Q-learning; in section 4 of the paper, they claim that
all of our results can be generalized to apply to standard Q-learning
Moreover, if you feel I am missing any paper that could be of interest with regards to the finite sample convergence of Q-learning, I would greatly appreciate it if you could post the name of it.