I am doing self study of Reinforcement Learning with Q-learning using online resources like blog posts, youtube videos and books and at this point, I have learned the underpinning concepts of Reinforcement learning and how to update the q values using look up table.

I came across that convergence is no longer guaranteed when the Q-function is approximated. https://wiki.ubc.ca/Course:CPSC522/Reinforcement_Learning_with_Function_approximation#:~:text=Function%20Approximation,-Optimal%20and%20Approximate&text=One%20thing%20that%20needs%20to,the%20weights%20of%20the%20network

I am trying to understand convergence considering the Bellman equation. Also trying to understand why when using approximation, convergence is no longer guaranteed.

Why is that?

  • $\begingroup$ Please, see the linked question (which I think your question is a duplicate of) and answers. If you think your question is different, let me know and I could re-open this post. $\endgroup$ – nbro Dec 19 '20 at 12:43