I am doing self study of Reinforcement Learning with Q-learning using online resources like blog posts, youtube videos and books and at this point, I have learned the underpinning concepts of Reinforcement learning and how to update the q values using look up table.
I came across that convergence is no longer guaranteed when the Q-function is approximated. https://wiki.ubc.ca/Course:CPSC522/Reinforcement_Learning_with_Function_approximation#:~:text=Function%20Approximation,-Optimal%20and%20Approximate&text=One%20thing%20that%20needs%20to,the%20weights%20of%20the%20network
I am trying to understand convergence considering the Bellman equation. Also trying to understand why when using approximation, convergence is no longer guaranteed.
Why is that?