# Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed.

The algorithm starts with an initial policy. Within each episode, the algorithm conducts policy evaluation and does NOT update the policy. Once the episode is done, the policy is updated using the greedy policy based on the current learnt Q-values. The process then repeats. I attached the algorithm as a picture.

Just to emphasize that the updating policy does not change within each episode. The policy at each state is updated AFTER one episode is done, using the Q-tables.

Has anyone seen this kind of Q-learning before? If so, could you please kindly guide me to some resources regarding the convergence? Thank you!

• It's on-policy generalised policy iteration as written - kind of a hybrid of Monte Carlo control and SARSA - so I am surprised to see it called Q learning. Also, no exploration, which could be a weakness in some environments. Unfortunately I have not seen it before and would not know where to find convergence resources. Could you perhaps link to where you found it, because that may give some clues? – Neil Slater May 22 at 23:20
• @NeilSlater Thanks for the comment. This is what I had in mind and had it written down, so I do not have a link. As for no exploration, I think the action selection here is not a big deal. One can just replace the sampling with an epsilon greedy selection. Also, could you please explain why this thing looks like a SARSA? – Scott Guan May 22 at 23:32
• To make it off-policy you could change the Q value update step to take a max over possible actions in $s_{t+1}$ instead of making the on-policy update. Doesn't fix the lack of exploration, but does mean you would be estimating a different target polciy from current behaviour policy. – Neil Slater May 22 at 23:35
• Yes it would be simple enough to have the behaviour policy as epsilon greedy. I suggest you write it like that though, because it is an important detail if you want to talk about convergence. Without exploration, convergence guarantees will be weaker – Neil Slater May 22 at 23:37
• @NeilSlater I will update it. Thanks. – Scott Guan May 23 at 0:47