In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

[![Cliff Walking][1]][1]

[![Figure 6.3 Sutton and Barto (second edition)][2]][2]


As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors however, fail to mention the reason for the divergence.
[![van Seijen et al's explanantion][3]][3]


What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?

According to me, SARSA might have higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA. Additionally, shouldn't Q-learning be at greater risk of diverging Q values since in it's update, we maximise over actions (And I have in fact seen a [number][4] of [instances][5] where there is a problem of diverging Q values in DQNs)? The [majority][6] of [papers][7] I have looked at only talk about the problem from the function approximation perspective.


  [1]: https://i.sstatic.net/GDH0C.png
  [2]: https://i.sstatic.net/h5d6p.png
  [3]: https://i.sstatic.net/h82Gb.png
  [4]: https://datascience.stackexchange.com/questions/22163/why-does-q-learning-diverge
  [5]: https://arxiv.org/abs/1903.08894
  [6]: https://arxiv.org/abs/1107.4606
  [7]: https://link.springer.com/chapter/10.1007/978-3-540-30115-8_44