In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors however, fail to mention the reason for the divergence.

What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?

According to me, SARSA might have higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA. Additionally, shouldn't Q-learning be at greater risk of diverging Q values since in it's update, we maximise over actions (And I have in fact seen a number of instances where there is a problem of diverging Q values in DQNs)? The majority of papers I have looked at only talk about the problem from the function approximation perspective.

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors, however, fail to mention the reason for the divergence.

What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?

According to me, SARSA might have a higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA. 

Additionally, shouldn't Q-learning be at greater risk of diverging Q values since, in its update, we maximise over actions (and I have in fact seen a number of instances where there is a problem of diverging Q values in DQNs)? 

The majority of papers I have looked at only talk about the problem from the function approximation perspective.

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors however, fail to mention the reason for the divergence.  

What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?

According to me, SARSA might have higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA. Additionally, shouldn't Q-learning be at greater risk of diverging Q values since in it's update, we maximise over actions (And I have in fact seen a number of instances where there is a problem of diverging Q values in DQNs)? The majority of papers I have looked at only talk about the problem from the function approximation perspective.

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors however, fail to mention the reason for the divergence. 

What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?

According to me, SARSA might have higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA. Additionally, shouldn't Q-learning be at greater risk of diverging Q values since in it's update, we maximise over actions (And I have in fact seen a number of instances where there is a problem of diverging Q values in DQNs)? The majority of papers I have looked at only talk about the problem from the function approximation perspective.