Expected SARSA vs SARSA in "RL: An Introduction" [closed]

Question

Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences:

Expected SARSA is more complex computationally than Sarsa but, in return, it eliminates the variance due to the random selection of $A_{t+1}$. Given the same amount of experience we might expect it to perform slightly better than Sarsa, and indeed it generally does.

I have three questions concerning this statement:

1. Why is the action selection random with Sarsa? Isn't it on-policy and therefore $\epsilon$-greedy?
2. Because Expected-Sarsa is off-policy the experience it learns from can be from any policy that at least explores everything in the limit e.g. random action-selection with equal probabilities for every action. How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an $\epsilon$-greedy policy, especially with the same amount of experience?
3. Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?

Answer 1

Why is the action selection random with Sarsa?

A policy could be stochastic. In the case of SARSA, it is stochastic because of the use of $\epsilon$-greedy.

Isn't it on-policy and therefore ϵ-greedy?

I don't quite understand the question. SARSA is on-policy evaluation with $\epsilon$-greedy policy. Q-learning is off-policy evaluation with $\epsilon$-greedy policy. $\epsilon$-greedy is just a way to turn an action-value function into a policy.

Because Expected-Sarsa is off-policy the experience it learns from can be from any policy ... How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an ϵ-greedy policy, especially with the same amount of experience?

It is unfair to compare different natures of experiences because off-policy experience contains less useful information. Thus, both SARSA and Expected SARSA should use their own on-policy experience for comparison.

While Expected SARSA update step guarantees to reduce the expected TD error, SARSA could only achieve that in expectation (taking many updates with sufficiently small learning rate). Through this perspective, there is little doubt that Expected SARSA should be better.

Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?

Same as the previous answer, it is unfair to compare them without the same quality of experiences.