Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Question

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):

Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Answer 1

Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.

It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much any baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.

If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the policy, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use advantage - the difference between Q(s,a) and V(s) - as the scaling to $\nabla \text{log}(\pi(a|s))$.