# Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):

Question: is $$\bar{R}_t$$ average of all rewards or average of rewards corresponding to $$A_t$$?

## 1 Answer

Question: is $$\bar{R}_t$$ average of all rewards or average of rewards corresponding to $$A_t$$?

It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.

It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much any baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.

If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the policy, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use advantage - the difference between Q(s,a) and V(s) - as the scaling to $$\nabla \text{log}(\pi(a|s))$$.

• @ Neil Slater, in the book it says "which can be computed incrementally as described in Section 2.4 (or Section 2.5 if the problem is nonstationary)". However, in sections 2.4 and 2.5, we calculate average of rewards for a given action, not the average of all rewards seen so far. That was the reason why I am confused. Apr 30 at 21:43
• @user3489173 I think you are expected to extract the averaging technique from that reference, not what was being averaged. May 1 at 8:58