Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.
It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much any baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.
If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the policy, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use advantage - the difference between Q(s,a) and V(s) - as the scaling to $\nabla \text{log}(\pi(a|s))$.
answered Apr 30, 2022 at 18:37
-
$\begingroup$ @ Neil Slater, in the book it says "which can be computed incrementally as described in Section 2.4 (or Section 2.5 if the problem is nonstationary)". However, in sections 2.4 and 2.5, we calculate average of rewards for a given action, not the average of all rewards seen so far. That was the reason why I am confused. $\endgroup$ Commented Apr 30, 2022 at 21:43
-
$\begingroup$ @user3489173 I think you are expected to extract the averaging technique from that reference, not what was being averaged. $\endgroup$ Commented May 1, 2022 at 8:58
-
$\begingroup$ @NeilSlater Here in this problem statement it is said "On each step, after selection action $A_t$ and receiving the reward $A_t$, the action preferences are updated by...:". I am confused because after you apply $A_t$, you obtain reward $R_{t+1}$. Hence. why in the update equation 2.12, do we not have $R_{t+1}$ instead of $R_t$? $\endgroup$ Commented Feb 25 at 18:09
-
$\begingroup$ @DSPinfinity The choice of labelling state, action, reward in a trajectory $s_t, a_t, r_t$ or $s_t, a_t, r_{t+1}$ is one of convention, with the majority of sources I have seen using the latter. Something to be aware of when using different learning sources, as it will change the notation in some of the equations, proofs etc $\endgroup$ Commented Feb 25 at 19:18
-
1$\begingroup$ @DSPinfinity Well chapter 2 is about bandits, and time steps are entirely separate interactions with the bandit environment. It would make no sense at all to have action $a_t$ associated with reward $r_{t+1}$ when talking about bandits. I missed that in my comment above that the source was S&B $\endgroup$ Commented Feb 25 at 19:36