# When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

Referring to this post, in the following formula to update the state-action value

$$Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$

is the value of $$G$$ (the return) the same for every state-action $$(s,a)$$ pair?

I am a little confused about this point, so I will thank any clarification.

## 1 Answer

The discussion uses poor notation, there should be a time index. You obtain a list of tuples $$(s_t, a_t, r_t, s_{t+1})$$ and then, for every visit MC, you update

$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (G_t - Q(s_t, a_t))\;;$$

where $$G_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$$, for each $$t$$ in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.

• Thank you @DavidlIreland. So, in the case that there is only one reward at the final step as in chess, will every state-action pair, sampled in this episode, have the same return? Or generally, in the case that there are succesive actions with no rewards, will they share the return value in order to update themselves? – Hermes Morales Mar 20 at 14:41
• They would be the same if the discount factor used is 1, if not then they will still be different. – David Ireland Mar 21 at 10:30