Referring to this post, in the following formula to update the state-action value

$$ Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$

is the value of $G$ (the return) the same for every state-action $(s,a)$ pair?

I am a little confused about this point, so I will thank any clarification.


The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s_t, a_t, r_t, s_{t+1})$ and then, for every visit MC, you update

$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (G_t - Q(s_t, a_t))\;;$$

where $G_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$, for each $t$ in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.

  • $\begingroup$ Thank you @DavidlIreland. So, in the case that there is only one reward at the final step as in chess, will every state-action pair, sampled in this episode, have the same return? Or generally, in the case that there are succesive actions with no rewards, will they share the return value in order to update themselves? $\endgroup$ – Hermes Morales Mar 20 at 14:41
  • 1
    $\begingroup$ They would be the same if the discount factor used is 1, if not then they will still be different. $\endgroup$ – David Ireland Mar 21 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.