1
$\begingroup$

Referring to this post, in the following formula to update the state-action value

$$ Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$

is the value of $G$ (the return) the same for every state-action $(s,a)$ pair?

I am a little confused about this point, so I will thank any clarification.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s_t, a_t, r_t, s_{t+1})$ and then, for every visit MC, you update

$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (G_t - Q(s_t, a_t))\;;$$

where $G_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$, for each $t$ in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you @DavidlIreland. So, in the case that there is only one reward at the final step as in chess, will every state-action pair, sampled in this episode, have the same return? Or generally, in the case that there are succesive actions with no rewards, will they share the return value in order to update themselves? $\endgroup$
    – Hermes Morales
    Commented Mar 20, 2021 at 14:41
  • 1
    $\begingroup$ They would be the same if the discount factor used is 1, if not then they will still be different. $\endgroup$
    – David
    Commented Mar 21, 2021 at 10:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .