3
$\begingroup$

In Q-learning, I know that the Q-values are updated using the Bellman equation.

$$ Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)] $$

However, for the last state in the Q-table, the Q-values cannot be updated using the maximum Q-value of the next state term since the next state itself does not exist.

So how do we updated the Q-values for the last state? In my mind, the logical answer would be to continue using the Bellman equation but equalling the maximum Q-value for the next state term to 0, which results in:

$$ Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} - Q(S_t,A_t)] $$

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It's just zeroed out (not considered), so $Q(a,s)$ given $s’$ terminal, should just be the expected immediate reward of the last step, thus the update should just be the one you reported (it's also reported in the original paper https://link.springer.com/article/10.1007/BF00992698 in the end notes, but I'm sure to have red it also in the Sutton and Barto book )

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is slightly wrong, where the OP is correct. if $s$ is terminal, then $Q(s, *) = 0$, by definition. perhaps you meant $s'$ and not $s$? $\endgroup$
    – Neil Slater
    Jul 29 at 8:19
  • 1
    $\begingroup$ @NeilSlater yup $\endgroup$
    – Alberto
    Jul 29 at 8:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .