# In Q-learning, how are Q values updated for the last state in the Q table?

In Q-learning, I know that the Q-values are updated using the Bellman equation.

$$Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)]$$

However, for the last state in the Q-table, the Q-values cannot be updated using the maximum Q-value of the next state term since the next state itself does not exist.

So how do we updated the Q-values for the last state? In my mind, the logical answer would be to continue using the Bellman equation but equalling the maximum Q-value for the next state term to 0, which results in:

$$Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} - Q(S_t,A_t)]$$

It's just zeroed out (not considered), so $$Q(a,s)$$ given $$s’$$ terminal, should just be the expected immediate reward of the last step, thus the update should just be the one you reported (it's also reported in the original paper https://link.springer.com/article/10.1007/BF00992698 in the end notes, but I'm sure to have red it also in the Sutton and Barto book )
• This is slightly wrong, where the OP is correct. if $s$ is terminal, then $Q(s, *) = 0$, by definition. perhaps you meant $s'$ and not $s$? Jul 29 at 8:19