Why are there two different q-learning formulas?

Question

I found the following q-learning formula:

in this youtube video: https://www.youtube.com/watch?v=4C133ilFm3Q&t=521s

I'm now a bit confused, since I thought, that the following one is the correct one:

Could somebody explain me which one is the correct one?

Answer 1

The first equation (the one from the video) looks very wrong, for a few different reasons:

• It doesn't involve state-action values $Q(s, a)$, but only something that looks like "state values" $Q(s)$. Q-learning is about state-action values, so there should be both a state and an action as "input" of the $Q$ function.
• It is not clear what the $\max$ operator is maximising over.
• There should be an $s'$ rather than $s$ for the first $Q(\cdot$) inside the $\max$.
• A part of the equation on the right-hand side is simply missing, it ends very abruptly and is incomplete.

---

The update rule that you think is correct, is indeed correct. However, it can also be rewritten (will be mathematically equivalent) in a form that is more similar to the first one, albeit with all of the mistakes that I pointed out above fixed. I'll start with the equation that you already recognised as being correct, and in a few small steps re-write it in a different form:

\begin{align*} Q(s, a) &\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') - Q(s, a) \right) \\ % Q(s, a) &\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right) - \alpha Q(s, a) \\ % Q(s, a) &\gets Q(s, a) - \alpha Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right) \\ % Q(s, a) &\gets \left( 1 - \alpha \right) Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right) \end{align*}