Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning:
$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha \times (R+Q(s_{t+1}, max_a)-Q(s_t, a_t))$$
However, I saw a Q-Learning example that uses a different formula, which I'm applying to my own problem and I'm getting good results:
$$Q(s_t, a_t) = R(s_t,a_t) + \alpha \times Q(s_{t+1}, max_a)$$
Is this valid?
No, your second statement does not correctly implement the Q-learning update rule, which the first statement correctly implements.
Your second code snippet is equivalent to this:
$$Q_{k+1}(s,a) \leftarrow r + \alpha \text{max}_{a'} Q_k(s', a')$$
This looks like a simplified Value Iteration update to me, where you have incorrectly switched $\alpha$ (the learning rate) for $\gamma$ (the discount rate).
The full Value Iteration update based on action values looks like this:
$$Q_{k+1}(s,a) \leftarrow \sum_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}_{a'} Q_k(s', a'))$$
This is almost the same as your equation when you have a deterministic environment (so you can directly predict single values $r$ and $s'$ from $s, a$)
As such, it will sort of work with certain assumptions:
You want a specific discount rate, or don't particularly care about predicting values, just finding a close-to-optimal policy
The environment is deterministic
The further away you are from those assumptions, the worse fit the simpler update method will be to your problem. It is definitely not Q-learning either way.
