# Why does Q-function training not query the Q-function value at unobserved states?

It says here in the Conservative Q-Learning paper that "standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions" (Section 3.1).

I don't see how this is true. For every $$(s,a)$$ pair, we need to update $$Q(s,a)$$ to reduce the value $$|Q(s,a) - R(s,a) - \gamma E[\max_{a'}Q(s',a')]|$$ until it converges to zero.

We see the existence of both $$a'$$ and $$s'$$, and $$s'$$ could be unseen, for example, on the very first update, where we are at $$s$$, take action $$a$$, and could arrive at any state $$s'$$.

Can someone explain this?

• I believe they are referring to the fact that we only update the Q-function for state values that we have seen. This is true as typically we use a replay buffer of stored experience to update the Q-function -- all of these values have been 'seen' as we have experienced them. Sep 7 at 15:44

We see the existence of both $$a'$$ and $$s'$$, and $$s'$$ could be unseen, for example, on the very first update, where we are at $$s$$, take action $$a$$, and could arrive at any state $$s'$$.
The very first update is made after taking action $$a$$ in state $$s$$ and observing reward $$r$$ plus next state $$s'$$. There is no other way in Q learning of knowing what the next state is in order to process the update. So $$s'$$ in not unseen, it has been observed.
Another way to put this: In model-free methods, the update to Q value estimates for state $$S_t$$ action $$A_t$$ on time step $$t$$ is always made on or after time step $$t+1$$, when $$R_{t+1}$$ and $$S_{t+1}$$ are known.
However, on that same time step, the action $$a'$$ does not yet need to have been taken. The value of $$a'$$ is not taken from $$A_{t+1}$$, but is evaluated for all possibilities. Even after $$A_{t+1}$$ is known and has been taken, the need to process all possible actions in the state $$s'$$ (in order to update $$Q(s,a)$$ or $$Q(S_t, A_t)$$) can often lead to needing action value estimates for never seen state/action pairs.
The update step you quoted includes the expected reward (or possibly a custom reward function known to the agent) $$R(s,a)$$, which is not standard for Q learning. It would be more usual to use the observed reward $$r$$. However, it may be ok because a lot of the time the developer/researcher is in charge of the reward signal and can provide $$R(s,a)$$ to the agent, even when the full model is not used.
• @Snowball The actual reward just $r$ - it is not a function of anything else, but an observed value. The random variable would be $R_{t+1}$ Sep 7 at 19:30