Why does Q-function training not query the Q-function value at unobserved states?

Question

In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that

standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions

I don't see how this is true. For every $(s,a)$ pair, we need to update $Q(s,a)$ to reduce the value $|Q(s,a) - R(s,a) - \gamma E[\max_{a'}Q(s',a')]|$ until it converges to zero.

We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the very first update, where we are at $s$, take action $a$, and could arrive at any state $s'$.

Can someone explain this?

Answer 1

We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the very first update, where we are at $s$, take action $a$, and could arrive at any state $s'$.

The very first update is made after taking action $a$ in state $s$ and observing reward $r$ plus next state $s'$. There is no other way in Q learning of knowing what the next state is in order to process the update. So $s'$ in not unseen, it has been observed.

Another way to put this: In model-free methods, the update to Q value estimates for state $S_t$ action $A_t$ on time step $t$ is always made on or after time step $t+1$, when $R_{t+1}$ and $S_{t+1}$ are known.

However, on that same time step, the action $a'$ does not yet need to have been taken. The value of $a'$ is not taken from $A_{t+1}$, but is evaluated for all possibilities. Even after $A_{t+1}$ is known and has been taken, the need to process all possible actions in the state $s'$ (in order to update $Q(s,a)$ or $Q(S_t, A_t)$) can often lead to needing action value estimates for never seen state/action pairs.

The update step you quoted includes the expected reward (or possibly a custom reward function known to the agent) $R(s,a)$, which is not standard for Q learning. It would be more usual to use the observed reward $r$. However, it may be ok because a lot of the time the developer/researcher is in charge of the reward signal and can provide $R(s,a)$ to the agent, even when the full model is not used.

Answer 2

In the offline dataset $\mathcal{D}$, each experience is of the form $(s, a, s^{\prime}, r)$ (state, action, next state, reward). So $s^{\prime}$ is in your dataset