1
$\begingroup$

It says here in the Conservative Q-Learning paper that "standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions" (Section 3.1).

I don't see how this is true. For every $(s,a)$ pair, we need to update $Q(s,a)$ to reduce the value $|Q(s,a) - R(s,a) - \gamma E[\max_{a'}Q(s',a')]|$ until it converges to zero.

We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the very first update, where we are at $s$, take action $a$, and could arrive at any state $s'$.

Can someone explain this?

$\endgroup$
1
  • $\begingroup$ I believe they are referring to the fact that we only update the Q-function for state values that we have seen. This is true as typically we use a replay buffer of stored experience to update the Q-function -- all of these values have been 'seen' as we have experienced them. $\endgroup$ Sep 7 at 15:44
0
$\begingroup$

We see the existence of both $a'$ and $s'$, and $s'$ could be unseen, for example, on the very first update, where we are at $s$, take action $a$, and could arrive at any state $s'$.

The very first update is made after taking action $a$ in state $s$ and observing reward $r$ plus next state $s'$. There is no other way in Q learning of knowing what the next state is in order to process the update. So $s'$ in not unseen, it has been observed.

Another way to put this: In model-free methods, the update to Q value estimates for state $S_t$ action $A_t$ on time step $t$ is always made on or after time step $t+1$, when $R_{t+1}$ and $S_{t+1}$ are known.

However, on that same time step, the action $a'$ does not yet need to have been taken. The value of $a'$ is not taken from $A_{t+1}$, but is evaluated for all possibilities. Even after $A_{t+1}$ is known and has been taken, the need to process all possible actions in the state $s'$ (in order to update $Q(s,a)$ or $Q(S_t, A_t)$) can often lead to needing action value estimates for never seen state/action pairs.

The update step you quoted includes the expected reward (or possibly a custom reward function known to the agent) $R(s,a)$, which is not standard for Q learning. It would be more usual to use the observed reward $r$. However, it may be ok because a lot of the time the developer/researcher is in charge of the reward signal and can provide $R(s,a)$ to the agent, even when the full model is not used.

$\endgroup$
2
  • $\begingroup$ Re the last point, my notation is incorrect, I meant to say r(s,a) for actual reward. I had also misremembered the algorithm, and added an expectation where it does not belong. Thanks for the answer. $\endgroup$
    – Snowball
    Sep 7 at 19:22
  • $\begingroup$ @Snowball The actual reward just $r$ - it is not a function of anything else, but an observed value. The random variable would be $R_{t+1}$ $\endgroup$ Sep 7 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.