When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

Question

I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when is more beneficial than the other? I have a large state space.

Answer 1

The core differences between using $V(s)$ or $Q(s,a)$ are:

$V(s)$ cannot be used stand-alone to decide a policy.

You either need a separate policy function $\pi(a|s)$ that it is the value function for, or you can derive a policy from it if you have access to the environment's distribution model $p(r,s'|s,a)$ - the probability of receiving reward $r$ and arriving in next state $s'$ given start $s,a$. The derived policy would be deterministic $\pi(s) = \text{argmax}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$

In comparison, $Q(s,a)$ can be used to derive a policy without reference to any model: $\pi(s) = \text{argmax}_a Q(s,a)$. This makes the action value function Q necessary for model-free value-based control methods. Hence Monte-Carlo Control, SARSA, Q-Learning will all use action values.

$V(s)$ is usually more compact than $Q(s,a)$.

Space used by a state value table is $O(|\mathcal{S}|)$

Space used by an action-value table is $O(|\mathcal{S} \times \mathcal{A}|)$

These space concerns also apply when you are using approximation - the input domain you need to learn to approximate $Q(s,a)$ is larger than the one you need to learn $V(s)$. Although for large state spaces with lots of dimensions, the relative difference after approximation may be small enough that you don't really care.

The smaller space makes $V(s)$ a good choice if you can use it. The benefits are smaller memory footprint, and often faster learning because there are fewer separate values to learn (not always faster, as you still need to explore all actions in all states to learn accurate values).

Typical scenarios where you can use $V(s)$ instead of $Q(s,a)$

• You have access to the full environment model, as described in the first section, above.

or

• You are working with a separate policy function. In optimal control, this typically occurs in actor-critic methods, which are relatively advanced RL approach, but a popular one, especially when the action space is large. The situation also occurs in prediction problems where the task is not to optimise anything, but to evaluate a fixed policy.

or

• Your actions are directly choosing the next state. This is called afterstate representation, and is subtly different to state values $V(s)$, but has similar benefit of smaller space used. It is useful in determining values in board games and similar environments.

If none of these situations applies, then you will have to use action-value function $Q(s,a)$. You might also choose to use $Q(s,a)$ because it is easier to stick with it in whatever framework you are working in. Sometimes the benefits of using $V(s)$ instead are not large enough to be worth the effort.