Is the expected value we sample in TD-learning action-value Q or state-value V?

Question

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, the underlying structure of both algorithms is exactly the same. However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that?

Answer 1

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that?

MC and TD are methods for associating value estimates to time step based on experienced gained in later time steps. It does not matter what kind of value estimate is being associated across time, because all value functions are expressing the same thing in general, which is the expected return conditioned on a "current position" within the MDP. In MC the association is directly with a sampled return, in TD with a sampled combination of immediate reward and a later value estimate - most commonly in TD the same kind of value estimate (e.g. matching later state value estimates to state values).

Both approaches can be analysed and used from the perspective of both state value (V) and action value (Q) functions. They also apply to other value functions - e.g. afterstate values.

It is quite common for textbooks and tutorials to use the slightly simpler state value function to explain how MC or TD learning work in general, outside of being used for any purpose. You can also use the state value function for model-free policy evaluation in MC and TD.

However, without a model, you cannot use state value function for control (i.e. to learn an optimal policy). To pick the best action using state values, you need to do something like this:

$$\pi(s) = \text{argmax}_a [ \sum_{r,s'} p(r,s'|s,a)(r + \gamma v(s'))]$$

The problem here is that $p(r,s'|s,a)$ is a model of the environment. So, if it is needed, the control method would not be model-free.

Hence when you learn about MC or TD in a control scenario, model-free methods to learn optimal policies, then you generally need to use an action value (sometimes you can use an afterstate value, if the action involves choosing the next state directly).

With an action value function, the greedy policy becomes:

$$\pi(s) = \text{argmax}_a q(s, a)$$

This does not refer to any model of the environment. So it can be used when you have none.