Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

Here is David Silver's lecture on that. Look at 9:30 to 10:30.

He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $$Q$$ is used.

• But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?

• Secondly, even if we were to use $$Q$$, we have $$Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$, so we still need to know the transition model, which is unknown.

What am I missing here?

• This seems to be a duplicate of this.
– nbro
Nov 20 '20 at 2:11

In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse Reinforcement Learning $$V^{\pi}(s) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$
In this setting, $$Q^{\pi}$$ can be obtained from $$V^{\pi}$$ because we have access to the transition model $$P(s'|s,a)$$. The $$Q^{\pi}$$ values allow us to carry out a step in $$\textbf{policy improvement}$$ as in policy iteration.
To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for $$\textbf{state values}$$ is not helpful in determining how to carry out model free control because we cannot compute $$Q^{\pi}(s,a)$$ from $$V^{\pi}$$ in the model free case.
The update for SARSA in model free control is $$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$
Even though we do not know the transition model, we are essentially $$\textbf{sampling}$$ from $$P(s'|s,a)$$ by allowing the environment to provide us the possible next states $$s'$$ that we may end up in. The following update for SARSA is equivalent to computing $$Q^{\pi}(s,a) = R(s) + \gamma E_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$ Essentially this should give the same $$Q^{\pi}(s,a)$$ values when we have the ground truth $$P(s'|s,a)$$ values for the model free case.
• Because SARSA update on the RHS utilises $Q(s',a')$. Also, as $a'$ is selected from the policy $\pi$, this is equivalent to the definition of $V^{\pi}$ Nov 20 '20 at 7:25