Here is David Silver's lecture on that. Look at 9:30 to 10:30.

He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.

  • But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?

  • Secondly, even if we were to use $Q$, we have $Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$, so we still need to know the transition model, which is unknown.

What am I missing here?

  • $\begingroup$ This seems to be a duplicate of this. $\endgroup$
    – nbro
    Commented Nov 20, 2020 at 2:11

1 Answer 1


In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse Reinforcement Learning $$V^{\pi}(s) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$

In this setting, $Q^{\pi}$ can be obtained from $V^{\pi}$ because we have access to the transition model $P(s'|s,a)$. The $Q^{\pi}$ values allow us to carry out a step in $\textbf{policy improvement}$ as in policy iteration.

To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for $\textbf{state values}$ is not helpful in determining how to carry out model free control because we cannot compute $Q^{\pi}(s,a)$ from $V^{\pi}$ in the model free case.

The update for SARSA in model free control is $$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$

Even though we do not know the transition model, we are essentially $\textbf{sampling}$ from $P(s'|s,a)$ by allowing the environment to provide us the possible next states $s'$ that we may end up in. The following update for SARSA is equivalent to computing $$Q^{\pi}(s,a) = R(s) + \gamma E_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$ Essentially this should give the same $Q^{\pi}(s,a)$ values when we have the ground truth $P(s'|s,a)$ values for the model free case.

  • $\begingroup$ Awesome! Why are the formulas for V and Q the same in your answer? Isn't V = Pi * Q ? $\endgroup$
    – Aung Khant
    Commented Nov 20, 2020 at 5:43
  • $\begingroup$ Because SARSA update on the RHS utilises $Q(s',a')$. Also, as $a'$ is selected from the policy $\pi$, this is equivalent to the definition of $V^{\pi}$ $\endgroup$
    – calveeen
    Commented Nov 20, 2020 at 7:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .