Here is David Silver's lecture on that. Look at 9:30 to 10:30.

He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.

  • But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?

  • Secondly, even if we were to use $Q$, we have $Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$, so we still need to know the transition model, which is unknown.

What am I missing here?

  • $\begingroup$ This seems to be a duplicate of this. $\endgroup$
    – nbro
    Nov 20 '20 at 2:11

In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse Reinforcement Learning $$V^{\pi}(s) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$

In this setting, $Q^{\pi}$ can be obtained from $V^{\pi}$ because we have access to the transition model $P(s'|s,a)$. The $Q^{\pi}$ values allow us to carry out a step in $\textbf{policy improvement}$ as in policy iteration.

To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for $\textbf{state values}$ is not helpful in determining how to carry out model free control because we cannot compute $Q^{\pi}(s,a)$ from $V^{\pi}$ in the model free case.

The update for SARSA in model free control is $$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$

Even though we do not know the transition model, we are essentially $\textbf{sampling}$ from $P(s'|s,a)$ by allowing the environment to provide us the possible next states $s'$ that we may end up in. The following update for SARSA is equivalent to computing $$Q^{\pi}(s,a) = R(s) + \gamma E_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$ Essentially this should give the same $Q^{\pi}(s,a)$ values when we have the ground truth $P(s'|s,a)$ values for the model free case.

  • $\begingroup$ Awesome! Why are the formulas for V and Q the same in your answer? Isn't V = Pi * Q ? $\endgroup$
    – Aung Khant
    Nov 20 '20 at 5:43
  • $\begingroup$ Because SARSA update on the RHS utilises $Q(s',a')$. Also, as $a'$ is selected from the policy $\pi$, this is equivalent to the definition of $V^{\pi}$ $\endgroup$
    – calveeen
    Nov 20 '20 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.