Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.

Can we say that $Q^\pi(s, a) = V^\pi(s)$?

The reason why I think this can be the case is because $Q^\pi(s, a)$ is defined as the value obtained from taking action $a$ and then following policy $\pi$ thereafter. However, the action $a$ taken is the policy according to $\pi$ for all $s \in S $. This seems to corresponds to the value function equation of $V^\pi(s_t) = r(s_t) + \gamma V^\pi(s_{t+1})$.


Can we say that $Q^\pi(s, a) = V^\pi(s)$


The correct relationship is this:

$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$

or, if you have a deterministic policy $a = \pi(s)$ you can instead write:

$$V^\pi(s) = Q^\pi(s, \pi(s))$$

Intuitively, this is because the $V^\pi(s)$ is the expected future return when following the policy $\pi$ from state $s$, whilst $Q^\pi(s, a)$ is the expected future return where it ignores the policy for only the next action $a$ (which will decide immediate reward $r$ and next state $s'$ independently of the policy), and thereafter follows $\pi$.

The above equations essentially show what happens when you apply the policy in state $s$ to decide which Q value(s) to use, they remove the independent choice of $a$ in $Q^\pi(s,a)$.

One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When any algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $s$ when making a free choice of $a$ for the next step, and thereafter strictly following the policy being evaluated.

What is different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different target policy. Both approaches have the same interpretation of what $Q(s,a)$ means otherwise, and have the same relationship between Q and V for their repsective policies.

  • $\begingroup$ Just a minor edit I would like to add. Instead of $s$ and $a$ its better to use $s_t$ and $a_t$ since $Q(s,a)$ is itself an expectation of to which state ($s_{t+1}$)the actions will take us next and what rewards it will produce on the way. $\endgroup$ – DuttaA Mar 21 '20 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.