# What is the relationship between the Q and V functions?

Suppose we have a policy $$\pi$$ and we use SARSA to evaluate $$Q^\pi(s, a)$$, where $$a$$ is the policy $$\pi$$.

Can we say that $$Q^\pi(s, a) = V^\pi(s)$$?

The reason why I think this can be the case is because $$Q^\pi(s, a)$$ is defined as the value obtained from taking action $$a$$ and then following policy $$\pi$$ thereafter. However, the action $$a$$ taken is the policy according to $$\pi$$ for all $$s \in S$$. This seems to corresponds to the value function equation of $$V^\pi(s_t) = r(s_t) + \gamma V^\pi(s_{t+1})$$.

Can we say that $$Q^\pi(s, a) = V^\pi(s)$$

No.

The correct relationship is this:

$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$

or, if you have a deterministic policy $$a = \pi(s)$$ you can instead write:

$$V^\pi(s) = Q^\pi(s, \pi(s))$$

Intuitively, this is because the $$V^\pi(s)$$ is the expected future return when following the policy $$\pi$$ from state $$s$$, whilst $$Q^\pi(s, a)$$ is the expected future return where it ignores the policy for only the next action $$a$$ (which will decide immediate reward $$r$$ and next state $$s'$$ independently of the policy), and thereafter follows $$\pi$$.

The above equations essentially show what happens when you apply the policy in state $$s$$ to decide which Q value(s) to use, they remove the independent choice of $$a$$ in $$Q^\pi(s,a)$$.

One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When any algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $$s$$ when making a free choice of $$a$$ for the next step, and thereafter strictly following the policy being evaluated.

What is different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different target policy. Both approaches have the same interpretation of what $$Q(s,a)$$ means otherwise, and have the same relationship between Q and V for their repsective policies.

• Just a minor edit I would like to add. Instead of $s$ and $a$ its better to use $s_t$ and $a_t$ since $Q(s,a)$ is itself an expectation of to which state ($s_{t+1}$)the actions will take us next and what rewards it will produce on the way. – DuttaA Mar 21 '20 at 14:55