1
$\begingroup$

Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$. If we are operating a greedy policy and hence acting optimally, doesn't this mean that in fact the policy is deterministic and then $\pi(a|s)$ is $1$ for the optimal action and $0$ for all others? Would this then lead to an equivalence between the two?

Here is my work to formulate some form of proof where I start with the idea that a policy is defined to be better than a current policy if for all states then $Q_{\pi}(S,\pi^∗(s))\geq Vπ_{\pi}(s)$ :

I iteratively apply the optimal policy to each time step until I eventually get to a fully optimal time step of rewards

$$Vπ_{\pi}(s)≤Q_{\pi}(S,\pi^∗(s))$$ $$=Eπ[R_{t+1}+\gamma V_{\pi}(St+1)|St=s]$$ $$\leq E[Rt+1+\gamma Q_{\pi}(S_{t+1},\pi^∗(S_{t+1})|S_t=s]$$ $$\leq E[Rt+1+\gamma Rt+2+\gamma 2Q \pi^*(S_{t+2},\pi^∗(S_{t+2})|S_t=s]$$ $$\leq E[R_{t+1}+\gamma R_{t+2}+....|S_t=s]$$ $$=V\pi^∗(s)$$

I would say that our final two lines are in fact inequalities, and for me this makes intuitive sense in that if we are always taking a deterministic greedy action our value function and Q function are the same. As detailed here, for a given policy and state we have that $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$ and if the policy is optimal and hence greedy then $\pi(a|s)$ is deterministic.

$\endgroup$
1
$\begingroup$

In general they are not the same and that should be clear as to why -- mathematically you are conditioning on an extra random variable being known in the state-action value function. You have the correct relationship between them, but I think your understanding of the two may be slightly off. The state-action value function is a function of both $s$ and $a$ whereas the value function is a function of just $s$. As you have noted, the value function is equal to the expected value of the state-action value function, with the expectation taken over the action distribution induced by the policy.

However, if you have a deterministic policy, then two are equivalent only if you evaluate the state-action value function at the action that the deterministic policy gives. That is, $v_\pi(s) = Q(s, \pi(s))$. This is because, as you say, for a given state the policy will assign probability one to a certain action and 0 to all other actions in the action space and so the equation I just wrote is what the expectation in your question reduces to.

Now, you might think 'well the policy is deterministic so we are always going to choose this action, so my theory was correct'. This is not true. The definition of the state-action value function is the value of the expected future (possibly discounted) returns given that we are in state $s$ and given that we have taken action $a$. This means that we have already chosen our action and we are not choosing it according to the policy -- the policy will only choose all future actions.

You also say that "If we are operating a greedy policy and hence acting optimally" which is not true. Just because we act greedily it does not imply we are acting optimally. Value functions are functions of a policy. If the policy is completely random then you would get certain value and state-action value functions which you could act greedily to but this would not be optimal.

$\endgroup$
4
  • $\begingroup$ Hi @DavidIreland I completely agree about the last paragraph. What I was trying to say was that acting greedily WITH an optimal policy then we would be acting optimally. Would this be correct? Given we have found an optimal policy and there is no noise in the environment if we act greedily then it should also be optimal, correct? $\endgroup$ – InvestingScientist Dec 15 '20 at 12:26
  • $\begingroup$ Your other points also makes sense, we are saying that in fact they wont be equivalent because although we evaluate all future steps with the optimal policy, the current step was chosen for this state, as defined by the definition of state action values. Is this correct? $\endgroup$ – InvestingScientist Dec 15 '20 at 12:27
  • 1
    $\begingroup$ If you are acting greedily according to optimal action-value functions then yes you are acting optimally. with regards to the second comment, yes that is indeed what we are saying. All future steps would agree except the returns would differ depending on what the current action taken was. $\endgroup$ – David Ireland Dec 15 '20 at 12:34
  • $\begingroup$ Perfect, thank you for explaining $\endgroup$ – InvestingScientist Dec 15 '20 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.