# Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $$\pi^*$$ as the policy that maximizes the value of the state:

$$\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}$$

In Q-learning, we try to find a policy that maximize the state-action value function Q:

$$\pi_q^*=\underset{\pi}{\operatorname{argmax}} {Q_{\pi}(s,a)}$$

However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, $$V_{\pi}(s)$$ is connected to $$Q_{\pi}(s,a)$$ by:

$$V_{\pi}(s)=E_{a\sim\pi}[Q_{\pi}(s,a)]$$

So

$$\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E_{a\sim\pi}[Q_{\pi}(s,a)]$$

And if $$\pi_v^*=\pi_q^*$$, mathematically I'm not sure why the expectation $$E_{a\sim\pi}$$ before $$Q_{\pi}(s,a)$$ can be simply dropped.

• "we try to find a policy that maximize the state-action value function Q", really? Or are we trying to find a policy that is assumed to be greedy wrt to the optimal state-action function? In an MDP, there's a unique optimal value function 𝑉 and state-action value function 𝑄. I am not sure I understand your question or confusion. It seems that you're mixing the theory (mathematical results) with the practice (algorithms). However, asking if two algorithms lead to the same optimal policy is reasonable because there can be more than one. So, can you clarify what you're really asking here?
– nbro
Jul 3, 2022 at 8:24
• Your first equation has a problem - an optimal policy is not the one that maximizes the value of a particular state $s$ - rather it maximizes the value of any state. The second equation has the same problem, along with having $a$ is unspecified - here, you should take the expectation w.r.t $\pi$. Jul 3, 2022 at 9:04

Your question can be answered by observing the expression of the Bellman Optimality Equation: $$v(s) = {\max_\pi} \sum_{a} {\pi(a|s)}\left(\sum_{r}p(r|s,a)r + \gamma \sum_{s'}p(s'|s,a)v(s')\right),\nonumber\\ \doteq {\max_\pi} \sum_{a} {\pi(a|s)} q(s,a),\quad \text{for all } s\in\mathcal{S}.$$

If $$v_{\pi^*}(s)$$ is the solution to this equation, then $$q(s,a)$$ in this case equals to $$q_{\pi^*}(s,a)$$, which is the action value under $$\pi^*$$. That is, the optimal state value of $$s$$ equals the maximum optimal action value at $$s$$.

You can check the details of the above equation in Equation (3.1) in the book: Mathematical foundation of reinforcement learning.

Moreover, the Bellman optimality equation can be expressed in either state values or action values. See the expression in action values in equation (7.16) in the book.

I am not sure your equations are correct or I understand very well your question but I write the main equations of $$V$$ and $$Q$$ functions from Sutton book that may help you:

$$v^{*}(s) = max v_{\pi}(s)$$

$$q^{*}(s,a) = max q_{\pi}(s,a)$$

$$v^{*}(s)=max\mathbb{E}[R_{t+1}+\gamma v^{*}(s+1)|S_{t}=s, A_{t}=a]$$

$$q^{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma max q^{*}(s+1,a^{'})|S_{t}=s, A_{t}=a]$$

$$v^{*}(s)=max q^{*}_{\pi}(s,a)$$

As you can see, the optimal value function is equal to the maximum Q function on all feasible actions. We will reach optimal policy if we calculate the $$argmax$$ of these functions on possible actions. Also, the policy found by the value function is not always the same as the Q function because their features are different. The Q function considers both actions and states that not only decrease the speed of the process but also choose the shortest optimal path. For example, you can test simple graphical navigation on below Github repository: https://github.com/pouyan-asg/global-path-planning

In this example, you will see the path that agent chooses to reach the destination is not similar in both cases, and the timing in value function mode is higher (because the agent does not consider the action). I hope my answer is clear.