Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

Question

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state:

$$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$

In Q-learning, we try to find a policy that maximize the state-action value function Q:

$$ \pi_q^*=\underset{\pi}{\operatorname{argmax}} {Q_{\pi}(s,a)} $$

However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, $V_{\pi}(s)$ is connected to $Q_{\pi}(s,a)$ by:

$$ V_{\pi}(s)=E_{a\sim\pi}[Q_{\pi}(s,a)] $$

So

$$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E_{a\sim\pi}[Q_{\pi}(s,a)] $$

And if $\pi_v^*=\pi_q^*$, mathematically I'm not sure why the expectation $E_{a\sim\pi}$ before $Q_{\pi}(s,a)$ can be simply dropped.

Answer 1

Your question can be answered by observing the expression of the Bellman Optimality Equation: $$ v(s) = {\max_\pi} \sum_{a} {\pi(a|s)}\left(\sum_{r}p(r|s,a)r + \gamma \sum_{s'}p(s'|s,a)v(s')\right),\nonumber\\ \doteq {\max_\pi} \sum_{a} {\pi(a|s)} q(s,a),\quad \text{for all } s\in\mathcal{S}. $$

If $v_{\pi^*}(s)$ is the solution to this equation, then $q(s,a)$ in this case equals to $q_{\pi^*}(s,a)$, which is the action value under $\pi^*$. That is, the optimal state value of $s$ equals the maximum optimal action value at $s$.

You can check the details of the above equation in Equation (3.1) in the book: Mathematical foundation of reinforcement learning.

Moreover, the Bellman optimality equation can be expressed in either state values or action values. See the expression in action values in equation (7.16) in the book.

Answer 2

I am not sure your equations are correct or I understand very well your question but I write the main equations of $V$ and $Q$ functions from Sutton book that may help you:

$ v^{*}(s) = max v_{\pi}(s) $

$ q^{*}(s,a) = max q_{\pi}(s,a) $

$ v^{*}(s)=max\mathbb{E}[R_{t+1}+\gamma v^{*}(s+1)|S_{t}=s, A_{t}=a] $

$ q^{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma max q^{*}(s+1,a^{'})|S_{t}=s, A_{t}=a] $

$ v^{*}(s)=max q^{*}_{\pi}(s,a) $

As you can see, the optimal value function is equal to the maximum Q function on all feasible actions. We will reach optimal policy if we calculate the $argmax$ of these functions on possible actions. Also, the policy found by the value function is not always the same as the Q function because their features are different. The Q function considers both actions and states that not only decrease the speed of the process but also choose the shortest optimal path. For example, you can test simple graphical navigation on below Github repository: https://github.com/pouyan-asg/global-path-planning

In this example, you will see the path that agent chooses to reach the destination is not similar in both cases, and the timing in value function mode is higher (because the agent does not consider the action). I hope my answer is clear.