In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state:
$$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$
In Q-learning, we try to find a policy that maximize the state-action value function Q:
$$ \pi_q^*=\underset{\pi}{\operatorname{argmax}} {Q_{\pi}(s,a)} $$
However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, $V_{\pi}(s)$ is connected to $Q_{\pi}(s,a)$ by:
$$ V_{\pi}(s)=E_{a\sim\pi}[Q_{\pi}(s,a)] $$
So
$$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E_{a\sim\pi}[Q_{\pi}(s,a)] $$
And if $\pi_v^*=\pi_q^*$, mathematically I'm not sure why the expectation $E_{a\sim\pi}$ before $Q_{\pi}(s,a)$ can be simply dropped.