The action-value function DOES take into account the policy being followed - that's precisely what the notation $\mathbb{E}_\pi$ is for. Specifically, $\mathbb{E}_\pi$ is a shorthand for
\begin{align*}
\mathbb{E}_{a_i \sim~ \pi(a_i \,|\, s_i), \, (r_{i+1}, s_{i+1}) \sim p(r_{i+1}, s_{i+1} \, |\, s_{i}, \, a_{i}), \, \forall i}
\end{align*}
where $p$ represents the environment's joint distribution of reward/transitions. This means that the expectation is with respect to the actions, states, and rewards you see under the policy $\pi$. If you want to make this more concrete, we can write out the expectation like this
\begin{align*}
& \mathbb{E}_\pi\left[\sum_{k=0}^{\infty}\gamma^k r_{k+t+1} | s_t, a_t \right] \\
:=& \int_{(r_{t+1}, s_{t+1})}p(r_{t+1}, s_{t+1})\bigg[r_{t+1} + \int_{a_{t+1}}\pi(a_{t+1})\int_{(r_{t+2}, s_{t+2})}p(r_{t+2}, s_{t+2})\bigg[r_{t+2} + \ldots
\end{align*}
(Really, you should be integrating over dummy variables, and I've omitted the conditional expectations, e.g. $p(r_{t+1}, s_{t+1})$ should be $p(r_{t+1}, s_{t+1} | s_t, a_t)$). Of course you can just replace the integrals with sums if you want discrete actions and/or states. So you can see just writing $\mathbb{E}_\pi$ sweeps a lot of the notation and true meaning under the rug, and I assume that is the source of the confusion.