Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on which actions follow it.
On page 58 of Sutton & Barto's book, we have
So, how is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter (i.e. the policy followed after having taken action $a$ in the state $s$)?
Q-learning can learn about the greedy policy (the policy that we define as $\pi(s) = \arg\max_a Q(s, a)$) whilst following some arbitrary exploratory policy because Q-learning is an off-policy algorithm.
In Q-learning, we are updating our values of $Q(s, a)$ using a bootstrapped value from one time step in the future. This means that we don't need to worry about any importance sampling type re-weighting of the action chosen by the exploratory policy that we use for action selection because of how $Q(\cdot, \cdot)$ is defined: $$Q(s, a) = \mathbb{E}_\pi \left[ G_t | S_t = s, A_t = a \right] \; ;$$ where $G_t$ is the (discounted) future returns defined in Sutton and Barto. Now, because we have defined $Q(\cdot, \cdot)$ such that we condition on knowing action $A_t$, it really doesn't matter which distribution this action came from, because as mentioned we have conditioned on knowing it. This allows us to make updates in Q-learning for any state-action pair using the target from the optimal policy despite us not using this for action selection in the environment.
If we were to do some kind of $n$-step Q-learning with an update target looking something like $$R(s_t, a_t) + R(s_{t+1}, a_{t+1}) + ... + R(s_{t+n}, a_{t+n}) + \max_{a'} Q(s_{t+n+1}, a')$$ then we would need to use importance sample to re-weight the trajectory to account for the actions $a_{t+i}$ for $i \geq 1$ as these are actions that the optimal policy may not have taken, as they were taken from the exploratory policy and the Q-function assumed that the trajectory (i.e. future actions) is generated under the policy associated with the Q-function.
The action-value function DOES take into account the policy being followed - that's precisely what the notation $\mathbb{E}_\pi$ is for. Specifically, $\mathbb{E}_\pi$ is a shorthand for \begin{align*} \mathbb{E}_{a_i \sim~ \pi(a_i \,|\, s_i), \, (r_{i+1}, s_{i+1}) \sim p(r_{i+1}, s_{i+1} \, |\, s_{i}, \, a_{i}), \, \forall i} \end{align*} where $p$ represents the environment's joint distribution of reward/transitions. This means that the expectation is with respect to the actions, states, and rewards you see under the policy $\pi$. If you want to make this more concrete, we can write out the expectation like this \begin{align*} & \mathbb{E}_\pi\left[\sum_{k=0}^{\infty}\gamma^k r_{k+t+1} | s_t, a_t \right] \\ :=& \int_{(r_{t+1}, s_{t+1})}p(r_{t+1}, s_{t+1})\bigg[r_{t+1} + \int_{a_{t+1}}\pi(a_{t+1})\int_{(r_{t+2}, s_{t+2})}p(r_{t+2}, s_{t+2})\bigg[r_{t+2} + \ldots \end{align*} (Really, you should be integrating over dummy variables, and I've omitted the conditional expectations, e.g. $p(r_{t+1}, s_{t+1})$ should be $p(r_{t+1}, s_{t+1} | s_t, a_t)$). Of course you can just replace the integrals with sums if you want discrete actions and/or states. So you can see just writing $\mathbb{E}_\pi$ sweeps a lot of the notation and true meaning under the rug, and I assume that is the source of the confusion.
