Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I often see, the state-action value function is expressed as:
$q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \mathbb{E}[R_{t+1}+\gamma v_{\pi}(s') |S_t = s, A_t =a]$
Why does expressing the future return in the time $t+1$ as a state value function $v_{\pi}$ make the expected value under policy change to expected value in general?
Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E,a_t \sim \pi}[r(s_t,a_t) + \gamma G_{t+1} | S_t = s, A_t = a]\; ;$$ where $r(s_t,a_t)$ is written to show that the reward gained at time $t+1$ is a function of the state and action tuple we have at time $t$ (note here that $G_{t+1}$ would be just the sum of future reward signals). This allows us to show that the expectation is taken under the joint distribution of $s,r\sim E$ where $E$ is the environment and actions are taken from our policy distribution.
As we have conditioned on knowing $a_t$ then the only random variable in the expectation that is dependent on our policy $\pi$ is $G_{t+1}$ because this is the sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as $$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma \mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] | S_t = s, A_t = a]\;,$$ where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e. $$\mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] = v_{\pi}(s')\;.$$ This leads us to get what you have $$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma v_{\pi}(s') | S_t = s, A_t = a]\;,$$ where the only difference is that we have made clear what our expectation is taken with respect to.
The expectation is taken with respect to the conditional distribution $S_{t+1},R_{t+1}|A_t,S_t$, and we usually include the $\pi$ subscript to denote that they are also taking the expectation with respect to the policy, but here this does not effect the first term as we have conditioned on knowing $A_t$ and only applies to the future reward signals.
David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts.
Next reward $R_{t+1}$: The next reward $R_{t+1}$ is solely dependent on the current state $S_t$ and action $A_t$. It is only dependent on the policy because the policy details the probability distribution of actions given a state. Since we assume that the current state and action are given when calculating the expectation $\left(S_t = s, A_t = a\right)$, then the policy does not give us any new information, and therefore the next reward is independent of the policy.
Return $G_{t+1}$: By definition, $v_{\pi}(s') = \mathbb{E}_{\pi}[G_{t+1}|S_{t+1} = s']$. The value function is unaffected by sampling actions from the policy in the outer expectation $\left(\mathbb{E}_{\pi}[v_{\pi}(s')] = \mathbb{E}[v_{\pi}(s')]\right)$ since the value function is an expectation under the policy, and hence samples actions from the policy already.
Dropping $\pi$ from $\mathbb{E}_{\pi}$: The expectation under the current policy samples next states and rewards from the environment and also samples actions from our policy $\pi$. Because the next reward is independent of the policy given the current state and action, and because the value function is unaffected by sampling actions from the policy in the outer expectation, we can simply drop the policy from the outer expectation (the outer expectation will still sample next states and rewards from the environment).
