1
$\begingroup$

I did not understand how third equality follows from the second equality. Could some expert explain?

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

As $q_{\pi}(s,a)$ is defined in S&B RL book as follows:

we define the value of taking action a in state s under a policy $\pi$, denoted $q_{\pi}(s,a)$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$

Therefore similar to state value, your highlighted step here is trying to represent your action value's $\mathbb{E}_{\pi}[G_{t+1}|S_t=s,A_t=a]$ component by the state values of all possible entering states $s'$ after taking action $a$ following the same policy $\pi$ thereafter. As for the other component involving $R_{t+1}$ you've probably already known it's only determined by the environment under above conditions.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged .