I'm trying to solve the exercise 3.11 from the second-edition from the Sutton and Barto's book
Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, then what is the expectation of $R_{t+1}$ in terms of $\pi$ and the four-argument function $p$ (3.2)?
Here's my attempt.
For each state $s$, the expected immediate reward when taking action $a$ is given in terms of $p$ by eq 3.5 in the book:
$r(s,a) = \sum_{r \in R} r \, \sum_{s'\in S} p(s',r| s,a) = E[R_t | S_{t-1} = s, A_{t-1} = a]$ (1)
The policy $\pi(a | s)$ on the other hand gives the probability of taking action $a$ given state $s$.
Is it possible to express the expectation of the immediate reward over all actions $A$ from the state $s$ using (1) as
$$E[R_t | S_{t-1} = s, A] = \sum_{a \in A} \pi(a|s) r(a,s) \ \ \ \ \ \ \ \ (2) ?$$
If this is valid, is this also valid in the next time step
$$E[R_{t+1} | S_{t} = s, A] = \sum_{a \in A} \pi(a|s) r(a,s) \ \ \ \ \ \ \ \ (3) ?$$
If (2) and (3) are OK, then
$$E[R_{t+1} | S_{t} = s, A] = \sum_{a \in A} \pi(a|s) \sum_{r \in R} r \, \sum_{s'\in S} p(s',r| s,a)$$