# Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Sutton and Barto define the state–action–next-state reward function, $$r(s, a, s')$$, as follows (equation 3.6, p. 49)

$$r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}=s^{\prime}\right]=\sum_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}}$$

Why is the term $$p(s' \mid s, a)$$ required in this definition? Shouldn't the correct formula be $$\sum_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$$?

Expectation of reward after taking action $$a$$ in state $$s$$ and ending up in state $$s'$$ would simply be

$$$$r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')$$$$

The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $$p(s', r|s, a)$$, which represents probability for ending up in state $$s'$$ with reward $$r$$ after taking action $$a$$ in state $$s$$. This probability can be separated in 2 parts using product rule

$$$$p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)$$$$

which represents the probability for getting to state $$s'$$ from $$(s, a)$$, and then probability for getting reward $$r$$ after ending up in $$s'$$.

If we define reward expectation through the joint distribution, we would have

\begin{align} r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\ &= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a) \end{align}

but this would not be correct, since we have this extra $$p(s'|s, a)$$, so we divide everything by it to get expression with only $$p(r|s', s, a)$$.

So, in the end we have

$$$$r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}$$$$

$$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$$ represents the probability of observing reward $$r$$ in state $$s'$$, given that state $$s'$$ is the next state transitioned to. The equation assumes a probability distribution of rewards $$r$$ over state $$s'$$, meaning that a different reward might be observed whenever a state transitions from $$s$$ to $$s'$$. In most cases, if $$r(s, a, s')$$ is a deterministic reward then $$p(s', r \mid s, a) = p(s' \mid s,a )$$.

• But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent. Commented Apr 14, 2020 at 8:02
• Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$ Commented Apr 14, 2020 at 8:05
• Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that. Commented Apr 14, 2020 at 8:08
• In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something? Commented Apr 14, 2020 at 8:10
• I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$. Commented Apr 14, 2020 at 8:38