# Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is:

According to what I understand, the formula should have been correct without the denominator (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.

Expectation of reward after taking action $$a$$ in state $$s$$ and ending up in state $$s'$$ would simply be $$$$r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')$$$$ The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $$p(s', r|s, a)$$ which represents probability for ending up in state $$s'$$ with reward $$r$$ after taking action $$a$$ in state $$s$$. This probability can be separated in 2 parts using product rule $$$$p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)$$$$ which represent probability for getting to state $$s'$$ from $$(s, a)$$ and then probability for getting reward $$r$$ after ending up in $$s'$$.
If we define reward expectation through joint distribution we would have \begin{align} r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\ &= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a) \end{align} but this would not be correct since we have this extra $$p(s'|s, a)$$ so we divide everything by it to get expression with only $$p(r|s', s, a)$$. So, in the end we have
$$$$r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}$$$$
$$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$$ represents the probability of observing reward $$r$$ in state $$s'$$, given that state $$s'$$ is the next state transitioned to. The equation assumes a probability distribution of rewards $$r$$ over state $$s'$$, meaning that a different reward might be observed whenever a state transitions from $$s$$ to $$s'$$. In most cases, if $$r(s, a, s')$$ is a deterministic reward then $$p(s', r \mid s, a) = p(s' \mid s,a )$$.
• Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$ – calveeen Apr 14 '20 at 8:05
• I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$. – calveeen Apr 14 '20 at 8:38