# Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is: According to what I understand, the formula should have been correct without the denominator (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.

## 2 Answers

Expectation of reward after taking action $$a$$ in state $$s$$ and ending up in state $$s'$$ would simply be $$\begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation}$$ The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $$p(s', r|s, a)$$ which represents probability for ending up in state $$s'$$ with reward $$r$$ after taking action $$a$$ in state $$s$$. This probability can be separated in 2 parts using product rule $$\begin{equation} p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a) \end{equation}$$ which represent probability for getting to state $$s'$$ from $$(s, a)$$ and then probability for getting reward $$r$$ after ending up in $$s'$$.
If we define reward expectation through joint distribution we would have \begin{align} r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\ &= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a) \end{align} but this would not be correct since we have this extra $$p(s'|s, a)$$ so we divide everything by it to get expression with only $$p(r|s', s, a)$$. So, in the end we have

$$\begin{equation} r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)} \end{equation}$$

$$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$$ represents the probability of observing reward $$r$$ in state $$s'$$, given that state $$s'$$ is the next state transitioned to. The equation assumes a probability distribution of rewards $$r$$ over state $$s'$$, meaning that a different reward might be observed whenever a state transitions from $$s$$ to $$s'$$. In most cases, if $$r(s, a, s')$$ is a deterministic reward then $$p(s', r \mid s, a) = p(s' \mid s,a )$$.

• But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent. – SAGALPREET SINGH Apr 14 '20 at 8:02
• Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$ – calveeen Apr 14 '20 at 8:05
• Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that. – SAGALPREET SINGH Apr 14 '20 at 8:08
• In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something? – SAGALPREET SINGH Apr 14 '20 at 8:10
• I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$. – calveeen Apr 14 '20 at 8:38