# Reward-related formulation in reinforcement learning

I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $$r(s, a, s')$$. My question is actually how do we come to that final derivation? Surprisingly, the denominator of $$p(s'|s, a)$$ can literally be replaced by $$p(s', r|s, a)$$ as eq. 3.4 suggests, then it will end up with "$$r$$" term only due to cancellation of numerator $$p(s', r|s, a)$$ and denominator $$p(s'|s, a)$$. Any explanation on that would be appreciated.

No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $$\sum_{r \in \mathcal{R}}$$ in the right-hand side Equation (3.4).

Equation (3.4) says (leaving out the middle part):

$$p(s' \vert s, a) \doteq \sum_{r \in \mathcal{R}} p(s', r \vert s, a).$$

If you plug this into Equation (3.6) to substitute the denominator, you can't forget about that sum, you have to include the complete sum in the denominator. Because we have two different sums summing over all rewards in $$\mathcal{R}$$, I'll change the symbol used in the second sum to $$r'$$ rather than $$r$$. This yields:

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{\sum_{r' \in \mathcal{R}} p(s', r' \vert s, a)}.$$

The numerator and denominator are different and will not cancel out.

Intuitively, the numerator is just a single probability; the probability of observing a specific next state $$s'$$ and a reward $$r$$ given a specific current state $$s$$ and action $$a$$.

The denominator is a sum of many such probabilities, for all possible rewards $$r'$$ rather than just one specific reward $$r$$. Because this sum "covers" all possible events for the rewards, it essentially represents simply the probability of observing state $$s'$$ given state $$s$$ and action $$a$$, paired with any arbitrary reward. That can more simply be denoted as $$p(s' \vert s, a)$$.... which makes sense because that's the very thing we started with :D

Equation (3.6) is as follows (again leaving the middle part out):

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

In normal English, the left-hand side says "what reward do we expect to get (on average) if we transition from state $$s$$ to state $$s'$$ by executing action $$a$$?"

Such an $$\color{red}{\text{expectation of a quantity }r}$$ can always be computed by multiplying $$\color{blue}{\text{the possible values that }r\text{ can take}}$$ by $$\color{orange}{\text{the probability of each particular value occurring}}$$, and summing up those multiplications. In math, this looks like:

$$\color{red}{r(s, a, s')} = \sum_{\color{blue}{r \in \mathcal{R}}} \color{blue}{r} \times \color{orange}{p(r \vert s, a, s')}.$$

Now, there is a rule in probability that says (see Rule of Multiplication on https://stattrek.com/probability/probability-rules.aspx):

$$p(A, B) = p(A) \times p(B \vert A),$$

so, if we take $$A = s'$$, $$B = r$$, and $$s, a$$ as additional givens for all probabilities, we get:

$$p(s', r \vert s, a) = p(s' \vert s, a) \times p(r \vert s, a, s').$$

This can be rewritten (dividing both sides of the equation by $$p(s' \vert s, a)$$ and swapping the left-hand and right-hand sides) to:

$$p(r \vert s, a, s') = \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

Plugging this in for the orange term in the coloured equation above (which itself is hopefully fairly easy to understand intuitively) yields Equation (3.6).

• @Shah Ah right, I completely missed that you also asked for a derivation of Eq. (3.6). That one actually indeed wasn't immediately obvious to me either. I've edited a derivation with a starting point that's hopefully intuitive to understand into my answer. – Dennis Soemers Oct 19 '18 at 17:37
• @Shah Ignore the givens $s$ and $a$, since those are givens everywhere. Intuition for $p(s', r) = p(s') \times p(r | s')$: "To observe $s'$ and $r$ at the same time, we can say that we first want to just observe $s'$ by itself, and then on top of that add the requirement of observing $r$ given that we have already observed $s'$". Does that help to build intuition? – Dennis Soemers Oct 19 '18 at 19:01
• @Shah The simplification you agree with and the unsimplified product rule your describe afterwards are identical, except for $s$ and $a$ being added as givens across the board. The probability of $r$ is conditioned on $s'$ in both of the equations you wrote there (and additionally also on $s$ in the unsimplified one) – Dennis Soemers Oct 19 '18 at 19:17
• @Shah You can apply the rule directly if you take $A = s' \vert s, a$ and take $B = r \vert s, a$. – Dennis Soemers Oct 20 '18 at 8:39
• @Shah On both. "$p(s', r \vert s, a)$" is read as "the probability of observing $s'$ AND $r$, given that we have observed $s$ AND $a$" – Dennis Soemers Oct 20 '18 at 11:51