I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below.

enter image description here

I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is actually how do we come to that final derivation?

Surprisingly, the denominator of $p(s'|s, a)$ can literally be replaced by $p(s', r|s, a)$ as eq. 3.4 suggests, then it will end up with "$r$" term only due to cancellation of numerator $p(s', r|s, a)$ and denominator $p(s'|s, a)$.

Any explanation on that would be appreciated.


1 Answer 1


No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4).

Equation (3.4) says (leaving out the middle part):

$$p(s' \vert s, a) \doteq \sum_{r \in \mathcal{R}} p(s', r \vert s, a).$$

If you plug this into Equation (3.6) to substitute the denominator, you can't forget about that sum, you have to include the complete sum in the denominator. Because we have two different sums summing over all rewards in $\mathcal{R}$, I'll change the symbol used in the second sum to $r'$ rather than $r$. This yields:

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{\sum_{r' \in \mathcal{R}} p(s', r' \vert s, a)}.$$

The numerator and denominator are different and will not cancel out.

Intuitively, the numerator is just a single probability; the probability of observing a specific next state $s'$ and a reward $r$ given a specific current state $s$ and action $a$.

The denominator is a sum of many such probabilities, for all possible rewards $r'$ rather than just one specific reward $r$. Because this sum "covers" all possible events for the rewards, it essentially represents simply the probability of observing state $s'$ given state $s$ and action $a$, paired with any arbitrary reward. That can more simply be denoted as $p(s' \vert s, a)$.... which makes sense because that's the very thing we started with :D

Equation (3.6) is as follows (again leaving the middle part out):

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

In normal English, the left-hand side says "what reward do we expect to get (on average) if we transition from state $s$ to state $s'$ by executing action $a$?"

Such an $\color{red}{\text{expectation of a quantity }r}$ can always be computed by multiplying $\color{blue}{\text{the possible values that }r\text{ can take}}$ by $\color{orange}{\text{the probability of each particular value occurring}}$, and summing up those multiplications. In math, this looks like:

$$\color{red}{r(s, a, s')} = \sum_{\color{blue}{r \in \mathcal{R}}} \color{blue}{r} \times \color{orange}{p(r \vert s, a, s')}.$$

Now, there is a rule in probability that says (see Rule of Multiplication on https://stattrek.com/probability/probability-rules.aspx):

$$p(A, B) = p(A) \times p(B \vert A),$$

so, if we take $A = s'$, $B = r$, and $s, a$ as additional givens for all probabilities, we get:

$$p(s', r \vert s, a) = p(s' \vert s, a) \times p(r \vert s, a, s').$$

This can be rewritten (dividing both sides of the equation by $p(s' \vert s, a)$ and swapping the left-hand and right-hand sides) to:

$$p(r \vert s, a, s') = \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

Plugging this in for the orange term in the coloured equation above (which itself is hopefully fairly easy to understand intuitively) yields Equation (3.6).

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – nbro
    Commented Jan 3, 2021 at 23:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .