While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is:

enter image description here

According to what I understand, the formula should have been correct without the denominator (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.


Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be \begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation} The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $p(s', r|s, a)$ which represents probability for ending up in state $s'$ with reward $r$ after taking action $a$ in state $s$. This probability can be separated in 2 parts using product rule \begin{equation} p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a) \end{equation} which represent probability for getting to state $s'$ from $(s, a)$ and then probability for getting reward $r$ after ending up in $s'$.
If we define reward expectation through joint distribution we would have \begin{align} r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\ &= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a) \end{align} but this would not be correct since we have this extra $p(s'|s, a)$ so we divide everything by it to get expression with only $p(r|s', s, a)$. So, in the end we have

\begin{equation} r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)} \end{equation}


$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$ represents the probability of observing reward $r$ in state $s'$, given that state $s'$ is the next state transitioned to. The equation assumes a probability distribution of rewards $r$ over state $s'$, meaning that a different reward might be observed whenever a state transitions from $s$ to $s'$. In most cases, if $r(s, a, s')$ is a deterministic reward then $p(s', r \mid s, a) = p(s' \mid s,a )$.

  • $\begingroup$ But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent. $\endgroup$ – SAGALPREET SINGH Apr 14 '20 at 8:02
  • $\begingroup$ Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$ $\endgroup$ – calveeen Apr 14 '20 at 8:05
  • $\begingroup$ Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that. $\endgroup$ – SAGALPREET SINGH Apr 14 '20 at 8:08
  • $\begingroup$ In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something? $\endgroup$ – SAGALPREET SINGH Apr 14 '20 at 8:10
  • $\begingroup$ I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$. $\endgroup$ – calveeen Apr 14 '20 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.