# What is the difference between these two versions of the Bellman equation?

The first version is the one I am most familiar with:

$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$

where $$T(s, a, s')$$ represents the probability of transitioning from state $$s$$ to state $$s'$$ given action $$a$$. And where $$R(s, a, s')$$ represents the expected value of the reward from transitioning from state $$s$$ to state $$s'$$ given action a.

In another textbook, I saw this version of the equation:

$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]$$

Is the second equation more general since it considers the probability of going to $$s'$$ AND getting a reward $$r$$?

Or are they somehow the same (if so, why)?

Or maybe they are the same under certain circumstances (like conditional independence of $$s'$$ and $$r$$)?

• I believe the second version might be more familiar to many people because it is from the classic book by Sutton and Barto:-) They are actually the same depending on how you define $T$ and $R$.
– user58136
Commented Nov 2, 2022 at 0:53
• By the way, here you can find a detailed explanation about the second version: github.com/MathFoundationRL
– user58136
Commented Nov 2, 2022 at 0:54
• @RLControl The second edition of that book uses that second form, but the first edition doesn't (if I remember correctly).
– nbro
Commented Dec 31, 2022 at 10:19

The two are equivalent.

\begin{align} V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma V_\pi(s')] \\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\sum_{r}p(r| s',a,s)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\left[\sum_{r}p(r| s',a,s)r + \gamma \sum_{r}p(r| s',a,s)V_\pi(s')] \right]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\left[R(s',a,s) + \gamma V_\pi(s')] \right]\\ \end{align}

We first split $$p(s',r|s,a) = p(s'|s,a)p(r|s',a,s)$$ by law of conditional probability, and we note that $$p(s'|s,a) = T(s',a,s)$$ by definition. We split the sum to parts over $$s'$$ and $$r$$, and then split it further to separate the two terms taking the expectation over $$r$$ and $$\gamma V_\pi(s')$$. By definition, the first expectation equals $$R(s',a,s)$$. In the second expecation, $$V_\pi$$ is independent of $$r$$ so the probability sums to one and vanishes.

• Just one note for people that get confused with the last part. If $k$ is a constant and we have $k \sum_{i=1}^N f(i)$, then this is equivalent to $k f(1) + k f(2) + \dots + k f(N) = k * (f(1) + f(2) + \dots + f(N))$. In this specific case, $p$ is a distribution. So, the sum of all $f(i)$ is equal to one, so we multiply $1$ times $k$, which in this case is $V_\pi(s')$.
– nbro
Commented Dec 30, 2022 at 15:12