2
$\begingroup$

The first version is the one I am most familiar with:

$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$

where $T(s, a, s')$ represents the probability of transitioning from state $s$ to state $s'$ given action $a$. And where $R(s, a, s')$ represents the expected value of the reward from transitioning from state $s$ to state $s'$ given action a.

In another textbook, I saw this version of the equation:

$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]$$

Is the second equation more general since it considers the probability of going to $s'$ AND getting a reward $r$?

Or are they somehow the same (if so, why)?

Or maybe they are the same under certain circumstances (like conditional independence of $s'$ and $r$)?

$\endgroup$
3
  • 1
    $\begingroup$ I believe the second version might be more familiar to many people because it is from the classic book by Sutton and Barto:-) They are actually the same depending on how you define $T$ and $R$. $\endgroup$
    – user58136
    Commented Nov 2, 2022 at 0:53
  • $\begingroup$ By the way, here you can find a detailed explanation about the second version: github.com/MathFoundationRL $\endgroup$
    – user58136
    Commented Nov 2, 2022 at 0:54
  • $\begingroup$ @RLControl The second edition of that book uses that second form, but the first edition doesn't (if I remember correctly). $\endgroup$
    – nbro
    Commented Dec 31, 2022 at 10:19

1 Answer 1

4
$\begingroup$

The two are equivalent.

\begin{align} V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma V_\pi(s')] \\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\sum_{r}p(r| s',a,s)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\left[\sum_{r}p(r| s',a,s)r + \gamma \sum_{r}p(r| s',a,s)V_\pi(s')] \right]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s',s,a)\left[R(s',a,s) + \gamma V_\pi(s')] \right]\\ \end{align}

We first split $p(s',r|s,a) = p(s'|s,a)p(r|s',a,s)$ by law of conditional probability, and we note that $p(s'|s,a) = T(s',a,s)$ by definition. We split the sum to parts over $s'$ and $r$, and then split it further to separate the two terms taking the expectation over $r$ and $\gamma V_\pi(s')$. By definition, the first expectation equals $R(s',a,s)$. In the second expecation, $V_\pi$ is independent of $r$ so the probability sums to one and vanishes.

$\endgroup$
1
  • $\begingroup$ Just one note for people that get confused with the last part. If $k$ is a constant and we have $k \sum_{i=1}^N f(i)$, then this is equivalent to $k f(1) + k f(2) + \dots + k f(N) = k * (f(1) + f(2) + \dots + f(N))$. In this specific case, $p$ is a distribution. So, the sum of all $f(i)$ is equal to one, so we multiply $1$ times $k$, which in this case is $V_\pi(s')$. $\endgroup$
    – nbro
    Commented Dec 30, 2022 at 15:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .