1
$\begingroup$

Why are there different forms of the value function in reinforcement learning?

Sutton & Barto (2nd edition, equation 3.14) define the state value function as follows

$$v_{\pi}(s) = \displaystyle\sum_a \pi(a|s) \displaystyle\sum_{s',r}p(s',r \mid s,a)[r + \gamma v_\pi(s')].$$

Many lecture notes (for example, these by Haim Sompolinsky, MCB 131, 2017) define it as follows (equation 18)

$$V^{\pi}(s) =\displaystyle\sum_a \pi(a,s) [R(s,a) + \gamma \displaystyle\sum_{s'}P(s' \mid s,a) V^\pi(s')].$$

Are these two forms the same?

Improve this question
$\endgroup$
3
  • $\begingroup$ Please, focus on one question. You're asking at least 2 distinct questions here. Also, provide the links to the slides from Carnegie Mellon. I think your formulas have been copied incorrectly or imprecisely. Definitely, the notation is not fully correct. $\endgroup$
    – nbro
    Feb 28, 2022 at 12:06
  • 2
    $\begingroup$ Have I seen this question before? In any case, the first equation contains some mistakes (no declaration of $r$, it should be in the second sum, and $r_{t+1}$ appears when there are no time steps - should be just $r$). Both equations seem to skip superscript on $V^{\pi}$. It would be good to fix these, and/or give full references to where the equations are from (Sutton&Barto numbers its key equations, so you could say exactly which one you meant to copy). I don't want to assume anything and "fix" equations myself when the question is directly about them. $\endgroup$
    – Neil Slater
    Feb 28, 2022 at 13:38
  • 1
    $\begingroup$ I edited the question so that it reflects the sources of the equations. $\endgroup$
    – cgo
    Mar 1, 2022 at 3:43

1 Answer 1

Reset to default
3
$\begingroup$

There are a few different, but equivalent, ways to express the relationships between value functions in the Bellman equations.

Some differences are just notation, but in the case of the two equations that you give, there is a more practical difference.

Specifcally, the term $R(s,a)$ from the second equation that you give is the expected reward from taking action $a$ in state $s$. Even though individual rewards may depend on $s'$, the expectation does not for a consistent MDP.

You can see that the two equations are equivalent, when you consider that:

$$R(s,a) = \sum_{r,s'} r p(r,s'|s,a)$$

and

$$P(s'|s,a) = \sum_r p(r,s'|s,a)$$

The first equation for me is the more intuitive version of the state value Bellman equation as it explicitly shows all the dependencies for the transition. The second equation can be more practical in some circumstances, because if you know or easily calculate $R(s,a)$ for any state/action pair, then it is often more convenient to use it than recalculate each time from first principles. In addition, $R(s,a)$ is often under the control of the developer, and may well be some constant or fixed simple equation based on $s$ and $a$, in which case the $R(s,a)$ notation captures that exactly.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is correct but not that this is just the definition of the state value function, so I don't think that the practical aspects is very important here, but I understand what you mean. It may also be worth noting that, in many cases, the reward is deterministic, so $r(s, a)$ will just be the expectation of a constant, which is a constant itself, i.e. $r(s, a)$ can be replaced with a deterministic reward function rather than being an expectation. $\endgroup$
    – nbro
    Mar 1, 2022 at 10:56
  • 1
    $\begingroup$ @nbro: I would say that the Bellman equations are not the same as the state value definition. The definition IMO is $v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+1+k} | S_{t} = s]$ and the Bellman equations in the question are derived from it. $\endgroup$
    – Neil Slater
    Mar 1, 2022 at 13:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .