2
$\begingroup$

I'm not having a lot of intuition about the equation. I have this Bellman update rule:

$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$

But where are the parenthesis? Is the second sum using the index $a$ from the first sum? Or is it independent, and can I move out the $[r+ \gamma v_{k}(s')]$ term out of the sum?

$\endgroup$

1 Answer 1

3
$\begingroup$

Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).

$$v_{\pi}(s) =\sum_a \pi(a \mid s) \left(\sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$

Now, let me answer your other questions.

Is the second sum using the index $a$ from the first sum?

Yes.

Or is it independent, and can I move out the $[r+ \gamma v_\pi(s')]$ term out of the sum?

No, and you cannot move this term out of the sum because the second sum is a sum over $r$ and $s'$ and $r+ \gamma v_\pi(s')$ depends on those terms.

Note that $v_{\pi}(s)$ is defined as an expectation and that $\pi(a \mid s)$ (the policy) and $p(s',r \mid s,a)$ (the model) are probability distributions.

$\endgroup$
3
  • 1
    $\begingroup$ Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this. $\endgroup$ Jan 18 at 9:00
  • 1
    $\begingroup$ I'm sorry, I'm coming back and got confused, shouldn't be like this: $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$ Since the second sum uses the same $a$ index.. Putting the indexes the way @nbro does implies the sums are independent $\endgroup$
    – nammerkage
    Jan 18 at 9:35
  • 1
    $\begingroup$ @nammerkage You can also write that and with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$ $\endgroup$
    – nbro
    Jan 18 at 11:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .