# Where are the parentheses in the Bellman update rule?

I'm not having a lot of intuition about the equation. I have this Bellman update rule:

$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$

But where are the parenthesis? Is the second sum using the index $$a$$ from the first sum? Or is it independent, and can I move out the $$[r+ \gamma v_{k}(s')]$$ term out of the sum?

Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).

$$v_{\pi}(s) =\sum_a \pi(a \mid s) \left(\sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$

Is the second sum using the index $$a$$ from the first sum?

Yes.

Or is it independent, and can I move out the $$[r+ \gamma v_\pi(s')]$$ term out of the sum?

No, and you cannot move this term out of the sum because the second sum is a sum over $$r$$ and $$s'$$ and $$r+ \gamma v_\pi(s')$$ depends on those terms.

Note that $$v_{\pi}(s)$$ is defined as an expectation and that $$\pi(a \mid s)$$ (the policy) and $$p(s',r \mid s,a)$$ (the model) are probability distributions.

• Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this. Jan 18 at 9:00
• I'm sorry, I'm coming back and got confused, shouldn't be like this: $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$ Since the second sum uses the same $a$ index.. Putting the indexes the way @nbro does implies the sums are independent Jan 18 at 9:35
• @nammerkage You can also write that and with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$
– nbro
Jan 18 at 11:47