The renowned book [Reinforcement Learning: An Introduction][1] (2nd edition), by Sutton and Barto, provides a different update rule than your first update rule for policy evaluation. Their update rule is more similar to your second update rule. See [section 4.1][2]. They also provide the pseudocode for policy evaluation on page 75 of the book. You can also find the pseudocode [here][3]. Moreover, note that the update rule doesn't need to change only because the rewards are tied to the outcome of an action. This information is encoded in the functions $p$ (the transition function) and $r$ (the reward function) of the Markov decision process, which is incorporated in the update rule. If you want to understand the update rule, you should read the relevant pages (especially, chapter 4) of the cited book. [1]: http://incompleteideas.net/book/RLbook2020.pdf [2]: http://incompleteideas.net/book/RLbook2020.pdf#page=96 [3]: https://github.com/udacity/rl-cheatsheet/blob/master/cheatsheet.pdf