Skip to main content
simplifications + latex
Source Link
nbro
  • 41.4k
  • 12
  • 114
  • 205

While reading about reinforcement learning, I have come acrossSutton and Barto define the following expression for expected rewards in terms of a summationstate–action–next-state reward function, the denominator of which I am not able to account for.

The formula given is:$r(s, a, s')$, as follows (equation 3.6, p. 49)

enter image description here$$ r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}=s^{\prime}\right]=\sum_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}} $$

According to what I understand, the formula should have been correct withoutWhy is the denominatorterm (that I have highlighted). How is$p(s' \mid s, a)$ required in this formuladefinition? Shouldn't the correct formula be $\sum_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$?

Source of image: Andrew G and Sutton's book on RL.

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is:

enter image description here

According to what I understand, the formula should have been correct without the denominator (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.

Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49)

$$ r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}=s^{\prime}\right]=\sum_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}} $$

Why is the term $p(s' \mid s, a)$ required in this definition? Shouldn't the correct formula be $\sum_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$?

edited tags; edited title
Link
nbro
  • 41.4k
  • 12
  • 114
  • 205

Formula for expected rewards for state–action–next-state triples as a three-argument Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Became Hot Network Question
added 1 character in body; edited title
Source Link
nbro
  • 41.4k
  • 12
  • 114
  • 205

MDP: Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about Reinforcement Learningreinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for. The

The formula given is: [![enter image description here][1]][1]

enter image description here

According to what I understand:, the formula should have been coorectcorrect without the denominator  (that I have highlighted). Please explain howHow is this formula correct?

Source of image: Andrew G and Sutton's book on RL [1]: https://i.sstatic.net/Mc7Te.png.

MDP: Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about Reinforcement Learning, I have come across following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for. The formula given is: [![enter image description here][1]][1]

According to what I understand: the formula should have been coorect without the denominator(that I have highlighted). Please explain how is this formula correct?

Source of image: Andrew G and Sutton's book on RL [1]: https://i.sstatic.net/Mc7Te.png

Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for.

The formula given is:

enter image description here

According to what I understand, the formula should have been correct without the denominator  (that I have highlighted). How is this formula correct?

Source of image: Andrew G and Sutton's book on RL.

Source Link
Loading