Timeline for How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 19, 2021 at 13:42 | vote | accept | Asher | ||
| Jan 20, 2021 at 17:05 | history | edited | nbro | CC BY-SA 4.0 |
deleted 62 characters in body; edited tags
|
| Jul 2, 2020 at 18:03 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Jun 2, 2020 at 17:26 | history | edited | Asher | CC BY-SA 4.0 |
the another -> another
|
| S Jun 1, 2020 at 12:22 | history | suggested | Pluviophile | CC BY-SA 4.0 |
fixed grammar
|
| May 30, 2020 at 9:40 | review | Suggested edits | |||
| S Jun 1, 2020 at 12:22 | |||||
| May 26, 2020 at 13:29 | history | edited | Asher | CC BY-SA 4.0 |
clearer notation
|
| May 26, 2020 at 11:37 | answer | added | Asher | timeline score: 3 | |
| May 25, 2020 at 23:25 | history | edited | nbro | CC BY-SA 4.0 |
edited tags; edited title
|
| May 25, 2020 at 15:44 | comment | added | David | If so then you would be able to get $\mathbb{E}[R_t | S_{t-1} = s, A_{t-1} = a] = \sum_{s'} \mathbb{E}[R_t | S_{t-1} = s, A_{t-1} = a, S_t = s'] \mathbb{P}(S_t = s' | s, a) = r(s,a)$ | |
| May 25, 2020 at 15:36 | comment | added | David | $r(s,a,s') = \mathbb{E}[R_t | S_{t-1} = s, A_{t-1} = a, S_t = s']$? | |
| May 25, 2020 at 15:24 | comment | added | Asher | I understand, but I think I didn't imply that...? In fact I'm kinda saying that you could get away by marginalizing s,a in the r(s,a,s') function (if you can say marginalize in this context, since r(s,a,s') is not a probability distribution). | |
| May 25, 2020 at 15:09 | comment | added | David | sorry, my bad for not noticing the link. Note that if you have a joint distribution of $(X,Y)$ you can't find $\mathbb{E}[Y]$ by simply summing over $y$ and using the joint pmf - you would first need to marginalise the joint pmf to get the single pmf of $Y$, so your expectation doesn't work out. | |
| May 25, 2020 at 14:58 | comment | added | Asher | I actually had that thread linked in my question, but: 1) I'm not claiming that the different reward functions can be made equivalent, but that the optimal policy to the overall MDP can; 2) In their solutions book, Norvig and Russell describe a transformation based on extending the state space with pre and post states, and a few more changes to the discount factor and the transitions to account for these additional states; 3) I wanted to know if taking the expectation over s' can do the trick too, at least for the R(s,a,s') to the R(s,a) case. | |
| May 25, 2020 at 13:18 | comment | added | David | You might want to see this How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent? answer. | |
| May 25, 2020 at 11:22 | review | First posts | |||
| May 26, 2020 at 0:36 | |||||
| May 25, 2020 at 11:19 | history | asked | Asher | CC BY-SA 4.0 |