Timeline for What does $v(S_{t+1})$ mean in the optimal state-action value function?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 15, 2021 at 9:32 | vote | accept | Daviiid | ||
| Jun 15, 2021 at 7:58 | comment | added | Daviiid | I understand now. Thank you for the explanations and all your clarifications. | |
| Jun 15, 2021 at 6:50 | comment | added | Neil Slater | @Daviiid: OK, I think I understand the concern. With discrete probability distributions then the problems with the convention are minor and do not impact any of the sums. For continuous distributions you have to be more careful, although I think most of the manipulations of the expectations etc in RL would still be fine. | |
| Jun 14, 2021 at 21:43 | comment | added | Daviiid | I'd like to apologize first for not being able to add your name with the @. There is no difference between what I've written and your answer I just wanted to rewrite it as a clarification to my question. I just wanted to know if there is a particular reason for the authors for choosing their convention for functions of random variables. Because normally a function of a random variable would change the distribution of probabilities, we just imagine a scalar multiplication of a Gaussian random variable and how the distribution changes. | |
| Jun 14, 2021 at 21:06 | comment | added | Neil Slater | @Daviiid I cannot see any difference between what you have written and I have done in the question, in terms of needing to interpret $v_*(S_{t+1})$. You have just resolved the expectation to the sum more conventionally (the double-variable sum $\sum_{r,s'}$ is a different conventon I have borrowed from Sutton & Barto). But I don't think it uses a different interpretation of what $v_*(S_{t+1})$ is? Sorry I am not a notation expert, and may have missed some subtlety. | |
| Jun 14, 2021 at 21:01 | comment | added | Daviiid | Thank you for your answer. Can I ask please if there is a particular reason for going with this convention ? I think we can still get the same formula with the normal association of functions and random variables; $\mathbb{E}[v_{*}(S_{t+1})|S_{t}=s,A_{t}=a]=\sum_{s'}{p(s'|s,a)v_{*}(s')}=\sum_{r,s'}{p(r,s'|s,a)v_{*}(s')}$ since $\sum_{r}{p(r,s'|s,a)}=1$, or am I missing something ? | |
| Jun 14, 2021 at 20:57 | comment | added | Neil Slater | @Daviiid Yes I think that has correctly put my first two paragraphs into notation. Sort of, I think you have some typos, i.e. $X$ is a random variable in $\Omega$ (the "to $\mathbb{R}$" after that looks like a typo), and $f(x): \Omega \rightarrow \mathbb{R}$ | |
| Jun 14, 2021 at 20:55 | comment | added | Daviiid | I'm not sure if I understood well so let me please ask a question in general terms. If $X$ is a random variable from $\Omega$ to $\mathbb{R}$ and $f$ is a function on $\mathbb{R}$ then according to the convention, $f(X)$ is a random variable from $\Omega$ to $\mathbb{R}$ and the values of this random variable are $f(x)$ for $x$ in $\Omega$ but $P[f(X)=f(x)]=P[X=x]$ right ? | |
| Jun 14, 2021 at 20:48 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 27 characters in body
|
| Jun 14, 2021 at 20:43 | history | answered | Neil Slater | CC BY-SA 4.0 |